Greetings, I'm denravonska (or ravon), one of the volunteer Gridcoin developers. In light of the recent superblock issues where no superblock was created for a few days, I thought it could be a good idea to explain what has happened; why it happened and how we could prevent it from happening in the future. Note that the Neural Network isn't my area of expertise so I might be getting some of the details wrong. However, the big picture still applies.

I am not going to go into details, so it is expected that you know the basics of Gridcoin such as staking, beacons and the Neural Net.

Why do Superblocks Matter?

Superblocks contain a report of every known CPID's magnitude so the nodes know how much each CPID is expected to be rewarded. Sometimes things happen, superblocks are delayed and people start getting anxious.

It is not super critical if a superblock is delayed a bit. What will happen is that new CPIDs won't get picked up while current mags are "locked" in place.

Giving Birth to a Superblock

1. Collect known beacons
2. Download statistics from BOINC servers
3. Discard beacons with no recent credits
4. Calculate each remaining CPID's magnitude into a contract
5. Broadcast contract hash and try to reach a consensus

Roughly speaking, if you're one of the nodes currently eligble for super block participation, and you hold the agreed hash, and you stake a block you can create a superblock. If the nodes have problem reaching a consensus it will delay the superblock. For example, this is the current status:

$ gridcoinresearchd execute currentneuralreport
[
    {
        "Command" : "currentneuralreport"
    },
    [
        {
            "Neural Hash" : "Popularity,Percent %",
            "2668d3b02c892081a05983ac5e4092c8" : "2; 1.53%",
            "2d4e4c50ba6ff5207ac0aa116a14d103" : "3; 1.85%",
            "48e25e844613915eac7a544530d8e7e5" : "31; 20.20%",
            "74007022ad3665961bcd7074d8b26aff" : "13; 8.76%",
            "7e15dab8447b3b3d632c01298095c2ab" : "33; 21.89%",
            "a52e735e1e1b5f8a4f26f1ef4cab9ab6" : "29; 18.76%",
            "c9a585cb4115ca69802e4d6354397e85" : "36; 23.88%",
            "e240be17fcde4b071f38e70dc49a094c" : "5; 3.13%",
            "Superblock Age" : 35314
        }
    ]
]

The nodes are having a heated debate on who's right, forming small gangs of hatred. This is a bit simplified and it's a little more to it but that doesn't matter for now.

Causes

Figuring out the root cause has been a joint effort by the entire community as usual. Lots of feedback, ideas, testing, brainstorming and debugging. The long delay between the two superblocks are believed to be a combination of small flaws in 1, 2 and 3.

1. Collect known beacons - Beacons are continously added and removed to the chain. When loading the wallet, all beacons older than 6 months are skipped but any subsequent beacon received are kept in memory. This means that wallets which have been running for a long time will know about more beacons than wallets just loaded.

2. Download statistics from BOINC servers - Since Gridcoin has seen an uptick in the user base we have also started to put a lot more strain on the BOINC servers. Some of these servers are run on very modest connections such as a home server. As a result some of the download request timed out for some of the nodes, causing them to accept different CPIDs and calculate different mags.

3. Discard beacons with no recent credits - After downloading the BOINC stats the nodes start weeding out CPIDs which are no longer active. They do this by checking the expavg_time XML field for each CPID. Any CPID with a credit time older than 32 days are discarded. This age check is done using the node's local time which means that different nodes will filter different CPIDs depending on their current timezone. While not massive, there can be up to a 25 hour difference in beacon acceptance, furthering diverging the final CPID list. Credits to bullshark.

These are not new problems but the more people joining the more likely it is to get different final hashes.

Upcoming Fixes

Within a few days Soon :) a leisure will be released aimed at adressing these three known issues.

• Nodes will agree on a fixed UTC time window or block height on which to base their filtering. This eliminates beacon collection size differences.

• A BOINC statistics proxy server will be setup to mitigate some of the server pressure when needed. Additionally, statistics files will be checked for modifications before download (also bullshark's idea).

• Statistics CPID filtering will be done on UTC time or by checking the expavg_credit field. See PR #399 and PR #396.

The new server temporarily increases centralization. This is something we just have to live with to get things rolling reliably again in the short term. The long term solution is to distribute the data within the network itself, minimizing the strain on the BOINC servers.