Empty Blocks, Full Mempool
Why do Bitcoin miners sometimes produce empty blocks despite there being unconfirmed transactions in the mempool?
In order to work on finding a valid block to become the new tip of the blockchain, miners need to set half a dozen fields for the header of the new block. The only two fields that are highly dependent on the data from the previous block are the previous block hash and the merkle root of all the transactions in the new block.
There is a period of time after which a miner has received the latest block's header from its network peers but has not yet checked all of the transactions in the block and updated its UTXO set. Because mining is essentially a global race to find the next block, every second of work counts - miners want to always be working on the next block. Thus during this period of time a miner typically avoids putting any transactions in the block (apart from the coinbase transaction), as they don't know which transactions were in the previous block and including any conflicting transactions could result in a double spend which would cause the network to reject the block and not pay the miner their reward. As such, during this period the merkle root of the next block template will be for a merkle tree that only contains a coinbase transaction.
The Theory
If you read my previous research into block time variance then you may recall that we can calculate the expected amount of blocks generated X seconds after the previous block. On average it takes my node at statoshi.info 2 seconds to process a block, but sometimes it takes as long at 7 seconds.
Let's be conservative and assume that it can take up to 10 seconds for a mining pool to validate the latest block and update their mining template. How many blocks should we expect will be mined less than 10 seconds after the previous block?
1 - exp(-10/600) = 0.0165... about 1.7% of blocks, or about 2 blocks per day. So we should expect that in a situation where the mempool doesn't clear out completely, we should have well under 1.7% of blocks being empty. That's the theory, but does it match reality?
The Reality
I wrote this script to find empty blocks that had been mined even when there were likely to be transactions sitting in the mempool. Since it's not possible for us to know the state of the mempool at any point in time in the past, I have to make some assumptions. As such, I'm checking the surrounding blocks mined before and after the empty block. If they are > 95% full in terms of the max block weight, I assume that it's highly unlikely the mempool was empty. The raw results of my analysis can be found here.
As of the past 6 months, we've been averaging about 0.8 empty blocks per day when the mempool is not empty; about half of my assumed max of 1.7 per day.
One thing worth noting is that the accuracy of block timestamps is not great. At the protocol level you only have a strong guarantee that a block's timestamp is within a couple hours of the real time. On the other hand, due to how mining pools and block templates work, trying to perform analysis on time scales below 10 seconds is likely a fruitless endeavor because block templates may be updated less often than that. But what happens when we look at a more granular breakdown of empty blocks that were sandwiched between full blocks?
All in all I found 1,567 empty blocks mined between full blocks in the entire history of the blockchain. 84 of these blocks had negative timestamp deltas where their timestamp was before that of the previous block; they are not displayed on the chart below.
What can we glean from this chart? Firstly, I'd suggest that any empty blocks with timestamps less than a minute after the previous block are noise. They could have easily been created just a few seconds after the previous block. For more details as to why, check out my prior post about block time variance.
There are two areas of the chart that stick out to me. The first is the horizontal cluster of blue unknown mining pool blocks in the 400K - 500K block height range. This is interesting because all of these blocks were mined around (the expected) 10 minutes after the previous block. They are clearly not "fast" blocks that were created empty for the miner's safety. The fact that this was happening for over a year makes me think it was unlikely to be an accidental misconfiguration. I also noticed that the later blocks in this striation were signaling for "NYA" AKA SegWit2X. Perhaps they were mining empty blocks in order to put more pressure on the network to upgrade the block size? Another point to note is that this group of blocks stopped being mined after height 481,149 - just 3 weeks after the Bitcoin Cash fork.
The other noteworthy area of this chart are the two vertical groups around block 550,000. These blocks were all mined by Antpool and ViaBTC. Given that they occurred over a fairly short time frame I'd wager to guess that these were two separate incidents of some sort of misconfiguration or infrastructure issue. What I find more interesting is that this also seems to be a new piece of evidence in support of the rumored relationship between the two pools. Why would both of these pools have the same issue at the same two times unless they were somehow sharing infrastructure? For those of you who were not immersed in the scaling debate back in 2017, allow me to insert a bit of history from Wikipedia:
After two years of debate between the two camps, a Shenzhen-based mining company called ViaBTC, which Bitmain had invested in, orchestrated a hard fork of Bitcoin, creating Bitcoin Cash in August 2017. Wu's critics accuse him of being the mastermind behind the fork, calling him "Jihad", a play on his given name, but Wu denied that he or Bitmain had so much influence in the matter.
Here's the total breakdown of sandwiched empty blocks by mining pool; nothing particularly surprising given that the pools with the most empty blocks are also the pools that had some of the highest hashrates and thus total blocks minted.
This BitMEX Research chart breaking down all empty blocks from 2017 relative to the blocks mined by each pool gives a better picture of each pool's "performance."
What's Missing
BATPOOL, BitClub Network, Bitfury, GBMiners, and Kano CKPool never mined sandwiched empty blocks as far as I can tell. Bitfury is proud that they never mine any empty blocks, as noted in this Bitfury research report from 2016.
Big Blockers, Small Blocks
I recall chuckling when I read this report from BitMEX Research in 2017.
BitMEX Research
However one ironic thing stands out to us, from this analysis. The pools arguing most vigorously for larger blocks, tend on average, to produce smaller blocks.
Interestingly, as of 2020, the only pool that is still sometimes minting egregious empty slow blocks sandwiched between full blocks is ViaBTC, as you can see in the upper right corner of the "empty blocks by time elapsed since previous block" chart. Recall that they were one of the primary orchestrators of the Bitcoin Cash fork.
Future Improvements
Braiins
We have reason to be hopeful that the rate of empty blocks will continue to decrease. With Stratum V2, a next generation mining protocol, pools will be able to dig deeper into the mempool (for transactions that are unlikely to be included in the current block) and construct full block templates for the grandchild block before the current block is found. Then they can send those block templates to miners ahead of time so that only the prevhash message needs to be sent to actually start a new round of mining.
As we can see from looking at years of blockchain history, the base layer of the Bitcoin network is continually improving and becoming stronger, faster, and more efficient.
