Block Fullness Investigation

[gregorycoppola]

Abstract

In early January, we initially noticed that blocks were, on average, around 15% full, or less (sheet). In other words, blocks are much less full than they could be.

This led to two questions:

1. Is this a problem?
   • i.e., are there valid transactions in the mempools of the miners that could have filled this space, but were prevented from by some unknown factor?
2. If it is a problem, what is the source of the problem?

The conclusions were:

1. Yes this is a problem
   • There are more transactions in the mempool than are being included in blocks
2. The source of the problem is transaction processing slowness
   • The speed of transactions is around: 1-5 tx/s
   • The first time out for checkpoints to be sent are: 1s, then 30s
   • So, the first attempt will have 1-5 tx / block

Possible States of Mempool

Given that the blocks are not very full, there are then three possibilities with respect to the mempool:

1. Mempool is empty
   • There is nothing more in the mempool to process
   • So, there is nothing more in the blocks
   • No problem
2. Mempool is largely invalid
   • There are transactions in the mempool but they are not ready to process, e.g., have bad nonce
   • So, there is nothing more to add to the blocks
   • No problem
3. Mempool contains valid transactions
   • The mempool has valid transactions to be processed
   • Therefore, some unknown problem prevents block fullness
   • Problem!

The conclusion, we will see, is that the mempool does contain valid transactions, that simply aren’t being included.

Mempool Evidence

We looked at three basic kinds of evidence

1. Measuring block fullness
2. Measuring the size of the mempool
3. User reports

Block Fullness

This graph of block fullness versus block height shows that high and low fullness blocks are interspersed.

Assuming the mempool transactions don't arrive all at once, or clear all at once, it would seem that there is a factor other than available mempool.

Mempool Analyzer

We made a tool which analyzes the mempool by creating an “unlimited size block”:

• feat: Mempool Analyzer #3020
  This allows us to know what the cost would be to clear the mempool.

On January 19, 2022, we measured the cost to clear all valid transactions in the mempool as:

cost ExecutionCost {
    write_length: 3687053, 
    write_count: 20758, 
    read_length: 139643627, 
    read_count: 141543, 
    runtime: 2443639433 
}

read_count is almost always the bottleneck, and 141543 represents about 10 blocks. In such a case, we would expect to see 10 full blocks in a row, and then an empty mempool. We don't see this. There does thus conclusively seem to be a problem.

User Reports

Organizing these is TBD, but users were complaining about:

• Getting coins off of exchanges
• Empty blocks were noticed while their transaction was pending
• MegaPont took a long time (?)

Diagnosing the Root Cause

Given that there were transactions in the mempool, we had the following hypotheses:

1. Miners were deliberately leaving the blocks unfull by adjusting their settings
2. The default settings were causing blocks to be unfull

We investigated this by:

• Checking the foundation miner’s logs

We realized that the following:

1. Transaction processing happens at a rate of about 1-5 tx/s
   • This is pretty slow
   • See discord
2. The timeouts at which the miner will send a candidate block to the BTC chain are at:
   • 1 second
   • 30 seconds
3. Thus, the transaction miner would make its first attempt with about 1-5 tx in a block
   • This results in a relatively empty block

The conclusion from this section of the experiments is that:

• Block emptiness is the result of running with default settings

This raises the prospect that miners with full blocks are actually the ones who have changed their algorithms!

Recommendations

Block emptiness is a result of the interplay between:

1. Transaction processing speed
   • slow
2. Processing timeouts
   • soon

This correspondingly leads to two solutions:

1. Speed up processing
2. Extend or adjust the timeouts

Of these, adjusting timeouts (2) is easy, and was implemented in https://github.com/stacks-network/stacks-blockchain/releases/tag/2.05.0.1.0.

Speeding up Processing

Speeding up processing is non-trivial.

Initial profiling suggested that there wasn’t an obvious “silly” bug that could easily be rectified to improve processing speed. It was concluded that real performance improvements would be necessary.

The highest leverage point identified was:

• address the MARF bottleneck
  • use caching
    • we initially suggested doing this in RAM
  • proposed in
    • Profile and Optimize Mining Code #3014
    • https://discord.com/channels/621759717756370964/730814571517968436/935252771500871680

This strategy has proven promising, with the following concrete issues open to improving the MARF:

• Issues
  • [chainstate] memoize block-commit descendancy information for SortitionHandleTx::descended_from #3045
  • [chainstate] implement cache strategies for nodes in tries that are persisted to disk #3042
  • [chainstate] defer trie root hash calculation until no more writes will occur on the trie #3041
  • [chainstate] store large MARF tries in a separate flat file #3059
• PR’s
  • MARF Performance Improvements  #3044
  • fix/memoize-block-commit-descendancy #3048

It’s unclear right now what the full improvement factor will be. The faster processing will lead to fuller blocks.

Exact calculations about how many X faster we need to fill the blocks is for future work.

Productionization of Dashboards

Tools created during this investigation have been partially checked in as durable tools:

• Block Fullness
  • Block fullness metrics are now output from the stacks-node, using a prometheus endpoint, and are displayed in the Grafana dash
  • feat: Prometheus Block Fullness Metrics #3025
• Mempool Analysis
  • Not checked in
  • Measures
    • Total transactions
    • Mineable transactions
    • Cost to clear mempool (including in read_count)
  • feat: Mempool Analyzer #3020