blockchain_utilities: add tool to track running counts of locked outputs

[jeffro256]

How to use: ./monero-blockchain-find-locked. This will generate a large CSV file containing locked transaction output information per-block. Here are all the options:

Monero 'Fluorine Fermi' (v0.18.1.0-9866a0e90)

Allowed options:

Command line options:
  --help                                Produce help message

Command line options and settings options:
  --data-dir arg (=/home/x/.bitmonero, /home/x/.bitmonero/testnet if 'testnet', /home/x/.bitmonero/stagenet if 'stagenet')
                                        Specify data directory
  --testnet                             Run on testnet. The wallet must be 
                                        launched with --testnet flag.
  --stagenet                            Run on stagenet. The wallet must be 
                                        launched with --stagenet flag.
  --log-level arg                       0-4 or categories
  --ignore-index arg (=500000000)       ignore non-standard unlock times of 
                                        outputs created after this block index
  --fcmp-index arg (=0)                 if the calculated first spendable block
                                        index is less than or equal to this 
                                        block index, simulate inserting 
                                        immediately
  --output-file arg (=blockchain-find-locked-output.csv)
                                        path to CSV output file

The two interesting options are "ignore-index" and "fcmp-index". Setting ignore-index simulates ignoring unlock times for outputs created after this block index. This means that they are only held around as long the standard time is for them. The fcmp-index is a bit harder to explain. Setting this option simulates the daemon/wallet performing a cache optimization where a locked tx output skips the cache and is inserted immediately into the tree if we can calculate the first spendable block index to be less than or equal than the specified FCMP++ activation block index. What this basically means is that up to ~60 blocks before the FCMP++ activation index, only non-standard unlock times very far in the future have to be held in the cache. If we choose to implement the behavior that is represented by the simulation of the fcmp-index option, we can trim the majority of the cache.

I will post results later.

@j-berman

[tobtoht]

I wanted to note that as currently configured, this binary will get included in release archives. Considering that it is mostly of use to developers, should it be excluded?

[tobtoht]

ARMv8 Android build failure needs an Android NDK bump (WIP here: #9456). To resolve the macOS build failure we will need to bump the minimum macOS target to 10.15. @selsta

This also affects #9514

[jeffro256]

Here's the results of running ./monero-blockchain-find-locked (and matplotlib) after extracting the total running balance of all locked outputs (including unlock_time=0 and coinbase unlock):

The max size of the cache at any specific block is 6522 as of 9ish hours ago. That's pretty manageable IMO. I will later post results from the chain simulating if we use the fcmp-index quick insert optimization.

[jeffro256]

Here's the same graph but excluding "standard" unlock times: unlock_time=0 and coinbase:

[jeffro256]

I wanted to note that as currently configured, this binary will get included in release archives. Considering that it is mostly of use to developers, should it be excluded?

Yeah I'm fine with keeping it out of release builds

[jeffro256]

Here's a graph of the cache size if we use the quick insert optimization, with a simulated FCMP++ activation height of 3500000:

Max cache size is only 57! This graph can also be interpreted as a running balance of all on-chain outputs which still haven't unlocked to this day or will unlock in the near future. Of course, this 57 value would go up to include all the recent standard locked outputs once the actual activation height approaches, but only 57 longstanding weirdly long locked outputs really isn't too bad to handle.

[j-berman]

Thank you for this :)

Does this end up producing reasonable / expected values? This seemed the most sane approach to me but I didn't test it. Some analysis of the results this function in particular is producing would be nice (focusing on block timestamp reasonability would be ok imo). Making sure this is correct is on the general TODO list for the fcmp++ integration

It's definitely wrong for any timestamp locked outputs created before 120s block times, so identifying the impact there would also be helpful (making sure all of those should all already be unlocked would be ok e.g.).

Overall likely shouldn't affect the results you highlighted, but relevant for shipping this code

[Zulu-1540]

Can fix this issue?
Stop from using this