Skip to main content

Limit Trie Logs for Bonsai

The early access feature --Xbonsai-limit-trie-logs-enabled minimizes the database size of Besu if you are using data-storage-format=BONSAI. When enabled, this feature can reduce database growth by more than 3 GB each week on mainnet.

Step by Step Guide​

caution

Limiting trie logs is an early access feature. Before executing the commands in this guide, ensure you review your node configuration.

  1. Update Besu config to add --Xbonsai-limit-trie-logs-enabled but don’t restart yet
  2. Stop Besu
  3. (optional) Run: sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune
  4. Start Besu (remembering to run sudo systemctl daemon-reload if you use a systemd service file as per CoinCashew and Somer)
  5. Look out for Limit trie logs enabled: retention: 512; prune window: 30000 in your Besu config printout during startup
  6. Enjoy more free space

How?​

If you want to use this feature before it is enabled by default, simply add this option to your Besu command: --Xbonsai-limit-trie-logs-enabled When you restart Besu it will begin pruning, block by block after an initial reduction in the database during each startup.

If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. Instead of resyncing, in order to do this we’re providing a “run once” offline command to immediately prune all your old trie logs in a few minutes or less. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will not need to run this command a second time if you keep --Xbonsai-limit-trie-logs-enabled.

For minimal downtime, we recommend running this command before restarting Besu with --Xbonsai-limit-trie-logs-enabled (easiest to do it all at the same time).

If you followed Somer Esat’s (https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-teku-f09ecd9ef2ee ) or CoinCashew’s guide (https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/step-3-installing-execution-client/besu) then you likely have these options set in your besu.service or execution.service systemd file:

...
ExecStart=/usr/local/bin/besu/bin/besu \
...
  --sync-mode=X_SNAP \
  --data-path="/var/lib/besu" \
  --data-storage-format=BONSAI \
...

So in order to prune the trie logs, your command should be something like:

sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune

The logs should look something like this:

2024-01-31 05:10:43.777+00:00 | main | INFO  | KeyPairUtil | Attempting to load public key from /data/besu/key
...
2024-01-31 05:10:47.574+00:00 | main | INFO  | TrieLogSubCommand | Calculating trie logs size before...
2024-01-31 05:10:51.100+00:00 | main | INFO  | TrieLogSubCommand | Trie logs size before: 8 GiB
2024-01-31 05:12:45.907+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)...
2024-01-31 05:12:45.916+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2024-01-31 05:12:46.036+00:00 | main | INFO  | TrieLogHelper | Clear trie logs...
2024-01-31 05:12:46.100+00:00 | main | INFO  | TrieLogHelper | Restoring trie logs retained from batch 1...
2024-01-31 05:12:46.168+00:00 | main | INFO  | TrieLogHelper | Key(0): 0xc0a1f26bf828ae6d9af8fda8af3126f12a6d986afea05fdc61b25d0d0ab3b8d4
...
2024-01-31 05:12:46.263+00:00 | main | INFO  | TrieLogHelper | Key(511): 0x5eb700164f4190b8806c7ce3de2f6e5b4f4c05a61bba4058f1b0a5ef63ec73af
2024-01-31 05:12:46.359+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2024-01-31 05:12:46.363+00:00 | main | INFO  | TrieLogSubCommand | Finished pruning - recalculating size...
2024-01-31 05:12:51.658+00:00 | main | INFO  | TrieLogSubCommand | Trie logs size after: 0 B
Prune ran successfully. We estimate you freed up 8 GiB!

If you use a toml config file, then you can simply do something like:

sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune

Troubleshoot​

The prune command should look something like this for mainnet users: sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune

and Besu should be shutdown before running it.

java.lang.IllegalArgumentException: Subcommand only works with data-storage-format=BONSAI

Are you missing --data-storage-format=BONSAI? It must be add before the storage subcommand, i.e.

sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune

java.lang.RuntimeException: Column handle not found for segment TRIE_BRANCH_STORAGE

Did you specify the data-path?

sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune

... | INFO | RocksDBKeyValueStorageFactory | No existing database detected at /tmp/besu. Using version 2 ... java.lang.IllegalArgumentException: Trying to retain more trie logs than chain length (0), skipping pruning

Did you specify the correct data-path for your node?

sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune

java.lang.IllegalArgumentException: Cannot store generated private key.

Did you specify the correct data-path for your node?

sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune

... | INFO | KeyPairUtil | Attempting to load public key from /data/besu/key java.lang.IllegalArgumentException: Supplied file does not contain valid keyPair pair.

Check your file permission, you made need to run the command as sudo: sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu storage --sync-mode=X_SNAP x-trie-log prune

java.lang.RuntimeException: Column handle not found for segment WORLD_STATE

Are you using data-storage-format=FOREST instead of data-storage-format=BONSAI on an existing bonsai database?

org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While lock file: /data/besu/database/LOCK: Resource temporarily unavailable

Is Besu already running? You need to shut the Besu client down before running this subcommand.

java.lang.IllegalStateException: Unable to change the sync mode when snap sync is incomplete, please restart with snap sync mode

Have you specified the sync-mode? Default is sync-mode=FAST. Most mainnet users use X_SNAP or X_CHECKPOINT

sudo /usr/local/bin/besu/bin/besu --sync-mode=X_SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune

java.lang.RuntimeException: No finalized block present, can't safely run trie log prune

If your node is relatively new or recently resynced, you might not need to run this command.

org.hyperledger.besu.util.InvalidConfigurationException: Supplied genesis block does not match chain data stored in /data/besu.

Are you running this command for a network other than mainnet? You need to specify the network…

sudo /usr/local/bin/besu/bin/besu --network=holesky --sync-mode=X_SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune

The Details​

Too much detail for users but if you’re interested: