Issue: High Disk Space Usage and Ledger Ingestion Problems in Stellar Captive Core, Horizon after update to 22 version

[racconozaur]

Problem Description

After updating our Stellar Captive Core node, we noticed a significant increase in disk usage. Before the update, the node successfully ingested 22 million ledgers without issues. However, following the update, only 1 million ledgers are consuming approximately 2 TB of disk space.

We need help identifying the cause of this issue and suggestions for resolving or optimizing storage usage.

System Configuration

OS: Ubuntu 23.10 (GNU/Linux 6.5.0-44-generic x86_64)
Disk Space: 3 TB

PostgreSQL Configuration

Version: 15.7 (Ubuntu 15.7-0ubuntu0.23.10.1)

Stellar Core Configuration

Package Version: 22.0.0
Git Version: 1cd8b8dca9aeeca9ce45b129cd923992b32dc258
Ledger Protocol Version: 22
Pre-release Version: 0

rs-stellar-xdr:

Package Version: 22.0.0
Git Version: 715003372ea6380044b5a4a02907ff73e56ae9e7
Base XDR Git Version: 529d5176f24c73eeccfa5eba481d4e89c19b1181

Horizon Configuration

Version: 22.0.2
Git Version: 5df7099e675469f409ad1b8ef5bb2a8a19db7f00
Go Version: go1.22.10

Observations

1. The disk usage increased drastically after the update, even though only a fraction of the previously ingested ledgers is stored.
2. No changes were made to PostgreSQL configurations during the update process.
3. Logs do not indicate errors, but disk utilization continues to grow at an abnormal rate.