High memory consumption using parquet encoder

[coutug]

Using the feature/parquet branch, I get really high memory usage from running this configuration:

substreams-sink-files run \
eos.substreams.pinax.network:443 \
https://github.com/pinax-network/substreams-antelope-tokens/releases/download/v1.0.0/antelope-tokens-v1.0.0.spkg \
map_events \
s3://pinax/eos?region=us-east-1 \
"2:" --encoder=parquet

This mainly happens running EOS blockchain specificaly when there is already data in the target s3 bucket. If the bucket is create by the process, it runs fine (using around 3Gi of RAM). But if the container gets killed and restarts, the memory consumption steadily starts to climb up, as shown in the image below.

[maoueh]

The bundle of a segment is all done in memory, so that is expected right now. I haven't implemented scratch space to put data on disk and limit memory for now.

Workaround if too bit currently is to reduce blocks per bundle and perform merging of "parquet bundle" off process.

[coutug]

Thank you for the quick answer. After further investigation, the problem was related to the cursor file being lost on container restart since I was not using a persistent volume. I am not sure why this use case was causing some sort of memory leak. I would have expect the process to simply start back at the start block.