Automatic Cache Refresh with iceberg.metadata-cache.enabled in Trino

[byunks]

Hello Trino Community,

I have a question regarding the metadata caching functionality in Iceberg. I am planning to use Trino with the iceberg.metadata-cache.enabled option set to true, which I understand helps to cache Iceberg table metadata and improve query performance. However, I am curious about how I can ensure that the cache is automatically refreshed when the underlying Iceberg table metadata changes.

Specifically:

1. Is there an automatic mechanism in Trino that invalidates or refreshes the metadata cache when changes occur in the Iceberg table schema or data?
2. If not, are there any recommended methods or configurations to manually control or trigger cache invalidation at set intervals (such as a TTL-like feature)?
3. If manual cache invalidation is required, what are the best practices for ensuring that queries always reflect the latest metadata without needing to restart the Trino server?

Any guidance or suggestions would be greatly appreciated. Thank you!

Best regards,
byunks

[raunaqmorarka]

This cache is based on iceberg metadata files which are guaranteed to be immutable by the iceberg spec.
So there is no real need for a refresh, as we never deal with stale metadata.
There are some configurables available at io.trino.filesystem.memory.MemoryFileSystemCacheConfig (undocumented at the moment) which can control TTL, max size and max content length for this cache.
cc: @mosabua

[mosabua]

#23704

[byunks]

@raunaqmorarka

Hello,

Regarding the response, you mentioned that the cache is based on Iceberg metadata files, which are guaranteed to be immutable by the Iceberg spec.

I would like to understand this in more detail. When there are changes to the Iceberg table spec (such as adding or modifying columns, or changing sort, Z-order, partition, etc.), new metadata files are created. Could you clarify what is meant by "immutable" in this context when the spec of the Iceberg table can change and generate new metadata files?

Thank you for your explanation.

[raunaqmorarka]

What I meant is that there are no in-place updates to iceberg metadata files, we only add new metadata or delete old metadata that is no longer referenced. So the cache will fetch new metadata as needed and never serve stale metadata. This makes it unnecessary to manually refresh or invalidate the cache.

[hashhar]

I think the confusion probably comes because you think cache is structured like:

Table 1 -> all metadata of the table

Instead it's structured at the file level like:

table1_manifest -> cached
table1_snapshot1 -> cached
table1_snapshot2 -> cached
table1_stats_file1 -> cached
table1_stats_file2 -> cached

[laithalzyoud]

@raunaqmorarka @hashhar Actually this might be needed in this context, we will need at least a manual way to invalidate the cache for a certain table if the existing metadata files are rewritten or modified in-place. We just ran through an issue where Trino complains about metadata not being valid (size diff) after rewriting a table's metadata absolute paths. To resolve it, we disabled metadata caching in our environment, but it will be really helpful to have manual invalidation in the future so we can enable caching again

[raunaqmorarka]

There's a JMX endpoint that can be used to invalidate the cache io.trino.filesystem.memory.MemoryFileSystemCache#flushCache.

[byunks]

@raunaqmorarka @hashhar
io.trino.filesystem.memory.MemoryFileSystemCache#flushCache
Does the above JMX endpoint appear to reset not only Iceberg's metadata but also other filesystem caches?

[raunaqmorarka]

It should only flush a specific iceberg catalog's cached metadata

[byunks]

@raunaqmorarka @hashhar

Question: As the number of tables grows, the size of Iceberg metadata also increases, which in turn affects the size of the metadata cache on the coordinator. Could you explain Trino's policy or strategy for managing Iceberg metadata cache under such conditions? Specifically:

How does Trino ensure that the coordinator doesn't run out of memory due to large metadata cache usage?
Is there a memory limit for the Iceberg metadata cache, and how does Trino decide when to evict cache entries?
Are there any best practices or recommendations for tuning cache settings when working with a large number of Iceberg tables?

Additionally, when iceberg.metadata-cache.enabled is set to true, fs.cache.enabled must be set to false. In this case, does that mean the io.trino.filesystem.memory.MemoryFileSystemCacheConfig is not used at all? How does Trino handle cache configuration in this scenario?

Thank you!

[raunaqmorarka]

See the configurables at io.trino.filesystem.memory.MemoryFileSystemCacheConfig for limits on memory usage of the cache.
It's an LRU cache, eviction policy is to remove "least recently used" entries when the cache reaches the configured maximum size.
You can tune fs.memory-cache.max-size higher if you need to for a large number of tables, if you have sufficient coordinator memory.
fs.cache.enabled handles caching of metadata on local disks in coordinator. Since it can handle caching far more data than the memory cache can, we turn the memory cache off when fs.cache.enabled is true.

[byunks]

@raunaqmorarka
I'm curious about how the cache is managed when fs.cache.enabled = false

For iceberg.metadata-cache.enabled = true and fs.cache.enabled = false, I am curious about how metadata cache management is configured.

For iceberg.metadata-cache.enabled = true and fs.cache.enabled = true, I understand that the Iceberg metadata cache is managed through the file system cache and can be controlled by options like fs.cache.max-sizes. Is that correct?

[raunaqmorarka]

When fs.cache.enabled is true, the value of iceberg.metadata-cache.enabled does not matter, we will always cache iceberg metadata.
When fs.cache.enabled is false and iceberg.metadata-cache.enabled is true (the default), configuration is used from io.trino.filesystem.memory.MemoryFileSystemCacheConfig

[byunks]

@raunaqmorarka
Thank you for your response, and I understood what you explained. I will close the discussion.

[mosabua]

fyi @sheajamba

[byunks]

@raunaqmorarka
I'm sorry, I have one more question
When is the Iceberg metadata cache

1. When the table is selected (queried)?
2. When Trino periodically refreshes?

Which of these two is correct?
And if it's option 1, does the fs.memory-cache.ttl time apply after the cache is created?

[raunaqmorarka]

It is 1, there is no background refresh of the cache.
fs.memory-cache.ttl time starts from the moment an entry is saved in the cache

[byunks]

@raunaqmorarka
Thank you!!