Test using Nessie instead of the Hive Metastore for Iceberg

[ravenac95]

What is it?

The Hive Metastore seems to randomly block trino operations (perhaps it needs to be scaled up) but it's already at a significant price point. I'd be curious to try something like Nessie and then also deploying a self managed hive metastore just for the interaction between the gcs bucket (import/export) into trino.

Since we don't need these things one 100% of the time we can shut them down when operations aren't using these features. This could save us a lot if it works.

SQLMesh makes mention it doesn't support Nessie as a metadata storage for iceberg + trino but they mention that's due to it not supporting views (which as of 2 months ago it supports).

[ravenac95]

Some notes:

• Ray and I also talked about potentially using the serverless biglake metastore.
  • I did some reading into it. We'd have to make some changes to our setup in order for it to work. Biglake only works with dataproc so we would need to stop deploying our own trino. I haven't look at how we might get all of that as IaC. Due to flux, I really like keeping things on the kubernetes side (I know... I am shocked too) because managing things with flux is really clean (I will soon have an experimental branch with a kind + flux test setup).