Improve performance of MosaicJSON tile requests at low zooms

[erhhung]

I'm trying to get ideas on how to improve the performance of tile requests for a large MosaicJSON config file (37 MB .json; 842 KB .json.gz) with ~150K quadkey tiles. I generated the config file using the cogeo-mosaic CLI with ~950 COG URLs worth over 75 TBs in total (continental US coverage).

I initially tried serving COG files located on a large NFS storage appliance directly from TiTiler to a MapProxy server for caching, where the MapProxy config file contained ~950 sources with URLs like: https://titiler.example.com/cog/tiles/%(z)s/%(x)s/%(y)s.%(format)s?url=file:///path/to/cog/file.tif. The performance was quite decent, even when MapProxy saw all cache misses:

Here are some average response times for a tile at different zooms. For each tile, MapProxy would be making multiple source requests to TiTiler, and then merging the pieces together.

• z=6 <1s
• z=5 ~3s
• z=4 ~6s
• z=2 ~8s
• z=1 ~50s

Then I decided to try MosaicJSON by moving the exact same set of ~950 MapProxy sources into the MosaicJSON config, thereby reducing the MapProxy config to a single source with a URL like: https://titiler.example.com/mosaicjson/tiles/%(z)s/%(x)s/%(y)s.%(format)s?url=file:///path/to/mosaic.json.gz.

Although, this time, again without cache hits, the tile requests are timing out left and right, especially at zooms lower than 6:

• z=10 ~4-10s
• z=8 ~8-16s
• z=7 ~35-60s
• z=6 timeout (>2m)

Around the same time, I see in the TiTiler logs plenty of [CRITICAL] WORKER TIMEOUT errors followed by [INFO] Started server process. I'm running TiTiler in a cluster of up to 10 pods, each getting up to 64 GB and 16 CPUs, so they're definitely no slouch. I see most pods are currently using >90% of their 64 GB limit, but there's definitely also heavy CPU usage during MosaicJSON handling. Even when processing stops—mostly by failure—and CPU usage settles, the pods still show >80% of their memory limit.

I understand that I've shifted all of the source merging burden from MapProxy onto TiTiler, but I don't understand why the performance turned so bad as it should be reading the same number of underlying COGs. I also have way more pods and resources allocated to TiTiler than I have to MapProxy.

[vincentsarago]

@erhhung

First, this is a huge mosaic and using mosaicJSON in a static file (.json) might not be the best here. For this kind of mosaic I usually use DynamoDB storage (sadly it's not well documented in detail).

I'm not sure how are your COG are but usually if they are small there overviews won't cover the low zoom level and thus you might have to open a lot of files and/or use a lot of memory. Also rio-tiler (which is what titiler use to read the tile) is not optimized to read low zoom level off file covering high zoom level (cogeotiff/rio-tiler#348).

About performance, you can also try running without using multi thread
https://github.com/developmentseed/titiler/blob/master/titiler/endpoints/factory.py#L1097

MOSAIC_CONCURRENCY=1
MAX_THREADS=1

You can also set TITILER_API_DEBUG=True to get more info in the tile response headers.

If your data is openly available I could have a deeper look.

[erhhung]

Hi @vincentsarago,

Thanks for the recommendations. I'll turn on the debug flag to see if there's more logging on the backend as well.

I understand that I'm working with a mosaic that's larger than most, and, unfortunately, it's from licensed commercial aerial imagery, so I'm unable to share. But I want to understand where the bottleneck might be compared to the /cog/tiles use case.

If I hit /mosaicjson/tiles at z >= 12, the response time is generally sub-2s, meaning the overhead of reading the gzipped config file isn't too horrible, even if it can be improved via DB queries. So let's assume that having a large config file isn't the real bottleneck.

If a client request comes in at z=1, we can be sure that TiTiler is going to have to access hundreds of COG files regardless if it's MapProxy making lots of concurrent requests to a cluster of TiTiler pods at /cog/tiles, each then asking rio-tiler to read a single COG to get an overview at z=1, or it's MapProxy making a single request to a single TiTiler pod at /mosaicjson/tiles, which then has to concurrently—in a thread-limited pool—read that many COGs using rio-tiler from the same NFS storage.

From my observations, I would have to assume either the COG access is just slow and/or concurrency is low, so that it just takes a long time to read all of the hundreds of COGs from a single TiTiler instance, or the mosaic merging is not efficient (compared to MapProxy merging the same number of pieces). I tend to think the bottleneck is the former since MapProxy is likely using similar Python-based libs to merge hundreds of small tiles, and should be pretty fast.

So, unless a single TiTiler pod receiving an "expensive" /mosaicjson/tiles request (based on the number of COGs it has to read?) can solicit the aid of other TiTiler pods in the cluster—like in the /cog/tiles use case—then it's hard to handle MosaicJSON scalability because of zoom-induced variability in the underlying COG accesses.

[erhhung]

If I understand what you're saying in cogeotiff/rio-tiler#348 (and I'm not yet familiar with the internal file structure of a COG), it's not that the COG doesn't contain overview images at low zooms, but rio-tiler is choosing not to read those, but choosing to read many more higher-res images from higher zooms? I could certainly see how that inefficiency multiplied by a large number of COGs could spell doom for a MosaicJSON request.

[vincentsarago]

there are 2 problems here

1. low zoom mosaic / quadkeys

When you create your mosaic, the quadkeys zoom is defines by default to match the min zoom of the COG.

In ☝️the mosaic quadkey zoom is 8, and thus the recommended zoom range for the mosaic is 8 to 13. Now when you try to read lower zoom, the mosaic backend will tell rio-tiler to create a tile from all the files which are the children of the input tile.

2. rio-tiler reads

As I mentioned, rio-tiler wasn't designed to read files at low resolution (outside the overview range). As shown in ☝️ if a COG as a native mercator resolution of 20m, it means the max_zoom is 13, this same file has 5 overviews which means the min overview as a resolution of (20 x 2**5)=640m which is around zoom 8.
Passed zoom 8, any read to this tiff will be really inefficient.

Why:
a COG is tiled usually using 256 or 512 internal tile, reading a map tile passed the native min zoom of the COG will mean that you'll at least fetch on of its internal tile but only use few pixel, which result in a lot of data transfert for nothing.

This all comes down to cogeotiff/rio-tiler#348, because as I mentioned, rio-tiler was designed to use COG internal overviews in the most efficient way. I'm sure there are ways to fix this but sadly I don't have time to work on it right now.

[caseyslaught]

Thanks for great discussion you two. Have there been any developments on this front? Or any workarounds that you've figured out?