Very large input data files in REMIND

[mikapfl]

Hi,

in REMIND, modules/35_transport/edge_esm/input/ is very large (1.1 GB) after input data is downloaded. That is somewhat wasteful of storage resources on the cluster and probably also pretty inefficient when reading the data.
As far as I understand, only a small handful of rather small files is read into GAMS code using $include, while most files (and in particular, the really big ones like pref.cs4r) are either never read or read in R. Only for reading into GAMS, the uncompressed cs4r file format is really necessary, for reading into R, the compressed binary format .mz would be more appropriate. Maybe you can check if you can use that to reduce the size of the input data? In my testing, this would reduce the input data size from 1.1 GB to something like 60 MB, which would be fantastic.

Cheers

Mika

[orichters]

@johannah-pik, @jmuessel: I also see these large files in the output folders and wonder if I really need the data for all gdp and EDGE-T scenarios in pref.cs4r when this folder clearly contains only a specific one.

Any progress planned on this? Maybe you could ask RSE to help with it? Thanks!