Problem with cube-gen: NetCDF: Unknown file format

[georgeblck]

Hello,

when trying to create a new Cube with the cube-gen tool, I am running into the following error.

(esdc) [user@cluster]$ cube-gen mycube burnt_area:dir=/data/cablab-datacube-0.2.4/low-res/data/burnt_area -c cube-low-res.config       
CAB-LAB command-line interface, version 0.2.2
Traceback (most recent call last):
  File "/home/user/.conda/envs/esdc/bin/cube-gen", line 11, in <module>
    load_entry_point('cablab-core', 'console_scripts', 'cube-gen')()
  File "/gpfs0/home/user/cablab-core/cablab/cube_gen.py", line 137, in main
    cube.update(source_provider)
  File "/gpfs0/home/user/cablab-core/cablab/cube.py", line 118, in update
    provider.prepare()
  File "/gpfs0/home/user/cablab-core/cablab/cube_provider.py", line 149, in prepare
    self._source_time_ranges = self.compute_source_time_ranges()
  File "/gpfs0/home/user/cablab-core/cablab/providers/burnt_area.py", line 43, in compute_source_time_ranges
    dataset = self.dataset_cache.get_dataset(file)
  File "/gpfs0/home/user/cablab-core/cablab/util.py", line 159, in get_dataset
    dataset = self.open_dataset(real_file)
  File "/gpfs0/home/user/cablab-core/cablab/util.py", line 205, in open_dataset
    return netCDF4.Dataset(real_file)
  File "netCDF4/_netCDF4.pyx", line 1795, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:12271)
RuntimeError: NetCDF: Unknown file format

Thanks in advance for any help!

[hans-permana]

hi @georgeblck ,

In order to troubleshoot your issue, could you please give me an overview on what you are trying to achieve by running this cube-gen? Also, what kind of input dataset did you use? If you are interested in the cube data itself, we could provide you an access to our FTP server, which has all of already generated cubes.

[georgeblck]

Hi @hans-permana ,

my goal is to create a cube based on some new data sets with a higher spatial resolution but a much lower temporal resolution. I am trying to understand the cablab-package and the providers so that I can extend them for the new data sets.

So I was playing around with the cube-gen tool, trying to replicate a usage example from your cube_gen.py file:

cube-gen "cablab-datacube/low-res" "burnt_area:dir=cablab-source/BurntArea"

which I extended by the -c option so that it looks like this:

cube-gen mycube burnt_area:dir=/path/to/netcdfs/burnt_area -c cube-low-res.config

mycube is the name/directory/target of the cube and the burnt_area netcdf-files are from your FTP server.
Does this help?

[hans-permana]

Thank you for a more detailed information, it has helped me to understand and hence to troubleshoot what your problem is.

It seems to me that there are a couple of issues here. First, the NetCDF error seems to have been caused by incompatibility between the writer and the reader. Could you do a conda list and paste the results here? I found it strange because I just tried it using the esdc environment created by using the environment.yml file and I could not replicate this issue.

Anyway, if you manage to solve this issue, there is another issue waiting. Each provider was created based on the source dataset. In the case of Burnt Area, it is the ones from GFED. So if we run the cube-gen with burnt_area provider, it will not generate anything. It simply does not understand about the different data structure. For this, we can create a provider that can re-sample the generated cube data. Another alternative is, of course, to run the same command on BurntArea source data.

[georgeblck]

Oh right, now I think I understand. The data on your FTP server are not the actual source data?
So the providers are not for reading the data sets when using them inside Python/Julia but for transforming them into cube conform data sets?

Here is the output of cube list:

# packages in environment at /home/user/.conda/envs/esdc:
#
backports                 1.0                      py35_1    conda-forge
backports.functools_lru_cache 1.4                      py35_1    conda-forge
bokeh                     0.12.9                   py35_0    conda-forge
bottleneck                1.2.1                    py35_1    conda-forge
ca-certificates           2017.7.27.1                   0    conda-forge
certifi                   2017.7.27.1              py35_0    conda-forge
click                     6.7                      py35_0    conda-forge
cloudpickle               0.4.0                    py35_0    conda-forge
coverage                  4.4.1                    py35_0    conda-forge
curl                      7.45.0                        0  
cycler                    0.10.0                   py35_0    conda-forge
cyordereddict             1.0.0                    py35_1    conda-forge
dask                      0.14.1                   py35_0    conda-forge
dask-core                 0.15.4                     py_0    conda-forge
dbus                      1.10.22                       0    conda-forge
decorator                 4.1.2                    py35_0    conda-forge
distributed               1.19.3                   py35_0    conda-forge
expat                     2.2.1                         0    conda-forge
fontconfig                2.12.1                        4    conda-forge
freetype                  2.7                           1    conda-forge
gettext                   0.19.7                        1    conda-forge
glib                      2.53.5                        0    conda-forge
gridtools                 0.3.0                    py35_0    forman
gst-plugins-base          1.8.0                         0    conda-forge
gstreamer                 1.8.0                         1    conda-forge
h5netcdf                  0.2.2                    py35_0    conda-forge
h5py                      2.6.0               np111py35_0  
hdf5                      1.8.15.1                      3  
heapdict                  1.0.0                    py35_0    conda-forge
icu                       58.1                          1    conda-forge
jinja2                    2.9.6                    py35_0    conda-forge
jpeg                      9b                            1    conda-forge
libffi                    3.2.1                         3    conda-forge
libgfortran               3.0.0                         1  
libiconv                  1.14                          4    conda-forge
libnetcdf                 4.3.3.1                       3  
libpng                    1.6.28                        1    conda-forge
libtiff                   4.0.6                         5    conda-forge
libxcb                    1.12                          1    conda-forge
libxml2                   2.9.3                         6    conda-forge
llvmlite                  0.13.0                   py35_0  
locket                    0.2.0                    py35_1    conda-forge
markupsafe                1.0                      py35_0    conda-forge
matplotlib                2.1.0                    py35_0    conda-forge
mkl                       2017.0.3                      0  
msgpack-python            0.4.8                    py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
netcdf4                   1.2.2               np111py35_0  
networkx                  2.0                      py35_0    conda-forge
numba                     0.28.1              np111py35_0  
numpy                     1.11.3                   py35_0  
olefile                   0.44                     py35_0    conda-forge
openssl                   1.0.2l                        0    conda-forge
pandas                    0.19.0              np111py35_0    conda-forge
partd                     0.3.8                    py35_0    conda-forge
pcre                      8.39                          0    conda-forge
pillow                    4.3.0                    py35_0    conda-forge
pip                       9.0.1                    py35_0    conda-forge
psutil                    5.4.0                    py35_0    conda-forge
py                        1.4.34                   py35_0    conda-forge
pyparsing                 2.2.0                    py35_0    conda-forge
pyqt                      5.6.0                    py35_4    conda-forge
pytest                    3.2.3                    py35_0    conda-forge
pytest-cov                2.5.1                    py35_0    conda-forge
python                    3.5.2                         0    conda-forge
python-dateutil           2.5.3                    py35_0    conda-forge
pytz                      2016.6.1                 py35_0    conda-forge
pyyaml                    3.12                     py35_1    conda-forge
qt                        5.6.2                         3    conda-forge
readline                  6.2                           0    conda-forge
scikit-image              0.12.3              np111py35_0    conda-forge
scipy                     0.19.0              np111py35_0  
setuptools                27.2.0                   py35_0    conda-forge
sip                       4.18                     py35_1    conda-forge
six                       1.10.0                   py35_0    conda-forge
sortedcontainers          1.5.7                    py35_0    conda-forge
sqlite                    3.9.2                         0    conda-forge
tblib                     1.3.2                    py35_0    conda-forge
tk                        8.5.19                        2    conda-forge
toolz                     0.8.2                     <pip>
toolz                     0.8.2                      py_2    conda-forge
tornado                   4.5.2                    py35_0    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
xarray                    0.8.2                    py35_0    conda-forge
xorg-libxau               1.0.8                         3    conda-forge
xorg-libxdmcp             1.1.2                         3    conda-forge
xz                        5.0.5                         1    conda-forge
yaml                      0.1.6                         0    conda-forge
zict                      0.1.3                      py_0    conda-forge
zlib                      1.2.8                         3    conda-forge

[hans-permana]

The data on your FTP server are not the actual source data?

Correct, although we also have an FTP server for the source data. However, the availability to these data is at the moment unclear to me, due to some licensing. Maybe @meggart can say more about this.

So the providers are not for reading the data sets when using them inside Python/Julia but for transforming them into cube conform data sets?

Also correct. Here is a link to more information about providers http://cablab.readthedocs.io/en/latest/esdc_prod.html#writing-a-new-provider

If you want to read the generated data, what you need is Cube Access API

Have fun and let me know if you need some more helps!

[hans-permana]

About the error, I checked your conda list and it looks ok to me. However I noticed that you don't have cablab-core installed in environment, so it is possible that your cablab-core uses dependencies with different versions, especially the netcdf4 (and its corresponding C-library). Could you try installing cablab-core inside the environment?

[georgeblck]

I installed the cablab-core in the environment; my conda list now looks like this:

# packages in environment at /home/user/.conda/envs/esdc:
#
backports                 1.0                      py35_1    conda-forge
backports.functools_lru_cache 1.4                      py35_1    conda-forge
bokeh                     0.12.9                   py35_0    conda-forge
bottleneck                1.2.1                    py35_1    conda-forge
ca-certificates           2017.7.27.1                   0    conda-forge
cablab-core               0.2.2                     <pip>
certifi                   2017.7.27.1              py35_0    conda-forge
click                     6.7                      py35_0    conda-forge
cloudpickle               0.4.0                    py35_0    conda-forge
coverage                  4.4.1                    py35_0    conda-forge
curl                      7.45.0                        0  
cycler                    0.10.0                   py35_0    conda-forge
cyordereddict             1.0.0                    py35_1    conda-forge
dask                      0.14.1                   py35_0    conda-forge
dask-core                 0.15.4                     py_0    conda-forge
dbus                      1.10.22                       0    conda-forge
decorator                 4.1.2                    py35_0    conda-forge
distributed               1.19.3                   py35_0    conda-forge
expat                     2.2.1                         0    conda-forge
fontconfig                2.12.1                        4    conda-forge
freetype                  2.7                           1    conda-forge
gettext                   0.19.7                        1    conda-forge
glib                      2.53.5                        0    conda-forge
gridtools                 0.3.0                    py35_0    forman
gst-plugins-base          1.8.0                         0    conda-forge
gstreamer                 1.8.0                         1    conda-forge
h5netcdf                  0.2.2                    py35_0    conda-forge
h5py                      2.6.0               np111py35_0  
hdf5                      1.8.15.1                      3  
heapdict                  1.0.0                    py35_0    conda-forge
icu                       58.1                          1    conda-forge
jinja2                    2.9.6                    py35_0    conda-forge
jpeg                      9b                            1    conda-forge
libffi                    3.2.1                         3    conda-forge
libgfortran               3.0.0                         1  
libiconv                  1.14                          4    conda-forge
libnetcdf                 4.3.3.1                       3  
libpng                    1.6.28                        1    conda-forge
libtiff                   4.0.6                         5    conda-forge
libxcb                    1.12                          1    conda-forge
libxml2                   2.9.3                         6    conda-forge
llvmlite                  0.13.0                   py35_0  
locket                    0.2.0                    py35_1    conda-forge
markupsafe                1.0                      py35_0    conda-forge
matplotlib                2.1.0                    py35_0    conda-forge
mkl                       2017.0.3                      0  
msgpack-python            0.4.8                    py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
netcdf4                   1.2.2               np111py35_0  
networkx                  2.0                      py35_0    conda-forge
numba                     0.28.1              np111py35_0  
numpy                     1.11.3                   py35_0  
olefile                   0.44                     py35_0    conda-forge
openssl                   1.0.2l                        0    conda-forge
pandas                    0.19.0              np111py35_0    conda-forge
partd                     0.3.8                    py35_0    conda-forge
pcre                      8.39                          0    conda-forge
pillow                    4.3.0                    py35_0    conda-forge
pip                       9.0.1                    py35_0    conda-forge
psutil                    5.4.0                    py35_0    conda-forge
py                        1.4.34                   py35_0    conda-forge
pyparsing                 2.2.0                    py35_0    conda-forge
pyqt                      5.6.0                    py35_4    conda-forge
pytest                    3.2.3                    py35_0    conda-forge
pytest-cov                2.5.1                    py35_0    conda-forge
python                    3.5.2                         0    conda-forge
python-dateutil           2.5.3                    py35_0    conda-forge
pytz                      2016.6.1                 py35_0    conda-forge
pyyaml                    3.12                     py35_1    conda-forge
qt                        5.6.2                         3    conda-forge
readline                  6.2                           0    conda-forge
scikit-image              0.12.3              np111py35_0    conda-forge
scipy                     0.19.0              np111py35_0  
setuptools                27.2.0                   py35_0    conda-forge
sip                       4.18                     py35_1    conda-forge
six                       1.10.0                   py35_0    conda-forge
sortedcontainers          1.5.7                    py35_0    conda-forge
sqlite                    3.9.2                         0    conda-forge
tblib                     1.3.2                    py35_0    conda-forge
tk                        8.5.19                        2    conda-forge
toolz                     0.8.2                     <pip>
toolz                     0.8.2                      py_2    conda-forge
tornado                   4.5.2                    py35_0    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
xarray                    0.8.2                    py35_0    conda-forge
xorg-libxau               1.0.8                         3    conda-forge
xorg-libxdmcp             1.1.2                         3    conda-forge
xz                        5.0.5                         1    conda-forge
yaml                      0.1.6                         0    conda-forge
zict                      0.1.3                      py_0    conda-forge
zlib                      1.2.8                         3    conda-forge

But the error is still the same. I also tried to do the same thing on a nother System and the same error came up.