Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Key Error using Natural Earth POV files #2319

Open
rcomer opened this issue Jan 29, 2024 · 3 comments · May be fixed by #2477
Open

Key Error using Natural Earth POV files #2319

rcomer opened this issue Jan 29, 2024 · 3 comments · May be fixed by #2477

Comments

@rcomer
Copy link
Member

rcomer commented Jan 29, 2024

Description

When trying to use Natural Earth's POV shapefiles, the unzipping step fails because they do not include a .cpg file. I wonder if we just need a try-except around the getinfo call. Having said that, out of the list

cartopy/lib/cartopy/io/shapereader.py

Line 342 in ec891b0

for ext in ['.shp', '.dbf', '.shx', '.prj', '.cpg']:

we only appear to use .shp and .shx in other parts of the library. So is there a reason we are unpacking all of them?

Code to reproduce

import cartopy
import cartopy.crs as ccrs
import matplotlib.pyplot as plt

mpov = cartopy.feature.NaturalEarthFeature('cultural', 'admin_0_countries_arg', '10m')

fig, ax = plt.subplots(subplot_kw={"projection": ccrs.PlateCarree()})
ax.add_feature(mpov, facecolor="none")

plt.show()

Traceback

On first run, when Cartopy is downloading and unzipping the files I get

[script_path]/natural_earth.py:3: UserWarning: Treat the new Tool classes introduced in v1.5 as experimental for now; the API and rcParam may change in future versions.
  import matplotlib.pyplot as plt
[git_clone_path]/cartopy/lib/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_cultural/ne_10m_admin_0_countries_arg.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)
Exception in Tkinter callback
Traceback (most recent call last):
  File "[conda_env_path]/cartopy-dev/lib/python3.12/tkinter/__init__.py", line 1962, in __call__
    return self.func(*args)
           ^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/tkinter/__init__.py", line 861, in callit
    func(*args)
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/_backend_tk.py", line 274, in idle_draw
    self.draw()
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/backend_tkagg.py", line 10, in draw
    super().draw()
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py", line 388, in draw
    self.figure.draw(self.renderer)
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 95, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/figure.py", line 3154, in draw
    mimage._draw_list_compositing_images(
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/mpl/geoaxes.py", line 524, in draw
    return super().draw(renderer=renderer, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/axes/_base.py", line 3070, in draw
    mimage._draw_list_compositing_images(
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/mpl/feature_artist.py", line 152, in draw
    geoms = self._feature.intersecting_geometries(extent)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 305, in intersecting_geometries
    return super().intersecting_geometries(extent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 108, in intersecting_geometries
    return (geom for geom in self.geometries() if
                             ^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 287, in geometries
    path = shapereader.natural_earth(resolution=self.scale,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/io/shapereader.py", line 306, in natural_earth
    return ne_downloader.path(format_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/io/__init__.py", line 203, in path
    result_path = self.acquire_resource(target_path, format_dict)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[git_clone_path]/cartopy/lib/cartopy/io/shapereader.py", line 364, in acquire_resource
    member = zfh.getinfo(member_path.replace('\\', '/'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[conda_env_path]/cartopy-dev/lib/python3.12/zipfile/__init__.py", line 1516, in getinfo
    raise KeyError(
KeyError: "There is no item named 'ne_10m_admin_0_countries_arg.cpg' in the archive"

On second run, the plot works fine.

Full environment definition

Operating system

RHEL7

Cartopy version

0.22 and main

conda list

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.16             pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.25.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cartopy                   0.22.1.dev89+g5e449397          pypi_0    pypi
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py312hf06ca03_0    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
cftime                    1.6.3           py312hc7c0aa3_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0           py312h8572e83_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
cython                    3.0.7           py312h30efb56_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
docutils                  0.20.1          py312h7900ff3_3    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_0    conda-forge
execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fonttools                 4.47.0          py312h98912ed_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
geos                      3.12.1               h59595ed_0    conda-forge
hdf4                      4.2.15               h2a13503_7    conda-forge
hdf5                      1.14.3          nompi_h4f84152_100    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
identify                  2.5.33             pyhd8ed1ab_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.20.0             pyh707e725_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py312h8572e83_1    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libblas                   3.9.0           20_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           20_linux64_openblas    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           20_linux64_openblas    conda-forge
libnetcdf                 4.9.2           nompi_h9612171_113    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.25          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.3               h232c23b_0    conda-forge
libxslt                   1.1.39               h76b75d6_0    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lxml                      5.1.0           py312h37b5203_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                2.1.3           py312h98912ed_1    conda-forge
matplotlib-base           3.8.2           py312h0e6d468_200    conda-forge/label/testing
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
netcdf4                   1.6.5           nompi_py312h26027e0_100    conda-forge
nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
numpy                     1.26.3          py312heda63a1_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openssl                   3.2.0                hd590300_1    conda-forge
owslib                    0.29.3             pyhd8ed1ab_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.4           py312hfb8ada1_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.2.0          py312hf3581a9_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
pre-commit                3.6.0              pyha770c72_0    conda-forge
proj                      9.3.1                h1d62c97_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydata-sphinx-theme       0.15.1             pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pykdtree                  1.3.10          py312hc7c0aa3_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.1           py312h38f1c37_5    conda-forge
pyshp                     2.3.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
pytest-mpl                0.16.1             pyhd8ed1ab_0    conda-forge
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge
python                    3.12.1          hab00c5b_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.4             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py312h98912ed_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruff                      0.1.11          py312h9118e91_0    conda-forge
scipy                     1.11.4          py312heda63a1_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.0.4              pyhd8ed1ab_0    conda-forge
setuptools_scm            8.0.4                hd8ed1ab_0    conda-forge
shapely                   2.0.2           py312h9e6bd2c_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sphinx                    7.2.6              pyhd8ed1ab_0    conda-forge
sphinx-gallery            0.15.0             pyhd8ed1ab_0    conda-forge
sphinxcontrib-applehelp   1.0.7              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.5              pyhd8ed1ab_0    conda-forge
sphinxcontrib-htmlhelp    2.0.4              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.6              pyhd8ed1ab_0    conda-forge
sphinxcontrib-serializinghtml 1.1.9              pyhd8ed1ab_0    conda-forge
sqlite                    3.44.2               h2c6b66d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
ukkonen                   1.0.1           py312h8572e83_4    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
virtualenv                20.25.0            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xarray                    2023.12.0          pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

pip list
@rcomer
Copy link
Member Author

rcomer commented Jan 29, 2024

OK, I see the .prj files were deliberately added to help geopandas: #1744, #1745, and .cpg came in with that.

@greglucas
Copy link
Contributor

Adding a try/except seems reasonable. I'm a bit confused as to why we don't just extract the members within the file using the namelist directly: https://docs.python.org/3/library/zipfile.html#zipfile.ZipFile.namelist
One could also check our current member list against that namelist and skip the ones that aren't present.

@nhsavage nhsavage mentioned this issue Feb 7, 2024
Open
@nhsavage
Copy link

nhsavage commented Feb 8, 2024 •
edited
Loading

I guess that the original idea of extracing these file explicitly is so that if the zipfile doesn't have the critical files it raises an error rather than carries on regardless.

should we split into "essential (.shp .dbx .shx)" and the rest. Then raise an error if the essential are not there and warn if any others are missing? cpg should certainly be optional based on:

Some info on extensions is here:
https://gisgeography.com/arcgis-shapefile-files-types-extensions/

CPG files are optional plain text files that describe the encoding applied to create the shapefile. If your shapefile doesn’t have a .cpg file, then it has the system default encoding.

@greglucas greglucas linked a pull request Nov 13, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

MNT: Extract all NE resources from the zip file greglucas/cartopy
3 participants
@nhsavage @rcomer @greglucas