Skip to content

Commit

Permalink
Solidify the Dataset class and adapt working with Multi-band rasters (#…
Browse files Browse the repository at this point in the history
…45)

* rename examples notebook

* deprecate the path parameter from the to_polygon method

* rename to_polygon to cluster2, the column name in the returned gdf has the band name

* arrage data

* rename to_geo_dataframe to to_feature_collection

* add missing data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* arrange test data

* update

* arrange tests data

* arrange test data

* change the era5 no data value to -9999

* add test case for the error in the to_dataset bug

* add layer_count property to featurecollection

* add layer_names property to featurecollection class

* change the return value of the _gdf_to_ds method to return Featurecollection not DataSource

* add test_gdf_to_ds_if_feature_is_already_ds test

* add columns property to feature collection class

* read folder using regex instead of renaming files

* python 3.10

* fix file path

* add docs for Datacube.read_separate_files

* add repre method, files property, update docs

* rename method read_separate_files to read_multiple_files

* rename data attribute in the datacube class to values

* rename data attribute in the datacube class to values

* rename the TestOpenDataCube test

* add read_multiple_files documentations

* update docs

* add datacube rst file to the index

* correct index datacube name in the index

* add docs missing images

* replace the convert_longitude method content to use gdal.warp

* add comments in the convert_longitude method

* update convert_longitude example files

* refactor

* change bounds to total_bounds

* get extent of datasource

* change the _gdf_to_ds implementation

* add _crop_with_polygon_warp method

* change the inputs for the read_multiple_files to take a regex string

* add docs to checklist

* test read files based on number in the file name

* update

* refactor the read_multiple_files method

* the start and end parameter in the read_multiple_files can take integer values not only dates

* allow values.setter to set values even if the files were not read yet

* path parameter to the to_file methof can be a list of paths

* deprecate update_Datacube and replace it with a setter for the values property

* refactor

* remove repaeted readthedocs badge

* update detailed scheme of datacube docs

* return the Array object from the plot method

* add datacube example

* add file_name to get the file name of an ogr.Datasource and gdal.Dataset, change the ogr.Datasource to a gdal.Dataset in the _gdf_to_ds

* add a gdal_dataset parameter to the _gdf_to_ds method

* add ogr_ds_togdal_dataset method

* refactor

* change the _ds_to_gdf to use VectorTranslate

* correct the name of the MEMORY_FILE

* print the gdal_ds for debugging

* add gdal.UseExceptions() in the featurecollection module

* change the layername to be extracted from the gdal_ds object

* remove the driver type in the gpd.read_file in _ds_to_gdf

* add layer name to gpd.read_file from memory

* remove the event branch to run on all branches

* correct name

* add pypi to the workflow name

* add ubuntu conda workflow

* install the dev packages using pip

* add conda optional packages

* remove installing cleopatra in the workflow file as it is in the requirement.txt/environment.yml file

* replace conda with the market place conda

* activate test env instead of the base env

* remove mamba

* trigger ci on push only

* import cleopatra inside the test as it raise an import error in case testing the main package-cleopatra not installed

* include python 3.11 in the yml file

* remove upper limit of python version

* remote test for python 3.11 as depencdencies still have the <3.11 in their yml file

* remove testing conda for python 3.9

* remove the python setup.py install as it reinstall packages with pip in conda test

* add shell specification line

* clean files

* gather workflow of conda and pypi

* remove redundent workflows

* add two cases to ds_to_gdf in memory and io

* refactor _ds_to_gdf

* the to_dataset uses the _gdf_to_ds to convert the gdf to a gdal dataset instead of writting the the vector to disk

* add pivot point attribute to the feature collection class

* change the gdal raster.RasterCount in the get_band_names method to use the method band_count

* add set_band_names method

* add band_name setter

* column_name can take a one column, multiple or all, multi-band raster is enabled

* remove writting the mask vector to disk as it is not used anymore

* comment and refactor to_dataset method

* add missing test data

* update pre-commit hooks

* migrate to shapely2.x

* update dependencies

* update gdal to 3.7.0

* rename _create_driver_from_scratch to create_driver_from_scratch

* add function to convert ogr datatype code into numpy data type

* update python version for conda build in actions to 3.11

* correct the ci to run plotiing tests in the optional package scenario

* _read_variable does not raise an error if the variable does not exist but return None

* add dtypes property to featurecollection object

* convert ogr.OFTInteger to int64 instead of int32 to unify it with how geopandas read it

* add qodana CI yaml file

* exclude paths fro qodana

* add conversion table between numpy, ogr, and gdal

* replace converting numpy data types to gdal data types to use the conversion table instead of the gdal_array.NumericTypeCodeToGDALTypeCode fns

* hard code the OGR.OFTInteger(0) to np.int32, OGR.OFTInteger64 (12) to np.int64, and OGR.Real (2) to np.float64

* add unit test for the gdal_to_numpy_dtype function

* use the DTYPE_CONVERSION_DF in the gdal_to_numpy_dtype function instead of the NUMPY_GDAL_DATA_TYPES dictionary

* raise error if the gdal data type code does not exist in the DTYPE_CONVERSION_DF

* move calculating the dtype from the __init__ method to the dtype property

* add numpy_dtype property

* rename dtype to gdal_dtype

* create dtype, and numpy_dtype properties

* change the scope of some fixtures to function

* change scopr of some features to function

* adjust the nodatavalue to use the new gdal_dtype, dtype and numpy_dtype

* use the DTYPE_CONVERSION_DF in the gdal_to_ogr_dtype function

* numpy_to_gdal_dtype can take a numpy dtype instead of an array

* add exceptions in the _check_no_data_value to capture when the no_data_value can not be set

* use the overflowerror generally to replace the no_data_value by the default one

* refactor tests

* clean

* ready array with the same dtype as the raster band, in the _crop_alligned use the DEFAULT_NO_DATA_VALUE if the no_data_value gives an error

* refactor

* deal with the problem of the None no_data_value with the unsigned integer dtype in the _check_no_data_value function

* correct the no_data_value in the raster

* change the to_featurecollection to use the crop method

* refactor

* add Numbers as a datatype

* add excetion handler in the to_file method to go around saving a raster with color table

* remove the test result file if exist

* rename ci files

* fix gdal versin to solve to gdal.Wrap error

* fix gdal version in ci to 3.7.1

* add missing test data

* add color_table property

* add color_table documentations

* add misssing test data

* move color_table to test_plot as it uses cleopatra package

* add the plot decorator on the color_table tests

* refactor

* update installation docs

* ignore old files

* refactor

* change return value from _window and convert _window, get_tile from static to normal mehtod

* refactor

* rest _window

* refactor

* reorder imports and rename locate_points to map_to_array_coordinates

* rename tests for the map_to_array_coordinates renamed method

* test array_to_map_coordinates

* refactor featurecollection module

* extend the functionality of create_point to create a feature_collection if epsg is given

* refactor

* clean

* update setup.py

* update history file

* update version number
  • Loading branch information
MAfarrag authored Oct 1, 2023
1 parent 726fac0 commit 9eadbfc
Show file tree
Hide file tree
Showing 134 changed files with 4,305 additions and 2,049 deletions.
13 changes: 7 additions & 6 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ Check relevant points.

# Checklist:

- [ ] updated version number in setup.py/pyproject.toml
- [ ] updated environment.yml and the lock file
- [ ] added changes to History.rst
- [ ] updated the latest version in README file
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] updated version number in setup.py/pyproject.toml.
- [ ] updated environment.yml and the lock file.
- [ ] added changes to History.rst.
- [ ] updated the latest version in README file.
- [ ] I have added tests that prove my fix is effective or that my feature works.
- [ ] New and existing unit tests pass locally with my changes.
- [ ] documentation are updated.
87 changes: 87 additions & 0 deletions .github/workflows/conda-deployment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: Conda
on: [push]
jobs:
Main-package:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
python-version: ["3.11"]
env:
OS: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
- uses: conda-incubator/setup-miniconda@v2
with:
mamba-version: "*"
#use-mamba: true
auto-update-conda: false
environment-file: environment.yml
auto-activate-base: false
activate-environment: test
python-version: ${{ matrix.python-version }}
channels: conda-forge,defaults
channel-priority: true
show-channel-urls: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: x64

- name: Install dev-dependencies
run: |
python -m pip install -r requirements-dev.txt
- name: Generate coverage report
shell: bash -el {0}
run: |
conda info
conda list
conda config --show-sources
conda config --show
pytest -sv -m "not plot"
Optional-packages:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
python-version: ["3.11"]
env:
OS: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
- uses: conda-incubator/setup-miniconda@v2
with:
mamba-version: "*"
#use-mamba: true
auto-update-conda: false
environment-file: environment-optional-packages.yml
auto-activate-base: false
activate-environment: test
python-version: ${{ matrix.python-version }}
channels: conda-forge,defaults
channel-priority: true
show-channel-urls: true

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: x64

- name: Install dev-dependencies
run: |
python -m pip install -r requirements-dev.txt
- name: Generate coverage report
shell: bash -el {0}
run: |
conda info
conda list
conda config --show-sources
conda config --show
pytest -sv
90 changes: 90 additions & 0 deletions .github/workflows/pypi-deployment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
name: Pypi
on: [push]
jobs:
Main-Package:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
python-version: ["3.9", "3.10", "3.11"]
env:
OS: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: x64

- name: Install GDAL
run: |
python -m pip install --upgrade pip
pip install --no-cache-dir Cython
pip install --find-links=https://girder.github.io/large_image_wheels --no-cache GDAL==3.7.1
- name: Test GDAL installation
run: |
python -c "from osgeo import gdal"
gdalinfo --version
- name: Install dependencies
run: |
pip install -r requirements.txt -r requirements-dev.txt
python setup.py install
- name: Generate coverage report
run: |
conda info
conda list
conda config --show-sources
conda config --show
python -m pytest -sv -m "not plot"
- name: Upload coverage reports to Codecov with GitHub Action
uses: codecov/codecov-action@v3

Optional-packages:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ ubuntu-latest]
python-version: ["3.9", "3.10", "3.11"]
env:
OS: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: x64

- name: Install GDAL
run: |
python -m pip install --upgrade pip
pip install --no-cache-dir Cython
pip install --find-links=https://girder.github.io/large_image_wheels --no-cache GDAL==3.7.1
- name: Test GDAL installation
run: |
python -c "from osgeo import gdal"
gdalinfo --version
- name: Install dependencies
run: |
pip install -r requirements-optional-packages.txt -r requirements-dev.txt
python setup.py install
- name: Generate coverage report
run: |
conda info
conda list
conda config --show-sources
conda config --show
python -m pytest -v --cov=pyramids --cov-report=xml
- name: Upload coverage reports to Codecov with GitHub Action
uses: codecov/codecov-action@v3
File renamed without changes.
54 changes: 0 additions & 54 deletions .github/workflows/ubuntu-optional-packages.yml

This file was deleted.

48 changes: 0 additions & 48 deletions .github/workflows/ubuntu.yml

This file was deleted.

2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -160,3 +160,5 @@ examples/coords.csv
examples/monthly_precipitation_amount_1hour_Accumulation_2022.05.01.00.00.00.tif
tests/data/raster-folder-coello/*
examples/old/*
tests/data/not-used/*
*.xml
10 changes: 5 additions & 5 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
fail_fast: true
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.1.0
rev: v4.4.0
hooks:
- id: end-of-file-fixer
name: "[py - check] validate yaml"
Expand Down Expand Up @@ -51,14 +51,14 @@ repos:
# args: [ -i, --wrap-summaries, "0" ]

- repo: https://github.com/PyCQA/pydocstyle
rev: 6.1.1
rev: 6.3.0
hooks:
- id: pydocstyle
name: "[py - check] pydocstyle"
files: ^Hapi/

- repo: https://github.com/pycqa/flake8
rev: 6.0.0
rev: 6.1.0
hooks:
- id: flake8
name: "[py - check] flake8"
Expand All @@ -70,7 +70,7 @@ repos:
# hooks:
# - id: black
- repo: https://github.com/ambv/black
rev: 22.8.0
rev: 23.7.0
hooks:
- id: black
name: "[py - format] black"
Expand All @@ -83,7 +83,7 @@ repos:

# pre-commit-shell: Checks shell scripts against shellcheck.
- repo: https://github.com/detailyang/pre-commit-shell
rev: v1.0.6
rev: 1.0.5
hooks:
- id: shell-lint
name: "[bash - lint] shell-lint"
Expand Down
37 changes: 37 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,40 @@ History
* fix bug in passing ot not passing band index in case of multi band rasters
* change the bounds in to_dataset method to total_bounds tp get the bbox of the whole geometries in the gdf
* add convert_longitude method to convert longitude to range between -180 and 180

0.5.0 (2023-10-01)
------------------
Dataset
"""""""
* The dtype attribure is not initialized in the __init__, but initialized when using the dtype property.
* Create band_names setter method.
* add gdal_dtype, numpy_dtype, and dtype arrtibute to change between the dtype (data type general name), and the coresponding data type in numpy and gdal.
* Create color_table attribure, getter & setter property to set a symbology (assign color to different values in each band).
* The read_array method returns array with the same type as the dtype of the first band in the raster.
* add a setter method to the band_names property.
* The methods (create_driver_from_scratch, get_band_names) is converted to private method.
* The no_data_value check the dtype before setting any value.
* The convert_longitude used the gdal.Wrap method instead of making the calculation step by step.
* The to_polygon is converted to _band_to_polygon private method used in the clusters method.
* The to_geodataframe is converted to to_feature_collection. change the backend of the function to use the crop function is a vector mask is given.
* the to_crs takes an extra parameter "inplace".
* The locate_points is converted to map_to_array_coordinates.
* Create array_to_map_coordinates to translate the array indices into real map coordinates.

DataCube
""""""""
* rename the read_separate_files to read_multiple_files, and enable it to use regex strigs to filter files in a given directory.
* rename read_dataset to open_datacube.
* rename the data attribute to values

FeatureCollection
"""""""""""""""""
* Add a pivot_point attribure to return the top left corner/first coordinates of the polygon.
* Add a layers_count property to return the number of layers in the file.
* Add a layer_names property to return the layers names.
* Add a column property to return column names.
* Add the file_name property to store the file name.
* Add the dtypes property to retrieve the data types of the columns in the file.
* Rename bounds to total_bounds.
* The _gdf_to_ds can convert the GeoDataFrame to a ogr.DataSource and to a gdal.Dataset.
* The create_point method returns a shapely point object or a GeoDataFrame if an epsg number is given.
Loading

0 comments on commit 9eadbfc

Please sign in to comment.