Extend gcs (#23)

* update versions * adjust flake8 config * create secret_manager module * add examples to the `decode` method * update pre-commit hooks * update poetry * update poetry * rename the `GCS` parameter in the constructor to `project_id` * add `__str__` method * add examples for the `GCS` constructor * clean * create module directories * reformat * rename secret_manager.py to secrets_manager.py * add `bucket_list` property * add type hints * add `get_bucket` method * add examples * add `GCSBucket` class * delete the downloaded file * check if the file is downloaded * add google cloud authentication step * do not print service account * remove the data from the path * use poetry in readthedocs * update poetry * use python 3.12 in readthedocs * correct modules in the autodocs * remove unused imported class * rename blob to file * add `file_exists` * rename blob to file * add `search` method * move gcs tests to google_cloud folder * restructure tests * add a case to download an entire directory from gcs * rename the `download_file` method to `download` * extend the `upload_file` to upload directories * extend the `delete_file` to delete directories * reformat * delete uploaded files at the end of the test * rename `delete_file` to `delete` * rename `upload_file` to `upload` * rename `GCSBucket` to `Bucket` * add missing upload test data * add missing upload test data * merge `search` and `list_files` * compare file per file in the downloaded data * clean documentations from static docs * improve docstrings * improve docstrings * split the upload method into two helper function * split the download method into two helper function * rename the `google_cloud.secret_manager` module to utils * move the utils module to the root level of the package * move the utils module to the root level of the package * clean aws tests * remove the environment.yml from the checklist * update checklists files
Serapieum-of-alex · Dec 15, 2024 · 429cbb7 · 429cbb7
1 parent 1877f65
commit 429cbb7
Show file tree

Hide file tree

Showing 30 changed files with 2,043 additions and 716 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -27,7 +27,6 @@ Check relevant points.
 # Checklist:
 
 - [ ] updated version number in setup.py/pyproject.toml.
-- [ ] updated environment.yml and the lock file.
 - [ ] added changes to History.rst.
 - [ ] updated the latest version in README file.
 - [ ] I have added tests that prove my fix is effective or that my feature works.

diff --git a/.github/workflows/conda-deployment.yml b/.github/workflows/conda-deployment.yml
@@ -43,13 +43,16 @@ jobs:
         run: |
           echo "$GCP_SA_KEY" > gcp-key.json
           echo "GOOGLE_APPLICATION_CREDENTIALS=${{ github.workspace }}/gcp-key.json" >> $GITHUB_ENV
+          # verify the credentials
+          gcloud auth activate-service-account --key-file=gcp-key.json
+          gcloud auth list --quiet
 
       - name: Install dependencies with Poetry
         run: |
             echo "Installing dependencies..."
             conda run --name test poetry install
 
-      - name: Generate coverage report
+      - name: Run Tests
         shell: bash -el {0}
         run: |
           conda info

diff --git a/.github/workflows/pypi-deployment.yml b/.github/workflows/pypi-deployment.yml
@@ -32,14 +32,18 @@ jobs:
         run: |
             poetry --version
             poetry install
+
       - name: Set up Google Cloud credentials
         env:
           GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
         run: |
           echo "$GCP_SA_KEY" > gcp-key.json
           echo "GOOGLE_APPLICATION_CREDENTIALS=${{ github.workspace }}/gcp-key.json" >> $GITHUB_ENV
+          # verify the credentials
+          gcloud auth activate-service-account --key-file=gcp-key.json
+          gcloud auth list --quiet
 
-      - name: Generate coverage report
+      - name: Run Tests
         run: |
           poetry run pytest -sv --cov=src/unicloud --cov-report=xml
 

diff --git a/.gitignore b/.gitignore
@@ -57,7 +57,7 @@ cover/
 *.pot
 
 # Django stuff:
-*.log
+#*.log
 local_settings.py
 db.sqlite3
 db.sqlite3-journal

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,16 +1,21 @@
 fail_fast: true
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v5.0.0
     hooks:
+      - id: check-toml
+        name: "[py -  check] check toml"
       - id: end-of-file-fixer
         name: "[py -  check] validate yaml"
+      - id: no-commit-to-branch
+        name: "[git -  check] no commit to branch"
+        args: [ --branch=main ]
       - id: trailing-whitespace
         name: "[file - format] trim trailing whitespace"
         args: [ --markdown-linebreak-ext=md ]
       - id: check-added-large-files
         name: "[file -  check] large file"
-        args: [ --maxkb=5000 ]
+        args: [ --maxkb=2000 ]
       - id: check-docstring-first
         name: "[py   -  check] docstring first"
         files: /examples
@@ -38,50 +43,44 @@ repos:
         name: "[reqs - format] fix requirements.txt"
       - id: check-yaml
         name: "[yaml -  check] validate yaml"
-#  - repo: https://github.com/pre-commit/mirrors-isort
-#    rev: v5.10.1
-#    hooks:
-#      - id: isort
-#        name: "[py   - format] isort"
-#  - repo: https://github.com/PyCQA/docformatter
-#    rev: v1.4
-#    hooks:
+  - repo: https://github.com/pycqa/flake8
+    rev: 7.1.1
+    hooks:
+      - id: flake8
+        additional_dependencies: [Flake8-pyproject]
+        name: "[py   - check] flake8"
+        exclude: ^(examples/|tests/)
+#   - repo: https://github.com/pycqa/docformatter
+#     rev: v1.4
+#     hooks:
 #      - id: docformatter
 #        name: "[py   - format] docformatter"
-#        args: [ -i, --wrap-summaries, "0" ]
-
-  - repo: https://github.com/PyCQA/pydocstyle
+#        description: 'Formats docstrings to follow PEP 257.'
+#        entry: docformatter
+#        args: [-i]
+#        language: python
+#        types: [python]
+  - repo: https://github.com/pycqa/pydocstyle
     rev: 6.3.0
     hooks:
       - id: pydocstyle
         name: "[py   - check] pydocstyle"
-        files: ^Hapi/
-
-  - repo: https://github.com/pycqa/flake8
-    rev: 6.1.0
+        files: ^src/.*\.py$
+  - repo: https://github.com/pycqa/isort
+    rev: 5.13.2
     hooks:
-      - id: flake8
-        name: "[py   - check] flake8"
-#        language_version: python3.9
-        exclude: ^(examples/|tests/)
-
-  #- repo: https://github.com/psf/black
-  #  rev: 22.8.0
-  #  hooks:
-  #    - id: black
-  - repo: https://github.com/ambv/black
-    rev: 23.11.0
+      - id: isort
+        name: "[py   - format] isort"
+  - repo: https://github.com/psf/black
+    rev: 24.10.0
     hooks:
       - id: black
-        name: "[py   - format] black"
-#        language_version: python3.9
   - repo: https://github.com/lovesegfault/beautysh
     rev: v6.2.1
     hooks:
       - id: beautysh
         name: "[bash - format] beautysh"
 
-  # pre-commit-shell: Checks shell scripts against shellcheck.
   - repo: https://github.com/detailyang/pre-commit-shell
     rev: 1.0.5
     hooks:
@@ -112,3 +111,19 @@ repos:
         language: system
         pass_filenames: false
         always_run: true
+
+  - repo: local
+    hooks:
+      - id: examples-notebook-check
+        name: nbval
+        entry: pytest --nbval
+        language: system
+        files: \.ipynb$
+
+  - repo: local
+    hooks:
+      - id: doctest
+        name: doctest
+        entry: pytest --doctest-modules
+        language: system
+        files: ^src/.*\.py$
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -11,13 +11,17 @@ sphinx:
 build:
   os: "ubuntu-22.04"
   tools:
-    python: "3.11"
+    python: "3.12"
+  jobs:
+    post_create_environment:
+      # Install poetry
+      # https://python-poetry.org/docs/#installing-manually
+      - python -m pip install poetry
+    post_install:
+      # Install dependencies with 'docs' dependency group
+      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
+      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs
 
-python:
-  install:
-    - requirements: docs/requirements.txt
-    - method: pip
-      path: .
 
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
diff --git a/HISTORY.rst b/HISTORY.rst
@@ -11,3 +11,18 @@ History
 ------------------
 
 * Bump up versions of dependencies.
+
+0.3.0 (2024-12-15)
+------------------
+
+* create a utils module.
+
+dev
+"""
+* move the aws, and gcs modules to submodules `aws` and `google_cloud`.
+
+GCS
+"""
+* Create a `Bucket` class that represents a Google Cloud Storage bucket.
+* The `Bucket` class has methods for uploading, downloading, and deleting files.
+* The `Bucket` class has methods for listing files in the bucket.
diff --git a/README.md b/README.md
@@ -43,7 +43,7 @@ Installing unicloud
 Installing `unicloud` from the `conda-forge` channel can be achieved by:
 
 ```
-conda install -c conda-forge unicloud=0.2.0
+conda install -c conda-forge unicloud=0.3.0
 ```
 
 It is possible to list all the versions of `unicloud` available on your platform with:
@@ -65,7 +65,7 @@ pip install git+https://github.com/Serapieum-of-alex/unicloud
 to install the last release, you can easily use pip
 
 ```
-pip install unicloud==0.2.0
+pip install unicloud==0.3.0
 ```
 
 Quick start

diff --git a/docs/requirements.txt b/docs/requirements.txt
diff --git a/docs/source/aws.rst b/docs/source/aws.rst
@@ -51,7 +51,7 @@ The ``AWS`` class provides methods to interact with Amazon Web Services (AWS) S3
          aws_client.download_data('my-s3-bucket', 'path/in/bucket/file.txt', '/local/path/to/save/file.txt')
 
 
-.. automodule:: unicloud.aws
+.. automodule:: unicloud.aws.aws
    :members:
    :undoc-members:
    :show-inheritance:
diff --git a/docs/source/gcs.rst b/docs/source/gcs.rst
@@ -3,48 +3,7 @@ GCS Class Documentation
 
 The ``GCS`` class provides methods to interact with Google Cloud Storage (GCS), allowing for uploading and downloading files.
 
-.. class:: GCS(project_name, service_key=None)
-
-   Initializes the GCS client with the specified project name and optional service key.
-
-   :param project_name: The Google Cloud project name.
-   :type project_name: str
-   :param service_key: The path to the service account key file (optional).
-   :type service_key: str, optional
-
-   .. method:: upload(file_path, destination)
-
-      Uploads a file to GCS.
-
-      :param file_path: The path to the file to upload.
-      :type file_path: str
-      :param destination: The destination path in GCS.
-      :type destination: str
-
-      **Example**:
-
-      .. code-block:: python
-
-         gcs_client = GCS('my-gcp-project')
-         gcs_client.upload_data('/local/path/to/file.txt', 'bucket-name/object-name.txt')
-
-   .. method:: download(source, file_path)
-
-      Downloads a file from GCS.
-
-      :param source: The source path in GCS.
-      :type source: str
-      :param file_path: The local path to save the downloaded file.
-      :type file_path: str
-
-      **Example**:
-
-      .. code-block:: python
-
-         gcs_client = GCS('my-gcp-project')
-         gcs_client.download_data('bucket-name/object-name.txt', '/local/path/to/save/file.txt')
-
-.. automodule:: unicloud.gcs
+.. automodule:: unicloud.google_cloud.gcs
    :members:
    :undoc-members:
    :show-inheritance: