Merge pull request #99 from cedadev/docs-1.0

Flesh out and update documentation
cedadev · Mar 12, 2024 · fd19d7e · fd19d7e
2 parents 5610c0d + 7def6c0
commit fd19d7e
Show file tree

Hide file tree

Showing 29 changed files with 873 additions and 152 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -109,7 +109,14 @@ jobs:
       uses: ammaraskar/sphinx-action@master
       with: 
         docs-folder: "docs/"
-        pre-build-command: "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chown 1001 docs/build/html/coverage"
+        pre-build-command: |
+          "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chmod -R 777 docs/build/html/cover/htmlcov && chown 1001 docs/build/html/coverage"
+    - name: "Chmod directory structure"
+      run: |
+        sudo chmod -R 777 docs/build/html
+        # sudo chmod -R 777 docs/build/html/coverage
+        # sudo chmod -R 777 docs/build/html/coverage/htmlcov
+        ls -la docs/build/html/coverage/htmlcov
     - name: "Download coverage report"
       uses: actions/download-artifact@v3
       with:
@@ -120,6 +127,9 @@ jobs:
       with:
         name: "covbadge"
         path: "docs/build/html"
+    - name: "List html dir"
+      run: |
+        ls -la docs/build/html/coverage/htmlcov
     - name: Upload artifacts
       uses: actions/upload-artifact@v3
       with:

diff --git a/README.md b/README.md
@@ -6,52 +6,145 @@ CEDA Near-Line Data Store
 [![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/)
 [![Coverage](https://cedadev.github.io/nlds/coverage.svg)](https://cedadev.github.io/nlds/coverage/htmlcov/)
 
-This is the HTTP API server code for the CEDA Near-Line Data Store (NLDS).
-It requires the use of the NLDS client, either the command line or library:
-[NLDS client on GitHub](https://github.com/cedadev/nlds-client).
+This is the server code for the CEDA Near-Line Data Store (NLDS), consisting of 
+an HTTP API and a cluster of rabbit consumer microservices. The 
+[NLDS client](https://github.com/cedadev/nlds-client) is required to communicate 
+with the API, either via the command line interface or python client library.
 
-NLDS server is built upon [FastAPI](https://fastapi.tiangolo.com).
+The NLDS is a unified storage solution, allowing easy use of disk, s3 object 
+storage, and tape from a single interface. It utilises object storage as a cache
+for the tape backend allowing for low-latency backup 
 
-NLDS requires Python 3.  It has been tested with Python 3.9, 3.10 and Python 3.11.
+The NLDS server is built upon [FastAPI](https://fastapi.tiangolo.com) for the 
+API, [RabbitMQ](https://www.rabbitmq.com/) for the message broker, 
+[minio](https://min.io/) for the s3 client, 
+[SQLAlchemy](https://www.sqlalchemy.org/) for the database client and 
+[xrootd](https://xrootd.slac.stanford.edu/) for the tape interactions.
+
+Documentation can be found [here](https://cedadev.github.io/nlds/index.html).
 
 Installation
 ------------
 
+If installing locally we strongly recommend the use of a virtual environment to 
+manage the dependencies.
+
 1.  Create a Python virtual environment:
-    `python3 -m venv ~/nlds-venv`
+
+    ```
+    python3 -m venv nlds-venv
+    ```
 
 2.  Activate the nlds-venv:
-    `source ~/nlds-venv/bin/activate`
-
-3.  Install the nlds package with editing capability:
-    `pip install -e ~/Coding/nlds`
-
-Running - Dec 2021
-------------------
-
-1. NLDS currently uses `uvicorn` to run.  The command line to invoke it is:
-```uvicorn nlds.main:nlds --reload```
-
-    This will create the NLDS REST-API server at the IP-address: `http://127.0.0.1:8000/`
 
-2. To run the processors, you have two options:
-    1. In unique terminals start each processor individually, after 
-    activating the virtual env, for example:
-       ```source ~/nlds-venv/bin/activate; python nlds_processors/index.py```
-       This will send the output to the terminal.
-
-    2. Use the script `test_run_processor.sh`.  This will run all five processors
-       in the background, sending the output to five logs in the `~/nlds_log/`
-       directory.
-
-Viewing the API docs
---------------------
-
-FastAPI displays automatically generated documentation for the REST-API.  To browse this go to: http://127.0.0.1:8000/docs#/
+    ```
+    source nlds-venv/bin/activate
+    ```
+
+3.  You could either install the nlds package with editing capability from a 
+    locally cloned copy of this repo (note the inclusion of the editable flag 
+    `-e`), e.g.
+
+    ```
+    pip install -e ~/Coding/nlds
+    ```
+
+    or install this repo directly from github:
+
+    ```
+    pip install git+https://github.com/cedadev/nlds.git
+    ```
+
+4.  (Optional) There are several more requirements/dependencies defined:
+    *   `requirements-dev.txt` - contains development-specific (i.e. not 
+    production appropriate) dependencies. Currently this consists of a psycopg2 
+    binary python package for interacting with PostgeSQL from a local NLDS 
+    instance. 
+    *   `requirements-deployment.txt` - contains deployment-specific 
+    dependencies, excluding `XRootD`. Currently this consists of the psycopg2 
+    package but built from source instead of a precompiled binary. 
+    *   `requirements-tape.txt` - contains tape-specific dependencies, notably 
+    `XRootD`. 
+    *   `tests/requirements.txt` - contains the dependencies for the test suite. 
+    *   `docs/requirements.txt` - contains the dependencies required for 
+    building the documentation with sphinx.
 
 Server Config
 -------------
 
-To interface with the JASMIN accounts portal, for the OAuth2 authentication, a `.server_config` file has to be created.  This contains infrastructure information and so is not included in the GitHub repository.
+To interface with the JASMIN accounts portal, for the OAuth2 authentication, a 
+`.server_config` file has to be created. This contains infrastructure 
+information and so is not included in the GitHub repository. See the 
+[relevant documentation](https://cedadev.github.io/nlds/server-config/server-config.html) 
+and [examples](https://cedadev.github.io/nlds/server-config/examples.html) for 
+more information.
+
+A Jinja-2 template for the `.server_config` file can also be found in the 
+`templates/` directory.
+
+Running the Server
+------------------
 
-A Jinja-2 template for the `.server_config` file can be found in the `templates/` directory.
+1.  The NLDS API requires something to serve the API, usually uvicorn in a local 
+    development environment:
+
+    ```
+    uvicorn nlds.main:nlds --reload
+    ```
+
+    This will create a local NLDS API server at `http://127.0.0.1:8000/`. 
+    FastAPI displays automatically generated documentation for the REST-API, to 
+    browse this go to http://127.0.0.1:8000/docs/
+
+2.  To run the microservices, you have two options:
+    *   In individual terminals, after activating the virtual env, (e.g. 
+        `source ~/nlds-venv/bin/activate`), start each of the microservice 
+        consumers:
+        ```
+        nlds_q
+        index_q
+        catalog_q  
+        transfer_put_q   
+        transfer_get_q
+        logging_q
+        archive_put_q
+        archive_get_q
+        ```
+        This will send the output of each consumer to its own terminal (as well 
+        as whatever is configured in the logger).
+
+    *   Alternatively, you can use the scripts in the `test_run/` directory, 
+        notably `start_test_run.py` to start and `stop_test_run.py` to stop. 
+        This will start a [screen](https://www.gnu.org/software/screen/manual/screen.html) 
+        session with all 8 processors (+ api server) in, sending each output to 
+        a log in the `./nlds_log/` directory.
+
+Tests
+-----
+
+The NLDS uses pytest for its unit test suite. Once `test/requirements.txt` have 
+been installed, you can run the tests with 
+```
+pytest
+```
+in the root directory. Pytest is also used for integration testing in the 
+separate [nlds-test repo](https://github.com/cedadev/nlds-test). 
+
+The `pytest` test-coverage report can (hopefully) be found [here](https://cedadev.github.io/nlds/coverage/htmlcov/).
+
+
+License
+-------
+
+The NLDS is available on a BSD 2-Clause License, see the [license](./LICENSE.txt) 
+for more info.
+
+
+
+Acknowledgements
+================
+
+NLDS was developed at the Centre for Environmental Data Analysis and supported 
+through the ESiWACE2 project. The project ESiWACE2 has received funding from the 
+European Union's Horizon 2020 research and innovation programme under grant 
+agreement No 823988.
diff --git a/docs/source/ceda.png → docs/source/_images/ceda.png b/docs/source/ceda.png → docs/source/_images/ceda.png
diff --git a/docs/source/_images/esiwace2.png b/docs/source/_images/esiwace2.png
diff --git a/docs/source/_images/icon-black.png b/docs/source/_images/icon-black.png
diff --git a/docs/source/_images/icon.png b/docs/source/_images/icon.png
diff --git a/docs/source/_images/logo.png b/docs/source/_images/logo.png
diff --git a/docs/source/_images/nlds-logo.png b/docs/source/_images/nlds-logo.png
diff --git a/docs/source/_images/nlds.pdf b/docs/source/_images/nlds.pdf
diff --git a/docs/source/_images/nlds.png b/docs/source/_images/nlds.png
diff --git a/docs/source/status_images/all_off.png → docs/source/_images/status/all_off.png b/docs/source/status_images/all_off.png → docs/source/_images/status/all_off.png
diff --git a/docs/source/status_images/failed.png → docs/source/_images/status/failed.png b/docs/source/status_images/failed.png → docs/source/_images/status/failed.png
diff --git a/docs/source/status_images/part_failed.png → docs/source/_images/status/part_failed.png b/docs/source/status_images/part_failed.png → docs/source/_images/status/part_failed.png
diff --git a/docs/source/status_images/short_table.png → docs/source/_images/status/short_table.png b/docs/source/status_images/short_table.png → docs/source/_images/status/short_table.png
diff --git a/docs/source/status_images/success.png → docs/source/_images/status/success.png b/docs/source/status_images/success.png → docs/source/_images/status/success.png
diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html
@@ -0,0 +1,35 @@
+{% extends "!layout.html" %}
+  {% block footer %} {{ super() }}
+
+  <style>
+    /* Sidebar header (and topbar for mobile) */
+    .wy-side-nav-search, .wy-nav-top {
+      background: #A6C36F;
+    }
+
+    .caption-text {
+        color: #828C51
+    }
+
+    .wy-nav-side {
+        background: #1E352F;
+    }
+
+    .note {
+        background: #BEEF9E
+    }
+    .admonition-title {
+        background: #335145
+    }
+
+  </style>
+{% endblock %}
+
+<!-- 
+    a {
+        color: #828C51;
+    }
+
+    a:visited {
+        color: #a2b442;
+    } -->
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -19,7 +19,7 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'Near-line Data Store'
+project = 'Near-line Data Store Server'
 copyright = '2022-2024, Neil Massey & Jack Leland'
 author = 'Neil Massey & Jack Leland'
 
@@ -39,6 +39,7 @@
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
+html_favicon = '_images/icon-black.png'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -58,8 +59,8 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
-html_logo = "ceda.png"
+html_logo = "_images/nlds.png"
 html_theme_options = {
-    'logo_only': True,
-    'display_version': False,
+    'logo_only': False,
+    'display_version': True,
 }
diff --git a/docs/source/coverage/coverage-report.rst b/docs/source/coverage/coverage-report.rst
@@ -0,0 +1,10 @@
+Test coverage report
+====================
+
+Link to the most `recent coverage report <https://cedadev.github.io/nlds/coverage/htmlcov/>`_.
+
+.. toctree::
+    :maxdepth: 1
+    :caption: Coverage report
+
+    Index <htmlcov/index>
diff --git a/docs/source/coverage/htmlcov/index.rst b/docs/source/coverage/htmlcov/index.rst