From 55ef24391b92e3dc4b995f78cef57679e276d838 Mon Sep 17 00:00:00 2001 From: Steven Bal Date: Mon, 22 Jul 2024 16:56:15 +0200 Subject: [PATCH 1/3] :arrow_up: [open-zaak/open-zaak#1649] Upgrade open-api-framework to 0.7.1 --- requirements/base.txt | 2 +- requirements/ci.txt | 3 ++- requirements/dev.txt | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/requirements/base.txt b/requirements/base.txt index bc31b14e..52d101d1 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -235,7 +235,7 @@ notifications-api-common==0.2.2 # via # -r requirements/base.in # commonground-api-common -open-api-framework==0.6.1 +open-api-framework==0.7.1 # via -r requirements/base.in orderedmultidict==1.0.1 # via furl diff --git a/requirements/ci.txt b/requirements/ci.txt index 8e259105..28914eed 100644 --- a/requirements/ci.txt +++ b/requirements/ci.txt @@ -234,6 +234,7 @@ django-solo==2.2.0 django-two-factor-auth[phonenumberslite,webauthn]==1.17.0 # via # -r requirements/base.txt + # django-two-factor-auth # maykin-2fa django-webtest==1.9.7 # via -r requirements/test-tools.in @@ -375,7 +376,7 @@ notifications-api-common==0.2.2 # via # -r requirements/base.txt # commonground-api-common -open-api-framework==0.6.1 +open-api-framework==0.7.1 # via -r requirements/base.txt orderedmultidict==1.0.1 # via diff --git a/requirements/dev.txt b/requirements/dev.txt index cd7ed4c3..ace85af1 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -260,6 +260,7 @@ django-solo==2.2.0 django-two-factor-auth[phonenumberslite,webauthn]==1.17.0 # via # -r requirements/ci.txt + # django-two-factor-auth # maykin-2fa django-webtest==1.9.7 # via -r requirements/ci.txt @@ -424,7 +425,7 @@ notifications-api-common==0.2.2 # via # -r requirements/ci.txt # commonground-api-common -open-api-framework==0.6.1 +open-api-framework==0.7.1 # via -r requirements/ci.txt orderedmultidict==1.0.1 # via From ccdabd48be11d20848ca169b63636714b71477b7 Mon Sep 17 00:00:00 2001 From: Steven Bal Date: Tue, 23 Jul 2024 12:05:49 +0200 Subject: [PATCH 2/3] :memo: [open-zaak/open-zaak#1649] Document envvars replace the manual envvar docs with OAf `generate_envvar_docs`, this ensure that generic envvars that are used by every component are documented centrally in OAF and custom envvars can be documented in the code where they are declared --- docs/installation/config.rst | 142 +++++++++--------- src/objects/conf/base.py | 53 +++---- src/objects/conf/dev.py | 5 +- .../open_api_framework/env_config.rst | 34 +++++ 4 files changed, 126 insertions(+), 108 deletions(-) create mode 100644 src/objects/templates/open_api_framework/env_config.rst diff --git a/docs/installation/config.rst b/docs/installation/config.rst index 90f32bbd..bd0009da 100644 --- a/docs/installation/config.rst +++ b/docs/installation/config.rst @@ -1,109 +1,100 @@ -.. _installation_environment_config: +.. _installation_env_config: =================================== Environment configuration reference =================================== -The Objects and Objecttypes APIs can be run both as a Docker container or a VPS / dedicated server. It relies on other services, such as -database and cache backends, which can be configured through environment -variables. -Available environment variables -=============================== - -Required settings ------------------ - -* ``DJANGO_SETTINGS_MODULE``: which environment settings to use. Available options: - - - ``objects.conf.docker`` or ``objecttypes.conf.docker`` - - ``objects.conf.dev`` or ``objecttypes.conf.dev`` - - ``objects.conf.ci`` or ``objecttypes.conf.ci`` - -* ``SECRET_KEY``: secret key that's used for certain cryptographic utilities. You - should generate one via - `miniwebtool `_ +The Objects and Objecttypes APIs can be run both as a Docker container or a VPS / dedicated server. +It relies on other services, such as database and cache backends, which can be configured through environment variables. -* ``ALLOWED_HOSTS``: A comma separated (without spaces!) list of domains that - serve the installation. Used to protect against ``Host`` header attacks. - Defaults to ``*`` for the ``docker`` environment and defaults to - ``127.0.0.1,localhost`` for the ``dev`` environment. -Common settings ---------------- - -* ``DB_HOST``: Hostname of the PostgreSQL database. Defaults to ``db`` for the - ``docker`` environment, otherwise defaults to ``localhost``. - -* ``DB_USER``: Username of the database user. Defaults to ``objects`` or ``objecttypes``, +Available environment variables +=============================== -* ``DB_PASSWORD``: Password of the database user. Defaults to ``objects`` or ``objecttypes``, -* ``DB_NAME``: Name of the PostgreSQL database. Defaults to ``objects`` or ``objecttypes``, +Required +-------- -* ``DB_PORT``: Port number of the database. Defaults to ``5432``. +* ``SECRET_KEY``: Secret key that's used for certain cryptographic utilities. You should generate one via `miniwebtool `_. +* ``ALLOWED_HOSTS``: a comma separated (without spaces!) list of domains that serve the installation. Used to protect against Host header attacks. Defaults to: ``(empty string)``. -* ``CELERY_BROKER_URL``: URL for the Redis task broker for Celery. Defaults - to ``redis://127.0.0.1:6379/1``. -* ``CELERY_RESULT_BACKEND``: URL for the Redis result broker for Celery. - Defaults to ``redis://127.0.0.1:6379/1``. +Database +-------- -Elastic APM settings --------------------- +* ``DB_NAME``: name of the PostgreSQL database. Defaults to: ``objects``. +* ``DB_USER``: username of the database user. Defaults to: ``objects``. +* ``DB_PASSWORD``: password of the database user. Defaults to: ``objects``. +* ``DB_HOST``: hostname of the PostgreSQL database. Defaults to ``db`` for the docker environment, otherwise defaults to ``localhost``. +* ``DB_PORT``: port number of the database. Defaults to: ``5432``. -An integration with `Elastic APM `_ -can be configured by setting the following environment variables -* ``ELASTIC_APM_SERVICE_NAME``: the name of the service in APM, i.e. "Objects API - staging" +Cross-Origin-Resource-Sharing +----------------------------- -* ``ELASTIC_APM_SERVER_URL``: the URL of the APM instance to connect with +* ``CORS_ALLOW_ALL_ORIGINS``: allow cross-domain access from any client. Defaults to: ``False``. +* ``CORS_ALLOWED_ORIGINS``: explicitly list the allowed origins for cross-domain requests. Example: http://localhost:3000,https://some-app.gemeente.nl. Defaults to: ``[]``. +* ``CORS_ALLOWED_ORIGIN_REGEXES``: same as ``CORS_ALLOWED_ORIGINS``, but supports regular expressions. Defaults to: ``[]``. +* ``CORS_EXTRA_ALLOW_HEADERS``: headers that are allowed to be sent as part of the cross-domain request. By default, Authorization, Accept-Crs and Content-Crs are already included. The value of this variable is added to these already included headers. Defaults to: ``[]``. -* ``ELASTIC_APM_SECRET_TOKEN``: the token that is required to communicate with the APM instance -Other settings --------------- -* ``ENVIRONMENT``: An identifier for the environment, displayed in the admin depending on - the settings module used and included in the error monitoring (see ``SENTRY_DSN``). - The default is set according to ``DJANGO_SETTINGS_MODULE``. Good examples values are: +Celery +------ - * ``production`` - * ``test`` +* ``CELERY_RESULT_BACKEND``: the URL of the backend/broker that will be used by Celery to send the notifications. Defaults to: ``redis://localhost:6379/1``. +* ``CELERY_TASK_HARD_TIME_LIMIT``: Task hard time limit in seconds. The worker processing the task will be killed and replaced with a new one when this is exceeded. Defaults to: ``900``. -* ``SITE_ID``: The database ID of the site object. Defaults to ``1``. -* ``DEBUG``: Used for more traceback information on development environment. - Various other security settings are derived from this setting! Defaults to - ``True`` for the ``dev`` environment, otherwise defaults to ``False``. +Elastic APM +----------- -* ``IS_HTTPS``: Used to construct absolute URLs and controls a variety of - security settings. Defaults to the inverse of ``DEBUG``. +* ``ELASTIC_APM_SERVER_URL``: URL where Elastic APM is hosted. Defaults to: ``None``. +* ``ELASTIC_APM_SERVICE_NAME``: Name of the service for this application in Elastic APM. Defaults to ``objects - ``. +* ``ELASTIC_APM_SECRET_TOKEN``: Token used to communicate with Elastic APM. Defaults to: ``default``. +* ``ELASTIC_APM_TRANSACTION_SAMPLE_RATE``: By default, the agent will sample every transaction (e.g. request to your service). To reduce overhead and storage requirements, set the sample rate to a value between 0.0 and 1.0. Defaults to: ``0.1``. -* ``SUBPATH``: If hosted on a subpath, provide the value here. If you provide - ``/gateway``, the component assumes its running at the base URL: - ``https://somedomain/gateway/``. Defaults to an empty string. -* ``SENTRY_DSN``: URL of the sentry project to send error reports to. Defaults - to an empty string (ie. no monitoring). +Optional +-------- -* ``NOTIFICATIONS_DISABLED``: indicates whether or not notifications should be - sent to the Notificaties API for operations on the Object endpoint. - Defaults to ``True`` for the ``dev`` environment, otherwise defaults to ``False``. +* ``SITE_ID``: The database ID of the site object. You usually won't have to touch this. Defaults to: ``1``. +* ``DEBUG``: Only set this to ``True`` on a local development environment. Various other security settings are derived from this setting!. Defaults to: ``False``. +* ``USE_X_FORWARDED_HOST``: whether to grab the domain/host from the X-Forwarded-Host header or not. This header is typically set by reverse proxies (such as nginx, traefik, Apache...). Note: this is a header that can be spoofed and you need to ensure you control it before enabling this. Defaults to: ``False``. +* ``IS_HTTPS``: Used to construct absolute URLs and controls a variety of security settings. Defaults to the inverse of ``DEBUG``. +* ``CACHE_DEFAULT``: redis cache address for the default cache. Defaults to: ``localhost:6379/0``. +* ``CACHE_AXES``: redis cache address for the brute force login protection cache. Defaults to: ``localhost:6379/0``. +* ``EMAIL_HOST``: hostname for the outgoing e-mail server. Defaults to: ``localhost``. +* ``EMAIL_PORT``: port number of the outgoing e-mail server. Note that if you're on Google Cloud, sending e-mail via port 25 is completely blocked and you should use 487 for TLS. Defaults to: ``25``. +* ``EMAIL_HOST_USER``: username to connect to the mail server. Defaults to: ``(empty string)``. +* ``EMAIL_HOST_PASSWORD``: password to connect to the mail server. Defaults to: ``(empty string)``. +* ``EMAIL_USE_TLS``: whether to use TLS or not to connect to the mail server. Should be True if you're changing the ``EMAIL_PORT`` to 487. Defaults to: ``False``. +* ``DEFAULT_FROM_EMAIL``: The default email address from which emails are sent. Defaults to: ``objects@example.com``. +* ``LOG_STDOUT``: whether to log to stdout or not. Defaults to: ``False``. +* ``LOG_LEVEL``: control the verbosity of logging output. Available values are ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO`` and ``DEBUG``. Defaults to: ``WARNING``. +* ``LOG_QUERIES``: enable (query) logging at the database backend level. Note that you must also set ``DEBUG=1``, which should be done very sparingly!. Defaults to: ``False``. +* ``LOG_REQUESTS``: enable logging of the outgoing requests. Defaults to: ``False``. +* ``SESSION_COOKIE_SAMESITE``: The value of the SameSite flag on the session cookie. This flag prevents the cookie from being sent in cross-site requests thus preventing CSRF attacks and making some methods of stealing session cookie impossible. Defaults to: ``Strict``. +* ``CSRF_COOKIE_SAMESITE``: The value of the SameSite flag on the CSRF cookie. This flag prevents the cookie from being sent in cross-site requests. Defaults to: ``Strict``. +* ``ENVIRONMENT``: An identifier for the environment, displayed in the admin depending on the settings module used and included in the error monitoring (see ``SENTRY_DSN``). The default is set according to ``DJANGO_SETTINGS_MODULE``. +* ``SUBPATH``: If hosted on a subpath, provide the value here. If you provide ``/gateway``, the component assumes its running at the base URL: ``https://somedomain/gateway/``. Defaults to an empty string. Defaults to: ``None``. +* ``RELEASE``: The version number or commit hash of the application (this is also sent to Sentry). +* ``NUM_PROXIES``: the number of reverse proxies in front of the application, as an integer. This is used to determine the actual client IP adres. On Kubernetes with an ingress you typically want to set this to 2. Defaults to: ``1``. +* ``CSRF_TRUSTED_ORIGINS``: A list of trusted origins for unsafe requests (e.g. POST). Defaults to: ``[]``. +* ``NOTIFICATIONS_DISABLED``: indicates whether or not notifications should be sent to the Notificaties API for operations on the API endpoints. Defaults to ``True`` for the ``dev`` environment, otherwise defaults to ``False``. +* ``DISABLE_2FA``: Whether or not two factor authentication should be disabled. Defaults to: ``False``. +* ``LOG_OUTGOING_REQUESTS_EMIT_BODY``: Whether or not outgoing request bodies should be logged. Defaults to: ``True``. +* ``LOG_OUTGOING_REQUESTS_DB_SAVE``: Whether or not outgoing request logs should be saved to the database. Defaults to: ``False``. +* ``LOG_OUTGOING_REQUESTS_DB_SAVE_BODY``: Whether or not outgoing request bodies should be saved to the database. Defaults to: ``True``. +* ``LOG_OUTGOING_REQUESTS_MAX_AGE``: The amount of time after which request logs should be deleted from the database. Defaults to: ``7``. +* ``SENTRY_DSN``: URL of the sentry project to send error reports to. Default empty, i.e. -> no monitoring set up. Highly recommended to configure this. -* ``USE_X_FORWARDED_HOST``: whether to grab the domain/host from the ``X-Forwarded-Host`` - header or not. This header is typically set by reverse proxies (such as nginx, - traefik, Apache...). Default ``False`` - this is a header that can be spoofed and you - need to ensure you control it before enabling this. -* ``DISABLE_2FA``: whether to disable two-factor authentication. Defaults to ``False``. - If set to ``False``, 2FA will be required if not using OIDC. -* ``LOG_REQUESTS``: whether to enable logging of outgoing HTTP requests. Defaults to ``False``. -* ``LOG_OUTGOING_REQUESTS_DB_SAVE``: whether to save logged requests to the database. Defaults to ``False``. -Initial superuser creation --------------------------- +Initial superuser creation (Docker only) +---------------------------------------- A clean installation of Objects API comes without pre-installed or pre-configured admin user by default. @@ -127,6 +118,7 @@ which allows configuration via environment variables. All these environment variables are described at :ref:`command line `. + Specifying the environment variables ===================================== diff --git a/src/objects/conf/base.py b/src/objects/conf/base.py index 8ab45a50..c3cb36eb 100644 --- a/src/objects/conf/base.py +++ b/src/objects/conf/base.py @@ -5,17 +5,7 @@ init_sentry() - -DATABASES = { - "default": { - "ENGINE": "django.contrib.gis.db.backends.postgis", - "NAME": config("DB_NAME", "objects"), - "USER": config("DB_USER", "objects"), - "PASSWORD": config("DB_PASSWORD", "objects"), - "HOST": config("DB_HOST", "localhost"), - "PORT": config("DB_PORT", 5432), - } -} +DATABASES["default"]["ENGINE"] = "django.contrib.gis.db.backends.postgis" # Application definition @@ -82,14 +72,17 @@ # settings for sending notifications NOTIFICATIONS_KANAAL = "objecten" -NOTIFICATIONS_DISABLED = config("NOTIFICATIONS_DISABLED", False) -# TODO should this be moved to open-api-framework? # Add (by default) 5 (soft), 15 (hard) minute timeouts to all Celery tasks. -CELERY_TASK_TIME_LIMIT = config("CELERY_TASK_HARD_TIME_LIMIT", default=15 * 60) # hard -CELERY_TASK_SOFT_TIME_LIMIT = config( - "CELERY_TASK_SOFT_TIME_LIMIT", default=5 * 60 -) # soft +CELERY_TASK_TIME_LIMIT = config( + "CELERY_TASK_HARD_TIME_LIMIT", + default=15 * 60, + help_text=( + "Task hard time limit in seconds. The worker processing the task will be " + "killed and replaced with a new one when this is exceeded." + ), + group="Celery", +) # hard # # Django setup configuration @@ -107,22 +100,24 @@ # setup_configuration command # sites config -SITES_CONFIG_ENABLE = config("SITES_CONFIG_ENABLE", default=True) -OBJECTS_DOMAIN = config("OBJECTS_DOMAIN", "") -OBJECTS_ORGANIZATION = config("OBJECTS_ORGANIZATION", "") +SITES_CONFIG_ENABLE = config("SITES_CONFIG_ENABLE", default=True, add_to_docs=False) +OBJECTS_DOMAIN = config("OBJECTS_DOMAIN", "", add_to_docs=False) +OBJECTS_ORGANIZATION = config("OBJECTS_ORGANIZATION", "", add_to_docs=False) # objecttypes config OBJECTS_OBJECTTYPES_CONFIG_ENABLE = config( - "OBJECTS_OBJECTTYPES_CONFIG_ENABLE", default=True + "OBJECTS_OBJECTTYPES_CONFIG_ENABLE", default=True, add_to_docs=False ) -OBJECTTYPES_API_ROOT = config("OBJECTTYPES_API_ROOT", "") -if OBJECTTYPES_API_ROOT and not OBJECTTYPES_API_ROOT.endswith("/"): +OBJECTTYPES_API_ROOT = config("OBJECTTYPES_API_ROOT", "", add_to_docs=False) +if OBJECTTYPES_API_ROOT and not OBJECTTYPES_API_ROOT.endswith("/", add_to_docs=False): OBJECTTYPES_API_ROOT = f"{OBJECTTYPES_API_ROOT.strip()}/" OBJECTTYPES_API_OAS = config( - "OBJECTTYPES_API_OAS", default=f"{OBJECTTYPES_API_ROOT}schema/openapi.yaml" + "OBJECTTYPES_API_OAS", + default=f"{OBJECTTYPES_API_ROOT}schema/openapi.yaml", + add_to_docs=False, ) -OBJECTS_OBJECTTYPES_TOKEN = config("OBJECTS_OBJECTTYPES_TOKEN", "") +OBJECTS_OBJECTTYPES_TOKEN = config("OBJECTS_OBJECTTYPES_TOKEN", "", add_to_docs=False) # Demo User Configuration -DEMO_CONFIG_ENABLE = config("DEMO_CONFIG_ENABLE", default=DEBUG) -DEMO_TOKEN = config("DEMO_TOKEN", "") -DEMO_PERSON = config("DEMO_PERSON", "") -DEMO_EMAIL = config("DEMO_EMAIL", "") +DEMO_CONFIG_ENABLE = config("DEMO_CONFIG_ENABLE", default=DEBUG, add_to_docs=False) +DEMO_TOKEN = config("DEMO_TOKEN", "", add_to_docs=False) +DEMO_PERSON = config("DEMO_PERSON", "", add_to_docs=False) +DEMO_EMAIL = config("DEMO_EMAIL", "", add_to_docs=False) diff --git a/src/objects/conf/dev.py b/src/objects/conf/dev.py index 7baf93a0..0c96e2f7 100644 --- a/src/objects/conf/dev.py +++ b/src/objects/conf/dev.py @@ -10,6 +10,7 @@ os.environ.setdefault("IS_HTTPS", "no") os.environ.setdefault("RELEASE", "dev") os.environ.setdefault("ENVIRONMENT", "development") +os.environ.setdefault("DISABLE_2FA", "True") os.environ.setdefault("DB_NAME", "objects"), os.environ.setdefault("DB_USER", "objects"), @@ -104,10 +105,6 @@ if "test" in sys.argv: NOTIFICATIONS_DISABLED = True -# None of the authentication backends require two-factor authentication. -if config("DISABLE_2FA", default=True): # pragma: no cover - MAYKIN_2FA_ALLOW_MFA_BYPASS_BACKENDS = AUTHENTICATION_BACKENDS - # Override settings with local settings. try: from .local import * # noqa diff --git a/src/objects/templates/open_api_framework/env_config.rst b/src/objects/templates/open_api_framework/env_config.rst new file mode 100644 index 00000000..6544c7db --- /dev/null +++ b/src/objects/templates/open_api_framework/env_config.rst @@ -0,0 +1,34 @@ +{% extends "open_api_framework/env_config.rst" %} + +{% block intro %} +The Objects and Objecttypes APIs can be run both as a Docker container or a VPS / dedicated server. +It relies on other services, such as database and cache backends, which can be configured through environment variables. +{% endblock %} + +{% block extra %} + +Initial superuser creation (Docker only) +---------------------------------------- + +A clean installation of Objects API comes without pre-installed or pre-configured admin +user by default. + +Users of Objects API can opt-in to provision an initial superuser via environment +variables. The user will only be created if it doesn't exist yet. + +* ``OBJECTS_SUPERUSER_USERNAME``: specify the username of the superuser to create. Setting + this to a non-empty value will enable the creation of the superuser. Default empty. +* ``OBJECTS_SUPERUSER_EMAIL``: specify the e-mail address to configure for the superuser. + Defaults to `admin@admin.org`. Only has an effect if ``OBJECTS_SUPERUSER_USERNAME`` is set. +* ``OBJECTS_SUPERUSER_PASSWORD``: specify the password for the superuser. Default empty, + which means the superuser will be created _without_ password. Only has an effect + if ``OBJECTS_SUPERUSER_USERNAME`` is set. + +Initial configuration +--------------------- + +Both Objects API and Objecttypes API support `setup_configuration` management command, +which allows configuration via environment variables. +All these environment variables are described at :ref:`command line `. + +{% endblock %} \ No newline at end of file From 85b78db22fe3d29d75bead2632037f3571db3ac9 Mon Sep 17 00:00:00 2001 From: Steven Bal Date: Tue, 23 Jul 2024 12:07:00 +0200 Subject: [PATCH 3/3] :construction_worker: [open-zaak/open-zaak#1649] Job to check if docs changed --- .github/workflows/ci.yml | 31 +++++++++++++++++++++++++++++++ bin/generate_envvar_docs.sh | 4 ++++ 2 files changed, 35 insertions(+) create mode 100755 bin/generate_envvar_docs.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fdc0b9bb..9f8a0ef0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,6 +63,37 @@ jobs: - name: Publish coverage report uses: codecov/codecov-action@v3 + docs: + runs-on: ubuntu-latest + name: Documentation build + + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: 'pip' + cache-dependency-path: 'requirements/*.txt' + - name: Install system packages + run: | + sudo apt-get update \ + && sudo apt-get install -y --no-install-recommends \ + libgdal-dev \ + gdal-bin + - name: Install dependencies + run: pip install -r requirements/ci.txt pytest + - name: Generate environment variable documentation using OAf and check if it was updated + run: | + bin/generate_envvar_docs.sh + changes=$(git diff docs/installation/config.rst) + if [ ! -z "$changes" ]; then + echo $changes + echo "Please update the environment documentation by running \`bin/generate_envvar_docs.sh\`" + exit 1 + fi + env: + DJANGO_SETTINGS_MODULE: objects.conf.ci + docker: needs: tests diff --git a/bin/generate_envvar_docs.sh b/bin/generate_envvar_docs.sh new file mode 100755 index 00000000..a5d751b3 --- /dev/null +++ b/bin/generate_envvar_docs.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +# Generates the documentation for environment variables +src/manage.py generate_envvar_docs --file docs/installation/config.rst \ No newline at end of file