Coverage, simple tooling config and `provision` command

This release is concentrated around stability and code quality: it brings code coverage, tooling config simplifactions and rename of install-site to provision command.

What's new since 1.16.1

🐳 Docker

New features

None

Fixes

• Removed DREVOPS_APP variable. @AlexSkrypnyk (#1011)
• Fixed variables format in Dockerfiles and added variables linter to the script. @AlexSkrypnyk (#999)
• Fixed deprectaed methods in test traits. @AlexSkrypnyk

Updates

• Enabled verbose output for Docker and composer. @AlexSkrypnyk (#1014)
• Update Docker images to 23.9.0 @renovate (#1008)

💧 Drupal

New features

• Added more test traits. @AlexSkrypnyk

Fixes

None

Updates

• Upgraded Drush to 12. @AlexSkrypnyk
• Removed custom DB overrides in favour of in-container variables. @AlexSkrypnyk (#1017)

🤖 CI and workflow

New features

• Added PHPUnit config file and coverage. @AlexSkrypnyk (#1016)
• Added support for Dockerfile updates in Renovate configurations. @AlexSkrypnyk (#1004)

Updates

• Renamed site-install to provision and updated env variables. @AlexSkrypnyk (#966) 🚀🚀🚀
  • DREVOPS_DRUPAL_INSTALL_FROM_PROFILE ➔ DREVOPS_PROVISION_USE_PROFILE
  • DREVOPS_DRUPAL_INSTALL_OVERRIDE_EXISTING_DB ➔ DREVOPS_PROVISION_OVERRIDE_DB
  • DREVOPS_DRUPAL_INSTALL_DB_SANITIZE_SKIP ➔ DREVOPS_PROVISION_SANITIZE_DB_SKIP
  • DREVOPS_DRUPAL_INSTALL_USE_MAINTENANCE_MODE ➔ DREVOPS_PROVISION_USE_MAINTENANCE_MODE
  • DREVOPS_DRUPAL_LOGIN_UNBLOCK_ADMIN ➔ DREVOPS_DRUPAL_UNBLOCK_ADMIN
  • DREVOPS_TASK_PURGE_CACHE_ACQUIA_SKIP ➔ DREVOPS_PURGE_CACHE_ACQUIA_SKIP
  • DREVOPS_DRUPAL_INSTALL_ENVIRONMENT ➔ DREVOPS_PROVISION_ENVIRONMENT
• Simplified testing tools to use config files. @AlexSkrypnyk (#1016) 🚀🚀🚀
• Simplified linting tools to use config files. @AlexSkrypnyk (#1018) 🚀🚀🚀
• Switched to the renamed CI runner image (drevops/ci-runner). @AlexSkrypnyk (#1024)
• Update toshimaru/auto-author-assign action to v2 @renovate (#1009)
• Renamed files with example suffix to use default suffix. @AlexSkrypnyk (#994)
• Updated email notification script to use sendmail before using mail. @AlexSkrypnyk (#983)

Fixes

• Fixed outdated Drupal version in .env. @AlexSkrypnyk (#1022)

🧩 Integrations

Lagoon

None (upstream fixes already included in images 23.9.0).

Acquia

• Fixed domain notification mapping in Acquia cache purging. @AlexSkrypnyk (#989)

📃 Documentation

• Updated docs. @AlexSkrypnyk (#1020)
• Fixed docs not correctly publishing multiple versions. @AlexSkrypnyk (#1023)
• Fixed example script not being included in docs. @AlexSkrypnyk (#985)
• Updated GitHub PR template. @AlexSkrypnyk (#968)

DrevOps

• Fixed coding standards in all scripts for all variables to be wrapped in ${}. @AlexSkrypnyk (#990)
• Consolidated "service" files to reside in .drevops directory. @AlexSkrypnyk (#1001)
• Install script:
  • Refactored installer into Symfony command app. @AlexSkrypnyk
• DrevOps tests:
  • Added test coverage for scripts. @AlexSkrypnyk (#995)
  • Updated bats-helpers. @AlexSkrypnyk (#1021)

Full Changelog: 1.16.1...1.17.0

@AlexSkrypnyk, @renovate and @renovate[bot]

Fixes for Installer, Acquia integrations and site install.

DrevOps has moved to git-flow!
Releases and hotfixes are pushed to main branch, while development takes place in develop branch.
This allows to easier manage hotfixes and release more often.

What's new since 1.16.0

🐳 Docker

New features

None

Fixes

None

Updates

• Updated images to 23.8.0. @AlexSkrypnyk

💧 Drupal

New features

None

Fixes

• Fixed deprecated methods in test traits. @AlexSkrypnyk
• Downgraded config_split to stable version 1. @AlexSkrypnyk (#964)

Updates

• Upgraded Drush to 12. 🚀 @AlexSkrypnyk

🤖 CI and workflow

New features

• Replaced PHP Email notification script with shell-based version. @AlexSkrypnyk (#956)
• Included .env into artifact deployment. This allows to control workflow from a single config file rather then having those variables being injected into provider's environment. @AlexSkrypnyk

Updates

• Updated CI builder version to 23.8.1. @AlexSkrypnyk (#963)

Fixes

🧩 Integrations

Lagoon

None

Acquia

• Fixed paths in Acquia hooks. @AlexSkrypnyk
• Fixed Acquia domain discovery in email notifications. @AlexSkrypnyk

📃 Documentation

None

DrevOps

• Refactored site-install to use drush:deploy (when config is provided) 🚀 🚀 🚀 @AlexSkrypnyk
  • Introduced DREVOPS_DRUPAL_INSTALL_ENVIRONMENT variable passed to the custom scripts. This allows to avoid querrying Drupal's config to get the current env type in the custom shell scripts. 🚀
  • Added passing of DREVOPS_DRUPAL_INSTALL_OVERRIDE_EXISTING_DB to the custom scripts if the DB was refreshed. This allows to know when the database was refreshed so that custom scripts could act appropriately. 🚀
  • Changed custom scripts to contain 2-digit suffix. This allows to order custom scripts better.
• Fixed several variable names and descriptions. @AlexSkrypnyk (#955)
• Added shfmt linting of scripts and fixed formatting. @AlexSkrypnyk
• Updated DREVOPS_DEBUG to be set by 1 rather than any value being present. This allows to cascade the override of the value set in "higher" configs (e.g., Lagoon's per-project value can be 1 but per-env can be set to 0). This approach is used for all variables in DrevOps. @AlexSkrypnyk (#964)
• Updated color messages to be based on the TERM's dumb value. @AlexSkrypnyk (#975)
• Installer:
  • Fixed latest version discovery in installer. @AlexSkrypnyk, @richardgaunt 🚀 🚀 🚀 (#976, #965)
• DrevOps tests:
  • Renamed test files for consistency.
  • Refactored site install tests and added a tests runner helper script. @AlexSkrypnyk

Full Changelog: 1.16.0...1.16.1

@AlexSkrypnyk, @richardgaunt

Drupal 10, Docker Compose 2, more tools and DrevOps test optimisations

🐳 Docker

New features

• Added support for Docker Compose v2 🚀🚀🚀.
• Made docker-compose.yml version-less.
• Added support for setting timezone in containers. Set timezone using DREVOPS_TZ variable in .env file. (#939)
• Added removing of orphan containers when removing images.
• Added running of the mariadb schema updates before capturing DB in a Docker image when exporting DB image.

Fixes

• Fixed missing user to run mariadb container. (#899)
• Fixed DREVOPS_DB_DOCKER_IMAGE_BASE not being pinned to a version. This now allows to choose a specific version of the DB image.
• Fixed local emails not being sent to Mailhog.

Updates

• Updated images to 23.7.0.
• Re-pinned Chrome image to 111.0 and pinned local Chrome override image.
• Replaced wait-for-dependencies container image with a pinned version of drevops/docker-wait-for-dependencies - DrevOps-owned image.
• Renamed 'default.docker-compose.override.yml' to 'docker-compose.override.example.yml' to group this file with docker-compose.yml.
• Renamed 'default.env.local' to '.env.local.example' to group this file with .env.
• Moved Docker network stub creation to CI config.
• Added back mounting volumes from pygmy.

💧 Drupal

New features

• Upgraded to Drupal 10 🚀🚀🚀.
• Added full Solr support via SearchAPI 🚀. (#859)
• Added support for Redis service 🚀. (#858)
• Added ClamAV support 🚀.
• Added "minimum-stability": "stable" and "prefer-stable": true stability flags to composer.json.
• Added support for composer packages authentication with GitHub token using GITHUB_TOKEN variable.
• Moved settings tests to be available for use in consumer sites.

Fixes

• Fixed automated cron should not run in CI. (#845)
• Fixed Shield configs in PROD environment: now allows to login to PROD without any env vars set. (#885)
• Fixed node_modules not being ignored for themes.

Updates

• Moved module-specific Drupal settings into separate files.
• Renamed docroot to web and made it configurable in installer 🚀. (#544)
• Removed custom generation of DB settings in favour of using database settings set in settings.php.
• Reverted ScriptHandler.php to the version from drupal-composer/drupal-project.
• Updated ScriptHandler.php to handle Drupal 10.1.x deprecations (see drupal-composer/drupal-project#626)
• Updated header color in a custom theme to better show that this is a custom override.

🤖 CI and workflow

New features

• Added support for PHPStan 🚀.
• Added support for PHPMD 🚀. (#908)
• Added support for Twigcs 🚀.
• Added release drafter configuration 🚀. (#789)
• Added support for setting tags from environment variables in tests (e.g. setting DREVOPS_TEST_UNIT_GROUP=wip1 will run unit tests tagged with wip1).
• Added new coloured verbose messages to all scripts.
• Show DB image in the ahoy info.
• Added 3 seconds waiting if Ahoy confirmation is suppressed.
• Added build time. (#835)
• Allow to disable lint tools by settings their targets to empty.
• Allow to pass DREVOPS_LINT_TYPE as a variable.

Updates

• Removed ahoy calls from CI 🚀. This was the last step for a complete separation of scripts from environment-based callers (ahoy is for local, CI config for CI, lagoon.yml for Lagoon etc).
• Pinned CI builder to 23.7.0. The image always must be pinned and updated manually to avoid breakages caused by upstream changes (#941)
• Removed flag to switch-off Compose lock file validation. Composer file always have to be valid.
• Made Lagoon webhook GitHub Actions handler configurable.
• Updated database dump filenames timestamp format to be db_YYYYMMDD_HHMMSS.sql.
• Added phpcs exclusion for tests.
• Updated release drafter to use MINOR version as the next one.
• Simplified artifact deployment gitignore config to deploy full code artifact.
• Separated test scripts into a router script: a single test.sh that calls other test-*.sh scripts, like test-unit.sh etc.
• Simplified DB download script to use globally defined environment variables.
• Separated lint scripts into a router script: a single lint.sh that calls other lint-*.sh scripts, like lint-be.sh etc.
• Simplified deploy scripts to use globally defined environment variables.
• Renamed DREVOPS_GITHUB_TOKEN to GITHUB_TOKEN as this is a widely-used env var name.
• Separated notification scripts into a router script: a single notify.sh that calls other notify-*.sh scripts, like notify-jira.sh, notify-email.sh etc.
• Renamed DOCKER_USERNAME to DOCKER_USER in all scripts.
• Renamed DREVOPS_AHOY_CONFIRM_RESPONSE to AHOY_CONFIRM_RESPONSE.
• Updated scripts to use full drush commands and improved site install information. (#922)
• Updated front-end linter bypass logic and messages.
• Updated test runner bypass logic and messages.
• Moved DREVOPS_DEV_VOLUMES_MOUNTED to commands in CI config.

Fixes

• Fixed Renovate configuration when it tries to update php version in composer.json. (#798)
• Fixed incorrect ahoy doctor status for when pygmy is not running.
• Fixed ahoy doctor incorrectly assessing loaded SSH key.
• Fixed SequelAce help string being shown in ahoy info if it is available on host.
• Fixed missing test artefacts and reports in CI configuration.
• Fixed BDD smoke tests not running in CI.
• Fixed second confirm question for 'ahoy reset'.
• Fixed github label creation script.
• Fixed JIRA deployment notification.

🧩 Integrations

Lagoon

• Added support for release/* and hotifx/* branches to be considered a test environment type in Lagoon.
• Fixed deployment to Lagoon when no DREVOPS_* env variables were defined. (#805)
• Simplified Lagoon configuration to use router scripts.
• Updated Drush aliases to support Lagoon environments.

Acquia

• Simplified Acquia variables to use globally defined environment variables.

📃 Documentation

• Added multi-version docs publishing 🚀. (#889)
• Updated feature list and added documentation for several sections.
• Simplified intro in README.txt (#833)
• Moved per-project documentation under docs dir.

DrevOps

• Moved DrevOps to semantic versioning 🚀.
• Added set -u to to all scripts to avoid unbound variables.
• Moved variables extractor into own package. (#814)
• Added linting of Dockerfiles. (#819)
• Install script:
  • Fixed site tokens not being processed in GitHub pull request template.
  • Renamed install.php to installer.php and moved away from root.
  • Move installer unit tests to the installer directory.
  • Added functional tests scaffold and updated to PHPUnit 10 for installer.
  • Updated installer main branch to 'main' in CI config.
• DrevOps tests:
  • Fixed spellchecker dictionary and added validation for it. (#817)
  • Fixed bats test framework setup to support a different temp dir.
  • Moved to using bats NodeJs library instead of host-installed binary. (#745)
  • Replaced using of real tokens with test tokens in tests.
  • Simplified utilities tests.
  • Renamed test assertions.
  • Updated docker-compose.yaml tests to support different sorting.
  • Improved test speed by using fast BDD tests.
  • Skipping CI tests that can run only in CI.

9.x-1.15 New variables system, PHP 8.1, self-hosted RenovateBot, improved integrations.

🐳 Docker

• Updated from PHP 7 to PHP 8.1 and optimised configuration files.
• Updated images to the 22.4.1 version.
• Renamed Docker files to use .dockerfile extension (i.e. Dockerfile.php -> php.dockerfile).
• Added auth.json to a list of included files into Docker build.
• Fixed forwarding emails to Mailhog locally.

💧 Drupal

• Upgraded to Drush 11.
• Added unit tests for settings: testing the resulting settings based on environment type and set environment variables.
• Added support for settings to be included from the includes directory (ueful to store contrib module settings).
• Added installer-path support for custom modules and themes in composer.json.
• Fixed site salt value in settings.
• Updated Shield settings to allow enabling from UI.
• Updated theme scaffold to use Olivero as a base theme.
• Updated theme npm run watch to run dev compilation.
• Updated theme NPM packages to the latest versions.
• Moved Livereload inclusion into theme.
• Switched from non-supported node-sass to sass ((Dart SASS)[https://sass-lang.com/dart-sass]) in theme

🤖 CI and workflow

New features

• New varibles system to clearly separate between DrevOps, Drupal and 3rd-party variables:
  • DrevOps variables use DREVOPS_ prefix.
  • Drupal variables use DRUPAL_ prefix.
  • 3-rd party services variables stay as-is.
• New site install logic and logging to prevent accidental environment overwrites and deletions.
• New Drupal deployment mechanism using drush deploy.
• New RenovateBot config (including self-hosted CI config) and removed dependencies.io.
• New deployment scripts for GitHub, NewRelic and JIRA notifications.
• New local Ahoy commands: commands can be added into .ahoy.local.yml which is automatically included int main Ahoy configuration and is excluded to be committed to the codebase.
• New script to mirror code between branches.
• New script to create GitHub labels. Used to configure repositories.
• New GitHub action to auto-assign PR author.

Updates

• Refactored scripts to use better logging output.
• Added Drupal login and logout scripts.
• Allow to skip unit, functional and kernel tests tagged with skipped group by providing envir9onment variables.
• Added skipping of site installation with DREVOPS_DRUPAL_INSTALL_SKIP.
• Removed DREVOPS_DRUPAL_CONFIG_LABEL as Drush 11 does not support it anymore.
• Added validation of the docker compose configuration.
• Moved network creation to run after composer validation.
• Added variables to skip linting (DREVOPS_LINT_SKIP) and tests (DREVOPS_TEST_SKIP).
• Added a flag to only build and export code DREVOPS_EXPORT_CODE_ONLY.
• Updated PHPCS ruleset to not contain overrides. Any file-specific overrides (like delimiting comments) were moved into files.
• Renamed master to main in branch references.
• Moved Behat screenshost directory under tests/bats/screenshots locally.
• Added support for Behat test to pass tags from the env variable.
• Added Behat test reports to CI: they can now be accessed via TEST tab.
• Added copying email to username after sanitization.
• Added Ahoy command to open DB in Sequel Ace.
• Added ahoy doctor info to gather information about OS to make debugging easier.
• Updated ahoy commands descriptions.

Fixes

• Fixed DrevOps and Docker Compose env variables not being automatically passed to running containers from the host env. Allows to avoid specifying variables that need to be passed into containers.
• Fixed skipping of BDD test re-runs when ran on local.
• Fixed one-time Drupal link not shown on build.
• Fixed DB Docker image not using base Docker image if it does not exist.
• Fixed config_split warning during site install if it is not available.
• Fixed only lint FE assets if FE theme has them and do not re-run FE assets compilation on host in CI to speed up build time.
• Fixed .editorconfig being overridden on composer update.
• Fixed deployment script to correctly handle deployment of skipped branches.
• Fixed email not being passed to site install from profile during installation.
• Fixed PHPCS ruleset to not scan .js files for uppercase TRUE/FALSE.
• Fixed sanitized passwords not being set to a random password by default during site install.
• Fixed failed deployments not failing CI.
• Fixed incorrect Drush was used for retrieving a core version.

🧩 Integrations

Lagoon

• Added backing up of DB before deployment to production in Lagoon.
• Added support for specifying custom LagoonCLI version and Lagoon instance in Lagoon integration scripts.
• Added support for LAGOON_PROJECT variable in .env.
• Added showing of DrevOps variables during Lagoon rollout.
• Added custom Lagoon task runner script.
• Fixed Lagoon deployment failing if there are no environments.
• Fixed sites installed from profile to handle DB override correctly in Lagoon.

Acquia

• Fixed TARGET_ENV_REMAP variable in Acquia cache purge script.
• Renamed code to artifact deployment type to push to Acquia.
• Updated webhook deployment to allow specifying the curl method.
• Updated notification hooks for Acquia.
• Fixed sanitize script not working in non-prod environments.

📃 Documentation

• Updated PR template to use checklist s instead of comments.
• Simplified README.md file.
• Added quickstart placeholder docs section.
• Added docs publishing to docs.drevops.com.
• Added a script to scan for shell variables and auto-generate docs.

Other

• Updated condition check for all boolean variables: use explicit 1 for a truthful value. This is to allow overriding of the value without unsetting a variable.
• Fixed unequal operator to be POSIX compliant in all scripts.
• Moved workflow tests into a separate CI job to ease DrevOps maintenance.
• Improvements in the DrevOps test system:
  • Updated Bats testing framework to Bats 1.7 and updated several helpers.
  • Updated Docker image workflow tests and added ARM support for tests.
  • Added DrevOps helpers test to the CI pipeline.
  • Updated Docker-Compose test fixtures.
• Updated installer script:
  • Fixed theme discovery in installer.
  • Updated installer path to use https://install.drevops.com
  • Updated installer to use install from profile and override DB variables.
  • Fixed installer to remove deploy job CI config when an option without a deployment is selected.

Apple M1 support!

Docker

• Updated images to be compatible with Apple M1 (thank to AmazeeIO for releasing images https://hub.docker.com/u/uselagoon).
• Added python and gcc to support Node packages compiling on M1.

CI and workflow

• Added ahoy composer.
• Fixed BUILD_VERBOSE variable name.
• Updated Drupal install script to be more verbose.
• Added skipping of deployments using env variables DEPLOY_SKIP_PR_<NUMBER> and DEPLOY_SKIP_BRANCH_<SAFE_BRANCH>.

Integrations

• Fixed Lagoon deployment to use moved Lagoon CLI binary.
• Fixed download from Lagoon script not using specified SSH key if multiple keys are loaded (permission denied on DB download).
• Updated Lagoon deployment script to support creating and deleting environments, including with fresh production database

Documentation

• Added documentation to handle additional M1 configuration.

Apple M1 support!

8.x-1.14

Added python and gcc to support Node packages compiling on M1.

Apple M1 support!

7.x-1.14

Added python and gcc to support Node packages compiling on M1.

First D9 release!

• All configuration and changes from 8.x-1.13

[8.x] Images updates to 21.7.0 and Composer 2

Docker

• Added Composer 2.
• Updated images to 21.7.0 and PHP 7.4.
• Updated Xdebug flag value to use true for the new Docker images. Upstream #251

CI and workflow

• Updated behat-screenshot to the latest version.
• Updated behat-steps to use the new namespace. (#633)
• Updated behat-format-progress-fail to the latest version.
• Removed obsolete AMAZEEIO_ variables in favour of MARIADB_.
• Removed port check from builds on Linux.
• Added support for DATABASE_SSH_FINGERPRINT in CI.
• Fixed parameter parsing within deployment notification script.

Back-end

• Added running of post_update hooks on core module install.

Front-end

• Updated FE development packages to the latest versions.
• Added ahoy fei command to install FE packages.

Integrations

• Fixed environment detection in Lagoon.
• Updated Lagoon config to use correct configuration for routes for production and development environments.
• Fixed Lagoon deployment not using production branch variable.
• Fixed Lagoon private registry password variable name.
• Fixed Lagoon dotfiles were not copied into container.
• Fixed DB being sanitised in Acquia.

Documentation

• Updated command descriptions and run all tests.

[7.x] Updated images to 21.7.0 and Composer 2

• Updated to Composer 2
• Updated images to 21.7.0 and PHP 7.4
• Ported changes from:
  • 8.x-1.11
  • 8.x-1.12
  • 8.x-1.13