This is a small utility, which can generate GitHub workflows for Common Lisp projects.
It generates workflow for running tests and building docs. These workflows
use 40ants/run-tests and 40ants/build-docs
actions and SBLint to check code for compilation errors.
- Description: A tool simplify continuous deployment for Common Lisp projects.
- Licence:
BSD - Author: Alexander Artemenko
- Homepage: https://40ants.com/ci/
- Source control: GIT
- Depends on: alexandria, serapeum, str, yason
- This system hides all entrails related to caching.
- Includes a few ready to use job types.
- Custom job types can be defined and distributed as separate
ASDFsystems. - You don't have to write
YAMLanymore!
This system allows you to define workflows in the lisp code. The best way is to make these
definitions a part of your ASDF system. This way 40ants-ci (1 2) will be able to
automatically understand for which system it builds a workflow.
Each workflow consists of jobs and each job is a number of steps.
There are three predefine types of jobs and you can create your own. Predefined jobs
allows to reuse steps in multiple CL libraries.
In next examples, I'll presume you are writing code in a file which is the part
of the package inferred ASDF system EXAMPLE/CI. A file should have the following header:
(defpackage #:example/ci
(:use #:cl)
(:import-from #:40ants-ci/workflow
#:defworkflow)
(:import-from #:40ants-ci/jobs/linter)
(:import-from #:40ants-ci/jobs/run-tests)
(:import-from #:40ants-ci/jobs/docs))This job is automates git tag placement on the commit where you have changed the ChangeLog.md.
This can be a useful to automate package deployment and releases. You update the changelog, a job pushes a new git tag and the next action triggers on this tag and build a release.
Or you if you publish your library at Quicklisp distribution, then you can change
it's source type to the latest-github-tag to provide more stable releases to your
users. This way you commits into master will be ignored until you change the changelog and
git tag will be pushed. Here is an example how to setup this kind of quicklisp project source.
(defworkflow release
:on-push-to "master"
:jobs ((40ants-ci/jobs/autotag:autotag)))function 40ants-ci/jobs/autotag:autotag &key (filename *default-filename*) (regex *default-regex*) (tag-prefix *default-tag-prefix*) (token-pattern *default-token-pattern*) env
Creates a job which will run autotagger to create a new git tag for release.
class 40ants-ci/jobs/autotag:autotag (job)
This type of the job created a git tag when finds a new tag in specified file.
The simplest job type is linter. It loads a
(defworkflow linter
:on-pull-request t
:jobs ((40ants-ci/jobs/linter:linter)))When you'll hit C-c C-c on this definition,
it will generate .github/workflows/linter.yml with following content:
{
"name": "LINTER",
"on": {
"pull_request": null
},
"jobs": {
"linter": {
"runs-on": "ubuntu-latest",
"env": {
"OS": "ubuntu-latest",
"QUICKLISP_DIST": "quicklisp",
"LISP": "sbcl-bin"
},
"steps": [
{
"name": "Checkout Code",
"uses": "actions/checkout@v4"
},
{
"name": "Setup Common Lisp Environment",
"uses": "40ants/setup-lisp@v4",
"with": {
"asdf-system": "example"
}
},
{
"name": "Install SBLint",
"run": "qlot exec ros install cxxxr/sblint",
"shell": "bash"
},
{
"name": "Run Linter",
"run": "qlot exec sblint example.asd",
"shell": "bash"
}
]
}
}
}Here you can see, a few steps in the job:
- Checkout the code.
- Install Roswell & Qlot using 40ants/setup-lisp action.
- Install
SBLint. - Run linter for
example.asd.
Another interesting thing is that this workflow automatically uses ubuntu-latest OS,
Quicklisp and sbcl-bin Lisp implementation. Later I'll show you how to redefine these settings.
class 40ants-ci/jobs/linter:linter (lisp-job)
This job is similar to linter, but instead of SBLint it runs
Lisp Critic.
Lisp Critic is a program which advices how to make you Common Lisp code more idiomatic, readable and performant. Also, sometimes it might catch logical errors in the code.
Here is how you can add this job type in your workflow:
(defworkflow ci
:on-pull-request t
:jobs ((40ants-ci/jobs/critic:critic)))Also, you might combine this job together with others, for example, with linter:
(defworkflow ci
:on-pull-request t
:jobs ((40ants-ci/jobs/linter:linter)
(40ants-ci/jobs/critic:critic)))and they will be executed in parallel. See docs on 40ants-ci/jobs/critic:critic function
to learn about supported arguments.
Another interesting job type is 40ants-ci/jobs/run-tests:run-tests (1 2).
When using this job type, make sure, your system
runs tests on (ASDF:TEST-SYSTEM :system-name) call
and signals error if something went wrong.
(defworkflow ci
:on-push-to "master"
:by-cron "0 10 * * 1"
:on-pull-request t
:jobs ((40ants-ci/jobs/run-tests:run-tests
:coverage t)))Here I've added a few options to the workflow:
by-cron- sets a schedule.on-push-to- defines a branch or branches to track.
It will generate .github/workflows/ci.yml with following content:
{
"name": "CI",
"on": {
"push": {
"branches": [
"master"
]
},
"pull_request": null,
"schedule": [
{
"cron": "0 10 * * 1"
}
]
},
"jobs": {
"run-tests": {
"runs-on": "ubuntu-latest",
"env": {
"OS": "ubuntu-latest",
"QUICKLISP_DIST": "quicklisp",
"LISP": "sbcl-bin"
},
"steps": [
{
"name": "Checkout Code",
"uses": "actions/checkout@v4"
},
{
"name": "Setup Common Lisp Environment",
"uses": "40ants/setup-lisp@v4",
"with": {
"asdf-system": "example"
}
},
{
"name": "Run Tests",
"uses": "40ants/run-tests@v2",
"with": {
"asdf-system": "example",
"coveralls-token": "${{ secrets.github_token }}"
}
}
]
}
}
}The result is similar to the workflow generated for Linter, but uses 40ants/setup-lisp action at the final step.
Also, I've passed an option :coverage t to the job. Thus coverage
report will be uploaded to Coveralls.io automatically.
Lisp has many implementations and can be used on multiple platforms. Thus
it is a good idea to test our software on many combinations of OS and lisp
implementations. Workflow generator makes this very easy.
Here is an example of workflow definition with three dimentional matrix.
It not only tests a library under different lisps and OS, but also checks
if it works with the latest Quicklisp and Ultralisp distributions:
(defworkflow ci
:on-pull-request t
:jobs ((run-tests
:os ("ubuntu-latest"
"macos-latest")
:quicklisp ("quicklisp"
"ultralisp")
:lisp ("sbcl-bin"
"ccl-bin"
"allegro"
"clisp"
"cmucl")
:exclude (;; Seems allegro is does not support 64bit OSX.
;; Unable to install it using Roswell:
;; alisp is not executable. Missing 32bit glibc?
(:os "macos-latest" :lisp "allegro")))))Besides a build matrix, you might specify a multiple jobs of the same type, but with different parameters:
(defworkflow ci
:on-push-to "master"
:on-pull-request t
:jobs ((run-tests
:lisp "sbcl-bin")
(run-tests
:lisp "ccl-bin")
(run-tests
:lisp "allegro")))This will generate a workflow with three jobs: "run-tests", "run-tests-2" and "run-tests-3".
Meaningful names might be specified as well:
(defworkflow ci
:on-push-to "master"
:on-pull-request t
:jobs ((run-tests
:name "test-on-sbcl"
:lisp "sbcl-bin")
(run-tests
:name "test-on-ccl"
:lisp "ccl-bin")
(run-tests
:name "test-on-allegro"
:lisp "allegro")))Here is how these jobs will look like in the GitHub interface:
Third predefined job type is 40ants-ci/jobs/docs:build-docs (1 2).
It uses 40ants/build-docs
action and will work only if your ASDF system uses a documentation builder supported by
40ants/docs-builder.
To build docs on every push to master, just use this code:
(defworkflow docs
:on-push-to "master"
:jobs ((40ants-ci/jobs/docs:build-docs)))It will generate .github/workflows/docs.yml with following content:
{
"name": "DOCS",
"on": {
"push": {
"branches": [
"master"
]
}
},
"jobs": {
"build-docs": {
"runs-on": "ubuntu-latest",
"env": {
"OS": "ubuntu-latest",
"QUICKLISP_DIST": "quicklisp",
"LISP": "sbcl-bin"
},
"steps": [
{
"name": "Checkout Code",
"uses": "actions/checkout@v4"
},
{
"name": "Setup Common Lisp Environment",
"uses": "40ants/setup-lisp@v4",
"with": {
"asdf-system": "example",
"qlfile-template": ""
}
},
{
"name": "Build Docs",
"uses": "40ants/build-docs@v1",
"with": {
"asdf-system": "example"
}
}
]
}
}
}To significantly speed up our tests, we can cache installed Roswell, Qlot and Common Lisp fasl files.
To accomplish this task, you don't need to dig into GitHub's docs anymore!
Just add one line :cache t to your workflow definition:
(defworkflow docs
:on-push-to "master"
:cache t
:jobs ((40ants-ci/jobs/docs:build-docs)))Here is the diff of the generated workflow file. It shows steps, added automatically:
modified .github/workflows/docs.yml
@@ -20,13 +20,40 @@
"name": "Checkout Code",
"uses": "actions/checkout@v4"
},
+ {
+ "name": "Grant All Perms to Make Cache Restoring Possible",
+ "run": "sudo mkdir -p /usr/local/etc/roswell\n sudo chown \"${USER}\" /usr/local/etc/roswell\n # Here the ros binary will be restored:\n sudo chown \"${USER}\" /usr/local/bin",
+ "shell": "bash"
+ },
+ {
+ "name": "Get Current Month",
+ "id": "current-month",
+ "run": "echo \"::set-output name=value::$(date -u \"+%Y-%m\")\"",
+ "shell": "bash"
+ },
+ {
+ "name": "Cache Roswell Setup",
+ "id": "cache",
+ "uses": "actions/cache@v3",
+ "with": {
+ "path": "qlfile\n qlfile.lock\n /usr/local/bin/ros\n ~/.cache/common-lisp/\n ~/.roswell\n /usr/local/etc/roswell\n .qlot",
+ "key": "${{ steps.current-month.outputs.value }}-${{ env.cache-name }}-ubuntu-latest-quicklisp-sbcl-bin-${{ hashFiles('qlfile.lock') }}"
+ }
+ },
+ {
+ "name": "Restore Path To Cached Files",
+ "run": "echo $HOME/.roswell/bin >> $GITHUB_PATH\n echo .qlot/bin >> $GITHUB_PATH",
+ "shell": "bash",
+ "if": "steps.cache.outputs.cache-hit == 'true'"
+ },
{
"name": "Setup Common Lisp Environment",
"uses": "40ants/setup-lisp@v4",
"with": {
"asdf-system": "40ants-ci",
"qlfile-template": ""
- }
+ },
+ "if": "steps.cache.outputs.cache-hit != 'true'"
},
{TODO: I have to write a few chapters with details on additional job's parameters
and a way how to create new job types.
But for now, I want to show a small example, how to define a workflow with a job which takes care about lisp installation and then calls a custom step:
(defworkflow ci
:on-push-to "master"
:by-cron "0 10 * * 1"
:on-pull-request t
:cache t
:jobs ((40ants-ci/jobs/lisp-job:lisp-job :name "check-ros-config"
:lisp "ccl-bin"
:steps ((40ants-ci/steps/sh:sh "Show Roswell Config"
"ros config")))))Here we are using the class 40ants-ci/jobs/lisp-job:lisp-job which is base for most classes in this ASDF system
and pass a custom 40ants-ci/steps/sh:sh (1 2) step to it. This step will be called after the repostory checkout and CCL-BIN lisp installation.
so, thus when this step will run ros config command, it will output something like that:
asdf.version=3.3.5.3
ccl-bin.version=1.12.2
setup.time=3918000017
sbcl-bin.version=2.4.1
default.lisp=ccl-bin
Possible subcommands:
set
show
Pay attention to the NAME argument of 40ants-ci/jobs/lisp-job:lisp-job class. If you omit it, then default "lisp-job" name will be used.
package 40ants-ci
function 40ants-ci:generate system &key path
Generates GitHub workflow for given ASDF system.
This function searches workflow definitions in all packages
of the given ASDF system.
If PATH argument is not given, workflow files will be written
to .github/workflow/ relarive to the SYSTEM.
package 40ants-ci/github
generic-function 40ants-ci/github:generate obj path
generic-function 40ants-ci/github:prepare-data obj
variable 40ants-ci/vars:*current-system* -unbound-
When workflow is generated for ASDF system, this variable will contain a primary ASDF system.
package 40ants-ci/jobs/autotag
class 40ants-ci/jobs/autotag:autotag (job)
This type of the job created a git tag when finds a new tag in specified file.
Readers
reader 40ants-ci/jobs/autotag:filename (autotag) (:filename = *default-filename*)
File where to search for version numbers.
reader 40ants-ci/jobs/autotag:regex (autotag) (:regex = *default-regex*)
Regexp used to extract version numbers.
reader 40ants-ci/jobs/autotag:tag-prefix (autotag) (:tag-prefix = *default-tag-prefix*)
Tag prefix.
reader 40ants-ci/jobs/autotag:token-pattern (autotag) (:token-pattern = *default-token-pattern*)
Auth token pattern.
function 40ants-ci/jobs/autotag:autotag &key (filename *default-filename*) (regex *default-regex*) (tag-prefix *default-tag-prefix*) (token-pattern *default-token-pattern*) env
Creates a job which will run autotagger to create a new git tag for release.
package 40ants-ci/jobs/critic
class 40ants-ci/jobs/critic:critic (lisp-job)
Readers
reader 40ants-ci/jobs/critic:asdf-systems (critic) (:asdf-systems)
Critic can validate more than one system, but for the base class we need provide only one.
reader 40ants-ci/jobs/critic:ignore-critiques (critic) (:ignore-critiques)
A list strigns with names of critiques to ignore.
function 40ants-ci/jobs/critic:critic &key asdf-systems asdf-version ignore-critiques env
Creates a job which will run Lisp Critic for given ASDF systems.
If argument ASDF-SYSTEMS is NIL, it will use ASDF system
to which current lisp file is belong.
You may also provide ASDF-VERSION argument. It should be
a string. By default, the latest ASDF version will be used.
package 40ants-ci/jobs/docs
class 40ants-ci/jobs/docs:build-docs (lisp-job)
Builds documentation and uploads it to GitHub using "40ants/build-docs" github action.
Readers
reader 40ants-ci/jobs/docs:error-on-warnings (build-docs) (:error-on-warnings = t)
function 40ants-ci/jobs/docs:build-docs &key asdf-system asdf-version (error-on-warnings t) env
Creates a job of class build-docs.
package 40ants-ci/jobs/job
class 40ants-ci/jobs/job:job ()
Readers
reader 40ants-ci/jobs/job:exclude (job) (:exclude = nil)
A list of plists denoting matrix combinations to be excluded.
reader 40ants-ci/jobs/job:explicit-steps (job) (:steps = nil)
This slot holds steps given as a STEPS argument to a job constructor. Depending on a job class, it might add additional steps around these explicit steps.
reader 40ants-ci/jobs/job:job-env (job) (:env = nil)
An alist of environment variables and their values to be added on job level. Values are evaluated in runtime.
reader 40ants-ci/jobs/job:name (job) (:name)
If this name was not given in constructor, then name will be lowercased name of the job class.
reader 40ants-ci/jobs/job:os (job) (:OS = "ubuntu-latest")
reader 40ants-ci/jobs/job:permissions (job) (:permissions = nil)
A plist of permissions need for running the job.
These permissions will be bound to secrets.GITHUB_TOKEN variable.
Use default-initargs to override permissions in subclasses:
(:default-initargs
:permissions '(:content "write"))generic-function 40ants-ci/jobs/job:make-env job
generic-function 40ants-ci/jobs/job:make-matrix job
generic-function 40ants-ci/jobs/job:make-permissions job
Should return an alist with mapping from string to string where keys are scopes and values are permission names. Default method generates this alist from the plist of job's "permissions" slot.
generic-function 40ants-ci/jobs/job:steps job
generic-function 40ants-ci/jobs/job:use-matrix-p job
package 40ants-ci/jobs/linter
class 40ants-ci/jobs/linter:linter (lisp-job)
Readers
reader 40ants-ci/jobs/linter:asdf-systems (linter) (:asdf-systems = nil)
Linter can validate more than one system, but for the base class we need provide only one.
reader 40ants-ci/jobs/linter:check-imports (linter) (:check-imports = nil)
Linter will check for missing or unused imports of package-inferred systems.
function 40ants-ci/jobs/linter:linter &key asdf-systems asdf-version check-imports env
Creates a job which will run SBLint for given ASDF systems.
If no ASD files given, it will use all ASD files from
the current ASDF system.
package 40ants-ci/jobs/lisp-job
class 40ants-ci/jobs/lisp-job:lisp-job (job)
This job checkouts the sources, installs Roswell and Qlot. Also, it caches results between runs.
Readers
reader 40ants-ci/jobs/lisp-job:asdf-system (lisp-job) (:asdf-system = nil)
reader 40ants-ci/jobs/lisp-job:asdf-version (lisp-job) (:asdf-version = nil)
ASDF version to use when setting up Lisp environment. If NIL, then the latest will be used.
reader 40ants-ci/jobs/lisp-job:lisp (lisp-job) (:LISP = "sbcl-bin")
reader 40ants-ci/jobs/lisp-job:qlfile (lisp-job) (:qlfile = nil)
reader 40ants-ci/jobs/lisp-job:qlot-version (lisp-job) (:qlot-version = nil)
Qlot version to use when setting up Lisp environment. If NIL, then will be used version, pinned in setup-lisp github action.
reader 40ants-ci/jobs/lisp-job:quicklisp (lisp-job) (:QUICKLISP = "quicklisp")
reader 40ants-ci/jobs/lisp-job:roswell-version (lisp-job) (:roswell-version = nil)
Roswell version to use when setting up Lisp environment. If NIL, then will be used version, pinned in setup-lisp github action.
package 40ants-ci/jobs/run-tests
class 40ants-ci/jobs/run-tests:run-tests (lisp-job)
This job test runs tests for a given ASDF system.
Readers
reader 40ants-ci/jobs/run-tests:coverage (run-tests) (:coverage = nil)
reader 40ants-ci/jobs/run-tests:custom (run-tests) (:custom = nil)
function 40ants-ci/jobs/run-tests:run-tests &rest rest &key coverage qlfile asdf-system asdf-version os quicklisp lisp exclude custom env
Creates a job step of class run-tests.
package 40ants-ci/steps/action
class 40ants-ci/steps/action:action (step)
Readers
reader 40ants-ci/steps/action:action-args (action) (:args)
A plist to be passed as "with" dictionary to the action.
reader 40ants-ci/steps/action:uses (action) (:uses)
function 40ants-ci/steps/action:action name uses &rest args &key id if env &allow-other-keys
package 40ants-ci/steps/sh
class 40ants-ci/steps/sh:sh (step)
Readers
reader 40ants-ci/steps/sh:command (sh) (:command)
reader 40ants-ci/steps/sh:shell (sh) (:shell = *default-shell*)
function 40ants-ci/steps/sh:sh name command &key id if (shell *default-shell*) env
macro 40ants-ci/steps/sh:sections &body body
Returns a string with a bash script where some parts are grouped.
In this example we have 3 sections:
(sections
("Help Argument"
"qlot exec cl-info --help")
("Version Argument"
"qlot exec cl-info --version")
("Lisp Systems Info"
"qlot exec cl-info"
"qlot exec cl-info cl-info defmain"))It will be compiled into:
echo ::group::Help Argument
qlot exec cl-info --help
echo ::endgroup::
echo ::group::Version Argument
qlot exec cl-info --version
echo ::endgroup::
echo ::group::Lisp Systems Info
qlot exec cl-info
qlot exec cl-info cl-info defmain
echo ::endgroup::package 40ants-ci/steps/step
class 40ants-ci/steps/step:step ()
Readers
reader 40ants-ci/steps/step:env (step) (:env = nil)
An alist of environment variables.
reader 40ants-ci/steps/step:step-id (step) (:id = nil)
reader 40ants-ci/steps/step:step-if (step) (:if = nil)
reader 40ants-ci/steps/step:step-name (step) (:name = nil)
package 40ants-ci/utils
generic-function 40ants-ci/utils:system-packages system
Returns a list of packages created by ASDF system.
Default implementation returns a package having the same name as a system and all packages matched to package-inferred subsystems:
CL-USER> (docs-builder/utils:system-packages :docs-builder)
(#<PACKAGE "DOCS-BUILDER">
#<PACKAGE "DOCS-BUILDER/UTILS">
#<PACKAGE "DOCS-BUILDER/GUESSER">
#<PACKAGE "DOCS-BUILDER/BUILDERS/GENEVA/GUESSER">
#<PACKAGE "DOCS-BUILDER/BUILDER">
#<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/GUESSER">
#<PACKAGE "DOCS-BUILDER/DOCS">
#<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/BUILDER">)
function 40ants-ci/utils:alistp list
Test wheather LIST argument is a properly formed alist.
In this library, alist has always a string as a key.
Because we need them to have this form to serialize
to JSON propertly.
(alistp '(("cron" . "0 10 * * 1"))) -> T
(alistp '((("cron" . "0 10 * * 1")))) -> NIL
function 40ants-ci/utils:current-system-name
function 40ants-ci/utils:dedent text
Removes common leading whitespace from each string.
A few examples:
(dedent "Hello
World
and all Lispers!")
"Hello
World
and all Lispers!"
(dedent "
Hello
World
and all Lispers!")
"Hello
World
and all Lispers!"
(dedent "This is a code:
(symbol-name :hello-world)
it will output HELLO-WORLD.")
"This is a code:
(symbol-name :hello-world)
it will output HELLO-WORLD."
function 40ants-ci/utils:ensure-list-of-plists data
function 40ants-ci/utils:ensure-primary-system system
function 40ants-ci/utils:make-github-workflows-path system
function 40ants-ci/utils:plist-to-alist plist &key (string-keys t) (lowercase t)
Make an alist from a plist PLIST.
By default, transforms keys to lowercased strings
function 40ants-ci/utils:plistp list
Test wheather LIST is a properly formed plist.
function 40ants-ci/utils:single list
Test wheather LIST contains exactly 1 element.
function 40ants-ci/utils:to-json data
package 40ants-ci/vars
variable 40ants-ci/vars:*current-system* -unbound-
When workflow is generated for ASDF system, this variable will contain a primary ASDF system.
variable 40ants-ci/vars:*use-cache* nil
Workflow will set this variable when preparing the data or YAML generation.