This repository contains part of Qubes OS infrastructure configuration.
This formula provides setup for Qubes OS build machine. It can consists of multiple build environments, each of them have at least two VMs:
- build-
ENV_NAME
- keys-
ENV_NAME
In addition to those, logs
VM is created. Configuration allows:
- build VM to upload build logs to
logs
VM logs
VM to unlock signing keys usage for appropriate keys VM- build VM access the keys VM
Each build VM have access to the network through Whonix Gateway, so all 3rd-party sources are downloaded through Tor.
Build environments list can be specified with build-infra:build-envs
pillar.
Example pillar is provided in /srv/pillar/base/qubesos-infra/build-infra.sls
.
-
Enable this formula with:
qubesctl top.enable build-infra qubesctl top.enable build-infra pillar=True
-
Apply the configuration:
qubesctl --all state.highstate
-
Provision gpg signing keys into appropriate
keys-
VMs, setup qubes-builder in appropriatebuild-
VMs. See below for detailed list.
Configure ssh in build VMs to bypass Whonix Gateway. The main reason here is performance. This shouldn't affect main goal of using Tor - making targeted attacks harder, for attacker controlling 3rd-party signing key. Ssh is used only to:
- download our code from github (where it's authenticated using gpg)
- upload binary packages to appropriate repositories
Technically it's done by configuring ssh (ProxyCommand
in
~/.ssh/config
) to use local.ConnectSSH
service to sys-net
(and pass
destination hostname in service argument), instead of establishing TCP
connection directly. local.ConnectSSH
service is implemented as simple netcat.
-
Enable this formula:
qubesctl top.enable build-infra.ssh-proxy
-
Apply the configuration:
qubesctl --all state.highstate
Configure a web server (Nginx) in sys-net
to receive github webhooks. The web
server receives push event
at the location /github-hooks/
with content
type application/json
. For triggering build from github, configure
the payload URL to be http://IPADDRESS/github-hooks/trigger-build
and for processing
github comments, configure payload URL to be http://IPADDRESS/github-hooks/process-comment
.
-
Enable this formula:
qubesctl top.enable build-infra.webhooks
-
Apply the configuration:
qubesctl --all state.highstate
Build infrastructure consists of several build environments. Each of them use 3 VMs:
- Build VM (
build-*
), responsible for:
- building the package with logging to
logs
VM in real time - sending build artifacts to Keys VM for signing (using split gpg)
- uploading signed packages to repository
- sending notifications to github (issue comments etc)
- Keys VM (
keys-*
), responsible for:
- keeping signing keys
- signing packages, but only if
logs
VM acknowledged build log reception
- Build VM (
logs-*
) VM (defaultlogs
), common for all their associated build environments, responsible for:
- receiving build logs
- sending them (in form of signed commits) to logs repositories
- signaling appropriate Keys VM of successful logs reception and sending
The above workflow and used qrexec services can be illustrated with the diagram below:
.-----------. .----------.
| build VM | | keys VM |
| | 4. qubes.Gpg | |
| |-------------------->| |
| | | |
| | | |
| | '----------'
'-----------' ^
| |
|1. qubesbuilder.BuildLog |
| |
v |
.---------------. 3. qubesbuilder.LogReceived|
| logs VM |----------------------------'
| | .-,( ),-.
| | 2. push signed logs .-( )-.
| |********************>( github )
| | '-( ).-'
| | '-.( ).-'
'---------------'
The build process may be triggered manually (be calling appropriate make
command in some of build VM), or in response to github notification. The later
is achieved using
builder-github, which
provide webhooks and qrexec services to handle this process. It boils down to:
- HTTP server in
sys-net
(nginx) receive HTTP POST request from github with information about an event in repository. It is handed to appropriate handler script fromgithub-webhooks
directory ofbuilder-github
. - Webhook handler extract essential data (for example repository name) and
send it to all build VMs using
qubesbuilder.TriggerBuild
qrexec service. - Qrexec service in a build VM check if the named component exists in any of
qubes-builder
instance and callscripts/auto-build
build script. This script check if anything new needs to be built - and if so, build it and upload to current-testing repository. This, among other things, report successful (or failed) build as an issue in appropriate repository.
Very similar approach is used to move packages from current-testing
to
current
repository:
-
HTTP server in
sys-net
(nginx) receive HTTP POST request from github with information about an comment on issue. It is handed to appropriate handler script fromgithub-webhooks
directory ofbuilder-github
. -
Webhook handler extract commend body, check if it have any PGP-signed data (but do not verify it yet) and send it to all build VMs using
qubesbuilder.ProcessGithubCommand
qrexec service. -
Qrexec service in a build VM check signature on the commend and if it's made by a trusted key, process the command. See documentation in builder-github for details.
.-,( ),-. .------------. .-( )-. HTTP POST | sys-net | ( github )****************>| | '-( ).-' | | '-.( ).-' '------------' ^ | * qubesbuilder.TriggerBuild github issue/comment qubesbuilder.ProcessGithubCommand * | * .----------. | ***********| build VM |<---------| * '----------' | * | * .----------. | ***********| build VM |<---------' * '----------' * packages * v .-,( ),-. .-( )-. ( repository ) '-( ).-' '-.( ).-'
In addition to the above, build VM use Tor to download 3rd-party software (including build dependencies etc). This is to make targeted attack as hard as possible even for someone having write access to 3rd-party source/binary repository (including access to a signing key).
The above architecture is designed to make the build process as transparent as possible, while limiting damages of a compromised single build environment. Especially:
- Build VM doesn't have direct access to a signing key, so it's impossible to leak it from there.
- Packages signed with different keys are built in separate build environments.
This means separate build environments for:
- Fedora packages,
- Debian packages,
- Community-supported templates,
- Contributed packages (TBD),
- Build VM send build logs in real time - so even if the build environment would be compromised in the build process, it won't be able to erase the evidences from the build log. Note that after build VM is compromised, one can no longer trust further build logs, as those can be easily falsified. But those produced in the past (including the exact compromise incident) should remain immutable.
- Build VM can request package signing only when sent a build log first. It's important to stress out here that Keys VM have no way to verify if the build log really corresponds to the binary received for signing. But since every build is logged in real time and it's a requirement for having package signed, logs will be available (publicly) and can be audited.
Configuration tasks not included in this formula:
-
In each keys VM:
- generate/import appropriate signing key
-
In each build VM:
-
generate/import ssh key used to upload packages
-
For any config
builder.conf
used, make sure to:- drop
NO_SIGN ?= 1
line - adjust
DISTS_VM
andDIST_DOM0
if needed (must be set using?=
operator) - adjust COMPONENTS, add builder-github there (must be set using
?=
operator, cannot use+=
) - set
SIGN_KEY
to appropriate key id (the one in matching keys VM) - set
LINUX_REPO_BASEDIR
to a appropriate directory (pointing exact release version, like$(SRC_DIR)/linux-yum/r4.0
) - include
$(HOME)/builder-github.conf
- drop
-
-
In each logs VM:
- generate/import ssh key, add it to each logs repositories (default QubesOS/build-logs) as deploy key with write access
- import logs signing key
- Make sure build VMs are large enough if not specified in
pillar
data.