Skip to content

Latest commit

 

History

History

backend

README.md

QA-Board Backend

QA-Board's backend built as a flask application. It exposes an HTTP API used to read/write all the metadata on QA-Board's runs.

How to start a development backend

  1. First get the code:
git clone git@gitlab-srv:common-infrastructure/qaboard.git
cd qaboard

  1. If you want to run a frontend, go to the README and without docker run cd webapp; npm install.

  2. Edit at the top-level of the repository development.yml, and replace arthurf with your user. Edit services/backend/passwd and add a line with your user, looking like arthurf:*:11611:10:Arthur Flam:/home/arthurf:/bin/tcsh. You can get it with getent passwd | grep arthurf.

  3. Start the server:

docker-compose -f docker-compose.yml -f development.yml up  -d 

# for more build logs
export BUILDKIT_PROGRESS=plain

Tip: If you called npm install in the webapp/, (see the README), a frontend connected to the dev backend will also be up on port 3000.

Get logs and a shell with:

docker-compose -f docker-compose.yml -f development.yml logs -f backend
docker-compose -f docker-compose.yml -f development.yml exec backend bash

Edit development.yml as suits your needs to e.g. change connect to another database using QABOARD_DB_HOST.

Consult also:

At SIRC:

sudo sysctl -w net.core.somaxconn=65536

Overview

sqlalchemy maps our classes (defined in /models) to database tables:

  • Projects
  • Versions of the code, called CiCommits
  • Each commit has Batches of related Outputs
  • Each output was run on a specific TestInputs

Flask helps us create an HTTP server. It exposes API endpoints defined in the api/ folder.

  • api.py: read/list data about projects/commits/outputs
  • webhooks.py: listens for (i) push notification from gitlab (ii) new results sent by qa.
  • tuning.py: ask for new tuning runs,

database.py manages how we access our database, and connects to the git repository via gitpython.

Changing the database schemas

  • when you add/rename/delete tables or fields to the database, you should define a migration

SQL performance

Custom database config

Get a sample config:

docker run -i --rm postgres:12-alpine cat  /usr/local/share/postgresql/postgresql.conf.sample > services/db/postgres.conf

And add it to your db container:

  db:
    volumes:
    - ./services/db/postgres.conf:/var/lib/postgresql/data/postgresql.conf

Tuning

Queries:

  • In the backend, set QABOARD_DB_ECHO=true to see all SQL queries
  • Get an SQL prompt with docker-compose exec db psql -U qaboard and play with EXPLAIN ANALYZE my-query.
  • pgadmin is available by default on port 5050.

Tuning:

  • Read here about how to investigate the database's performance.
# Check performance issues with
# https://github.com/jfcoz/postgresqltuner
apt-get install -y libdbd-pg-perl
postgresqltuner.pl --host=localhost --database=qaboard --user=ci --password=password

# or we can also use pgbadger:
# https://github.com/dalibo/pgbadger

Monitoring & Application performance (WIP)

To get information about how much time is spend where in the python code:

from ..utils import profiled
with profiled():
  ... # code to be profiled