v1.0.0

This new release brings ~400 new commits with improvements all over the place: docs, features, fixes... QA-Board is used in many projects at Samsung, and we don't have plans to break backward compatibility. Therefore we feel ready to call it this version v1.

If you run into issues with QA-Board, or want to know more about its use-cases, know it's actively developed. Don't hesitate to contact me.

New Features

• Display run badges to e.g. show inference results in QA-Board and link to model training logs
• Compare versus runs from different projects
• Soft-deletes to keep run metadata but remove all/some files
• Parallel sampling for auto-tuning
• User accounts, required to trigger experiments from the app.
• Per-user custom tests in the web application.
• Move runs to an existing or new batch
• Built-in distributed workers with celery, and monitoring with flower.
• Redo: all a batch's run, or just failed ones, or per-run... and with the same parameters as the original.
• Usage outside of git repos: support GIT_MESSAGE / GIT_AUTHORED_DATETIME / GIT_COMMIT / GIT_BRANCH in ENV.
• Support for default database per batches.yaml file
• Use indexing when using matrix configurations, e.g. ${matrix.value[0]}
• Input manifests support, and a --strict mode in bit-accuracy checks that forbids unexpected output files
• distributed workers via Jenkins
• Custom bit-accuracy compare function to e.g. ignore metadata
• More flexible per-input configurations in batches
• Can use globs to define in qaboard.yaml inputs.batches. In this case it can be helpful to enable the optional flag allow_duplicate_batches.
• Internal: for "user-side" DAG/pipelines make it possible to call e.g. CI=true GIT_COMMIT=xxxxx qa batch --action-on-existing=assert-exists

UX improvements

• Tons of improvements to the logs, status indicators and error messages. Both in the CLI and web application.
• We tried to make the defaults of everything (UI, config...) more natural.
• Tuning analysis / tradeoffs tab enabled with runs started on the CLI, not just with experiments from the app
• UI: layout improvements in many places
• Image viewer: No animations to make the diff feel smoother, tooltip with real image values, hide the reference if same as new. Allow ref images with different dimensions, and behave nicely.
• File viewer icons
• Filter list of projects
• Scroll to bottom button in logs
• target_passfail metrics to show in green/red instead of a gradient
• Text viewer: toogle side-by-side / inline
• Indicator when identical files are compared
• Filter milestones in the selection menu. Direct links to milestones in the project sidebar.
• Remove columns without metrics from the table view
• Mark pending runs as finished from the UI
• Highlight commit that have milestones and prevent deleting them
• Avoid recoloring plotly plots if we don't overlay new/reference
• ...and many minor things

Performance

• Support for custom postgresql configuration
• Removed re-renders and overhead in the web application
• Transparent pre-compressed gzip files (.gz) to limit storage usage with custom viewers
• Memoize string edit distances

Fixes

• Many minor issues all over the place...
• Gitlab not needed anymore. But if used, added a proxy to get project avatars, and support triggering manual jobs in CI pipelines with tons of jobs...
• hardcoded ports, URL...
• Don't rely on conventions for the output directory locations
• Better uswgi setup: no startup failures, "cheaper" algo, set queue size...
• Fix imports when not running under a QA-Board directory

Upgrades

• Upgraded dependencies of the web application and backend
• Website updated to Docusaurus v2

v0.8.13

Changes

• Fix; Removed the default runners.lsf config from qa init
• Removed the warnings about QABOARD_HOST not being set. We default to a local QA-Board server, and will update the docs to help users that use a centralized server. At some point we'll change the default to a hosted version, it will help users can started quicker.

v0.8.12

Changes:

• Options when a batch's runs are already started in a different batch: wait/skip/continue
• context.output_dir is always absolute and the max length further limited (thanks Windows!).
• Don't show empty tuning parameters in context.configs
• Minor bug and backward-compatibility fixes

v0.8.11

Some major updates:

1. The API was simplified:

• Instead of the ugly context.obj["configurations"], there is context.configs. And so on.
• Tuning comes for free, no need to worry anymore about context.obj["extra_parameters"]
• For convenience there is now context.params
• The docs were updated and changes are backward compatible.

2. Users can now save artifacts and outputs at different locations, and it's all a bit simpler to specify. Go to the docs to read more.

Maintenance release

Changes

• Choose the fist configured task runner as default runner, else locally.
• Use the correct database path when using multiple input types

v0.8.8.8

• Improved compatibility of the CLI client with older git versions
• Fix name collisions between run metrics and the metrics configuration, now that both are sent to the server

v0.8.7

In the release QA-Board no longer requires read rights to repositories.

• The qa client sends all the git info needed by QA-Board.
• The server is no longer tied to webhooks from Gitlab.

Releases on PyPy with GH Actions

We're testing the release process.