Improve developer experience

[cbeauchesne]

Issue list

TODO

Build time, Exec time

• build time is too slow
  • Pre-build static docker images #562
    • Optimize python build time  #1331

Hard to write tests

• Add a linter for manifest file #1722
• Hard to know what are the scenario useful for a given feature
  • Collect automatically all scenario docs and export then in docs/ #1088
• Sugars
  • On parametric test, add a way to specify a commit SHA for building the tracer #1094

Debuggability

• Log requests done during system/parametric tests in curl format #1696
• Write a cheat sheet : where to look for in log folder #1723

CI ecosystem

• Make system tests works on forks ? #975
• If parametric is in PR title, or of a file is changed in tests/parametric, add run-parametric-scenario-label #1475

Done during Q2-Q3 2023

Build time, Exec time

• Setup a cache for runner files #1278
• try to understand why nodejs tracer take 15mn to build (and improve).
• Exec time is too slow
  • Add a feature that replay the tests based on logs #1093 and Add a mode that replay tests based on logs #1169
  • option to not restart containers ? (need to know what is the time saved with that, and be sure we do not introduce flakyness)

Hard to write tests

• Hard to debug test locally // Hard to write a new test from scratch
  • Runner is now in host  #958
  • Add VsCode file to run in debug mode #1084
  • Bug: can't execute an arbitrary test if it's not in the default scenario #1123 and Allow to specify an arbitrary file in a non-default scenario #1124
  • On failure, requests sent during the setup are not show in the logs #1126 and Show requests/response in log on failure  #1128
• Hard to know what are the scenario useful for a given feature
  • Write documentation on what is a scenario #1086
  • More Documentation on each scenario #1087
  • ./run.sh SCENARIO --help. -> print the documentation of scenario

PR process in system tests

• Egg-chicken issue between system tests PR and distant PR
  • Add a way fo force a test to be executed if a component is on a given branch #1054
  • Add a way to force a test to be executed #913
  • Communicate about the current recipe
• Decorators merge conflicts hell // Maintenance
  • Rewrite released decorator to only use one tracer (solving decorator hell issue) #1089
• PR review bottleneck -> Implementation of manifest files #1481
• PR without description
• Improve system tests CI
  • Launch every relevant scenarios if build is successful #1120

Maintain our CI

• maintenance of scenarios in each CI in each repo
  • Reduce maintenance of CI : ./run.sh <SET_OF_SCENARIO> #1085 and Add set of scenario feature #1133
• Reduce SAMPLING scenario flakyness threshold #1210

Other

• Execution en local des N variants

  • N variants. Indeed. Hard to tackle, as one run require mostly all your laptop ressources. Maybe a script that run all variant for a given tracer, but it'll be very slow.

• Importance to follow feature matrix

[anna-git]

Could we run the whole system tests CI without stopping on failure maybe?

[cbeauchesne]

You mean, on the system test CI ?

[cbeauchesne]

@anna-git

Could we run the whole system tests CI without stopping on failure maybe?

FYI, it's done