Issue report priority calculation proposal

[Crola1702]

Description

Prioritizing a job would be a great metric to know which issues are the ones that are causing the most stress on the buildfarm. This metric is calculated based on a job priority and the flaky ratio of the test in the affected jobs.

This metric is calculated based on the reports from: Buildfarm Issue Triage

Explanation

Currently, there are two ways of calculating an issue priority:

1. Suppose every job has the same priority, then the priority of an issue is the sum of the flaky ratio of the test in the affected jobs.
2. Suppose every job has a different priority (defined in Buildfarm Issue Triage), then the priority of an issue is the sum of the flaky ratio of the test in the affected jobs multiplied by the priority of the job. If the issue is consistent, the priority multiplier increases by 50% (to give more importance to consistent issues).

Examples

Example: TestServiceIntrospection in Rolling, Jazzy and Iron connext

1. Get information from the issue saved in the database

First we need to get the test regressions and jobs affected by the issue:

Using buildfarm tools:

$ ./sql_run.sh get_known_issues_job_link.sql "https://github.com/ros2/rclcpp/issues/2613" # This is a new script that will be added to the buildfarm tools

error_name|package_name|job_name|github_issue|status
projectroot.test.rclcpp.test_service_introspection|projectroot|Ici__nightly-connext_ubuntu_jammy_amd64|https://github.com/ros2/rclcpp/issues/2613|OPEN
projectroot.test.rclcpp.test_service_introspection|projectroot|Jci__nightly-connext_ubuntu_noble_amd64|https://github.com/ros2/rclcpp/issues/2613|OPEN
projectroot.test.rclcpp.test_service_introspection|projectroot|Rci__nightly-connext_ubuntu_noble_amd64|https://github.com/ros2/rclcpp/issues/2613|OPEN

We can see there is only a single test regression in the report and it is affecting three jobs.

2. Get the flaky ratio of the test in the affected jobs

For each job we will calculate the flaky ratio:

$ ./sql_run.sh calculate_flakiness_jobs.sql "projectroot.test.rclcpp.test_service_introspection" "15 days" "Jci__nightly-connext_ubuntu_noble_amd64"
job_name|last_fail|first_fail|build_count|failure_count|failure_percentage|project|domain
Jci__nightly-connext_ubuntu_noble_amd64|2024-09-14|2024-09-02|5|5|100.0|ROS|http://build.ros2.org
...
Ici__nightly-connext_ubuntu_jammy_amd64|2024-09-16|2024-09-04|5|4|80.0|ROS|http://build.ros2.org
...
Rci__nightly-connext_ubuntu_noble_amd64|2024-09-15|2024-09-02|13|12|92.31|ROS|http://build.ros2.org

Note: We calculate the flaky ratios based on the FLAKY_BUILDS_DEFAULT_RANGE in buildfarm tools, which is 15 days.

3. Get the priority of the jobs

Now we can calculate the priority of this issue using both methods:

• Jazzy connext priority: 0.8
• Iron connext priority: 0.75
• Rolling connext priority: 0.85

	Procedure	Result
First method	$1+0.8+0.923$	$2.723$
Second method	$\text{Jazzy}: 0.8*(1.5) * 1.0 = 1.2$ (x1.5 from consistent multiplier) $\text{Iron}: 0.75 * 0.8 = 0.6$ $\text{Rolling}: 0.85 * 0.923 = 0.785$	$2.585$