query-tee: Add an option to shift the query times for comparison

[codesome]

What this PR does

Adds an option proxy.shift-comparison-queries-by to shift the query times for query-tee comparisons.

Sometimes proxy.compare-skip-recent-samples option may not be enough because if you want to set it to a higher duration (say few hours), it will completely skip comparing the queries for recent times, which in practice forms a big % of all queries (like recording rules, many most instant queries, etc).

With this change, the query-tee still does the query with original times for the preferred backend, but for comparison, it shifts the query times, and does another query to the preferred backend with the shifted times and only one shifted query to the secondary backend and uses those results for comparison.

Added proxy.shift-comparison-sampling-ratio to samples the queries for shifting the query times.

Draft PR because there are still some TODOs and need to write tests.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[pracucci]

The overall design LGTM. I left a couple of comments. Would be glad to see a unit test.

[codesome]

Added a unit test

[codesome]

Tried it in a dev environment, weird that all shifted queries are being skipped for comparison because the response format is text/plain and not application/json. (I increased the shift ratio multiple times, hence the jumps)

[codesome]

I converted this to draft since there is the above bug that I need to fix.

[codesome]

It is working locally with the docker-compose setup, so I have no clue why it gave issues in the dev environment. I will try running it again in dev next week and try to debug it there.