Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite score calculation in Rust #25

Open
nicoburns opened this issue Jul 4, 2024 · 2 comments
Open

Rewrite score calculation in Rust #25

nicoburns opened this issue Jul 4, 2024 · 2 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@nicoburns
Copy link
Contributor

Score recalc is pretty slow currently. Slow enough that I left it and came back to it several times and it still wasn't done. We should consider rewriting it in Rust for better performance. Such an implementation could:

  • Use streaming parsing for the JSON (and LZMA decompression) using a custom serde Deserialize implementation, avoiding allocating huge arrays of test results. This could be done both for top-level tests and sub-tests.
  • Avoid allocating test name strings entirely by matching directly against the values in the raw file (which will never include escaped characters).
@nicoburns nicoburns added enhancement New feature or request good first issue Good for newcomers labels Jul 4, 2024
@mukilan
Copy link
Member

mukilan commented Jul 4, 2024

I agree, rewriting in Rust makes sense. Originally we were planning to have the dashboard show the scores for just 1 year, with a fresh graph for each year like the interop dashboard, but now that we have a single graph, there is a lot of data to process.

Another option is to truncate and archive data beyond a cutoff date (which is what I do when testing changes locally).

@mukilan
Copy link
Member

mukilan commented Jul 4, 2024

Also, at some point we'll be dropping the data for Layout 2013.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants