What is the product roadmap for Touca Comparator?

[ghorbanzade]

This Q/A is inspired by conversation with @cthulhu-irl on Touca Community on Discord.

[ghorbanzade]

Touca Comparator is a server component that ships as a separate docker container touca-cmp and lives in /cmp. The development plans for this component are as follows:

Short term:

1. Upgrade to using C++ SDK v1.5.2. At the moment the Comparator is pinned to using v1.4.2 which was released 7 month ago. See Upgrade comparator dependency on C++ SDK #66.
2. Improve the implementation. There are so many implementation details in the code that are simply bad practice and could be improved for better performance, design and easier maintainability.

Mid term:
3. Server API routes for fetching jobs and receiving their outcome are insecure because Comparator is not performing authentication with the server. See #67.

Long term:
4. Rewrite Comparator in TypeScript and merge it into the Server Api (/api) code.

We'd love to jump to 4 and save ourselves the time to resolve 2 and 3 but it's difficult: All the code for deserialization and comparison are currently in the C++ SDK and doing 4 requires rewriting those code in TypeScript as well. While it pays in the long run to do so, we never had a high-enough business value to resolve this technical debt. We don't like that we have a separate C++ component in the Server code but this component is working just fine and well shielded from the user which makes this work less important than the other requested features.

The largest technical debt associated with the Comparator is having this component separate from the Backend which makes processing jobs an async workflow: The CLI and SDKs don't interact with the Comparator directly. They submit results to the Backend, the back-end creates jobs for the Comparator to process and mark as done, then fetches the results of comparison jobs marked as done when asked about them. This behavior is fine for our currently supported use-cases but really bad for the workflows we'd like to support.

Imagine, as a user, I have a tool that generates an output. I'd like to refactor my tool and use Touca to check whether this output changes in the new version. I want to be able to do:

./mytool | touca check
./rebuild-mytool.sh
./mytool | touca check

But for the second touca check call to return whether your output has changed, we need to find the earlier version of the output on the server, perform the comparison, and then visualize potential differences. Since comparison is async, we cannot do that on demand and need to use some polling or callback in the CLI code to see when comparison is done.

[ghorbanzade]

We have implemented the long-term task discussed here as part of #294. Touca Comparator is now an internal to the Touca server.