Import tracker statistics in batches

[josecelano]

The Index has a cronjob called "statistics importer". It imports torrent statistics from the associated tracker. The default configuration executed the process every hour:

[tracker_statistics_importer]
torrent_info_update_interval = 3600
port = 3002

This tasks:

1. Gets all torrents from the database. A compact version with only the internal ID and the infohash (SELECT torrent_id, info_hash FROM torrust_torrents).
2. It iterates over all torrents requesting the statistics from the Tracker REST API.

That solution has some problems:

• We do not have the same torrents in the Index and the Tracker. The tracker could have 1 torrent and the Index 1000 or the opposite. If the tracker only has one we are making 999 extra requests. The opposite case is not a problem because we are only making one request.
• If we have too many torrents in the Index making all the requests could take more the the interval you set in the configuration (default to 1h). That means it could take more than 1 hour to import all the statistics.

It seems that the process does scale well and it's not even efficient. There are different solutions depending on which aspect you want to improve. For example, if you want to scale it you can add an importation date and just fetch statistics continuously only importing the torrents that have not been imported in the last hour. This solution does not solve the problem of making request to the tracker for torrents it does not have.

Proposed solution 1

A naive implementation could be:

• Ask the tracker for the torrent statistics that have changed in the last hour.
• Use pagination for that response. The tracker could have millions of torrents.
• The index only has to process the tracker torrents and update the statistics.

Proposed solution 2

We could also not import statistics at all for all torrents and only import statistics for a single torrent when the users load the torrent details page. That would require an extra request to the tracker every time you load the page. It could work if we do not need statistics for other cases.

Other solutions

• Use other types of APIs where you can subscribe to changes in statistics.
• Use message queues to subscribe to statistics changes events.
• Etcetera

I will go with solution 2 and that would make the tracker API even simpler. This is related to using a DashMap for the tracker data. We would not need to use a BTreeMap to order torrents, because right now we only need it for the API. In addition, background tasks are hard to maintain:

• It's hard to detect problems.
• There can be deadlock, overlapping, ...
• Performance can be a problem as I described.
• ...

cc @da2ce7 @WarmBeer @mario-nt @grmbyrn

[mario-nt]

If we don't need the statistic of all torrents at any given time, it makes sense to import the statistics for a particular torrent when the user visits the details page. Also background tasks has their own drawbacks as you mentioned.

I'd like to do some research on message queues, but as of right now, I think option 2 it is a good idea.

[josecelano]

Hi in the live demo with a small server, we can only import 1 torrent per second with the current server load. See #459

And we always import the same ones because the query is SELECT torrent_id, info_hash FROM torrust_torrents.

At least we should introduce an importation_date so we can prioritize the ones that have not been imported recently. This way at least all of them would be importer sooner or later even if it takes long.

cc @da2ce7 @WarmBeer @mario-nt

[mario-nt]

@josecelano
I think it is a good idea to have some way of prioritizing updating the statistics from torrents that haven't been updated recently. I supposed
adding a new field to order the torrent import priority would not affect the query execution time or importing performance.

[josecelano]

Relates to: #569