Distributed arenaclients

[danielvschoor]

If we want to scale properly on Kubernetes, we need to find a way to dynamically allocate arenaclients.

Suggested flow:

• Spin up an AC with a "master" token. This will probably be a new user with read access to tokens
• AC makes a request to the API for an AC token
• The website looks for an unallocated AC user, e.g. arenaclient_008, and sends the name and token to the AC
• The AC makes a new request to the API for a match, using the token received in the previous step
• Everything continues as it did previously

Task list:

• Add a status field to the user model. The possible statuses would be "Allocated" and "Free"
• Add an API endpoint for the ACs to request a "Free" AC token. This endpoint needs to lock the table, find an AC with "Free" status, update the status to "Allocated" and return the name and token.
• Add an API endpoint to deallocate an AC. The AC will call this endpoint before it exits.
• Add a way to deallocate ACs if no activity for a few hours
• Add a new user that can view tokens. We can call the user "arenaclient"
• We need a way to differentiate between this "master" user and normal ACs/users. A new field on the user model?
• If a normal user calls the endpoint to request an AC token, it needs to return their own token

[lladdy]

This is some good analysis - thanks Dan.

@ratosh and I have discussed something similar to this in the context of distributed "untrusted" ACs.
Currently the untrusted ACs operate in the same way as our regular ACs - requiring a user to manually create each individual AC on their machine and a corresponding website user to be created by an admin.
Something like what you're proposing would also assist setup for this use case.

Some small differences from what you've described that @ratosh and I discussed:

• Website "ArenaClients" are no longer authenticated users. They are instead owned by a website user - the person who manages the deployment and owns the API token.
• The "master" AC/coordinator is provided the website users' API token, which it then passes to child containers it spawns.
• When said child containers request matches from the website, they provide the API token and an AC ID (managed by the master AC)
• At the time a match is requested, the website creates the AC entry in the DB if it doesn't already exist and serves the match to the AC.

One thing we need to consider is; if we create some sort of "master AC/coordinator" service, we potentially lose some of the robustness of the current setup. I'm talking about how nothing persists, and therefore there is no cleanup.
A "master AC/coordinator" service might need to manage container states and cleanup and risk leftover hanging instances. Though, potentially we could work out a way for the underlying technologies (docker, kubernetes, rancher) to take this off our hands.

@danielvschoor maybe you already understand these enough to comment?

[ratosh]

We do want to be able to scale and make it easy to setup ACs:

• User just need to use his API token when setting up ACs;
• AC db info can be created on demand (using AC API);
• Each AC instance is linked to a single AC db info by a identifier;
• Ping (every 5 mins) our BE when AC is running (and we know it is still alive) is a nice to have;

IMO:

• Never reallocate a AC db entry to another AC instance (it can still be done on AC side if you really want);
• We can improve AC view to only show Active ACs once we have more running;

Chess test frameworks references:

• https://tests.stockfishchess.org/
• http://chess.grantnet.us/