Double the number of partitions

[gherceg]

This should help iterate over the backlog of repeat records faster. Rolling this out is a bit tricky since we want to avoid processing a repeat record simultaneously. A lock is held for each partition until the check_repeaters_in_partition task queues all records for that partition, so I think the safest way to do this is to revoke all check_repeaters tasks to prevent check_repeaters_in_partition tasks from being spawned, wait for all check_repeaters_in_partition tasks to finish, and then apply this change.

Environments Affected

Production

[millerdev]

I think the safest way to do this is to revoke all check_repeaters tasks to prevent check_repeaters_in_partition tasks from being spawned, wait for all check_repeaters_in_partition tasks to finish, and then apply this change.

Is that a straightforward set of steps? Is it possible to list out the commands that will be run to accomplish it?

[gherceg]

Yeah reasonable request. So once the PR is merged, I would run the following:

cchq --control production update-config # updates localsettings
cchq production django-manage revoke_celery_tasks corehq.motech.repeaters.tasks.check_repeaters # revoke check_repeaters tasks to prevent check_repeaters_in_partition tasks from being spawned
cchq --control production service celery restart # settings change to take effect on celery workers
cchq production forward-port flower # monitor for check_repeaters_in_partition tasks
# once I don't see any more of those tasks (there are only 4 ever running at a time), kill the revoke_tasks command