Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Creating in case this is needed. Right now it is not.
Efforts to improve rate limiting and reduce the risk of unreasonable pillow lag are actively in progress, but in the meantime we remain susceptible to spikes in pillow lag. Apart from waiting out the increased load causing the backlog in pillows, our only other short term option is to increase the size of the pillow machine that handles processing case pillows so that we can increase the number of case pillow processes (we are currently memory bound on the r6a.4xlarge machine size).
This PR is only here in case it is absolutely needed.
Steps to rollout
r6a.8xlarge
in AWS for pillow_b2000cchq --control production after-reboot pillow_b2000
cchq --control production update-supervisor-confs --limit=pillowtop
Environments Affected
Production