Unable to add when date range is too big

[parberge]

It seems that we can't create all the events requested when they exceed a certain number. I tried with over 30 days when I ran into this problem

[parberge]

I can see that the timereport-api respons with a 500 (log taken from timereport-slack):

[DEBUG]	2019-09-03T16:26:45.194Z	3db0ae5d-238f-4b9a-8579-a8922a45d524	"POST /api/event/users/<user_id_removed> HTTP/1.1" 500 2030

[parberge]

This is the reason:

pynamodb.exceptions.PutError: Failed to put item: An error occurred (ProvisionedThroughputExceededException) on request (THTAD2C3FPSS3KTEOGIU6MI4TNVV4KQNSO5AEMVJF66Q9ASUAAJG) on table (dev_event) when calling the PutItem operation: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.

Questions:

• What is the limit, i.e how fast can we write?
• Should we introduce sleep to prevent to fast writes or increase "capacity"

[parberge]

@tommymalmqvist @kamigerami Any inputs on this issue?

[parberge]

@kamigerami suggested that we should try changing the settings via pynamodb: https://pynamodb.readthedocs.io/en/latest/settings.html

These two might help us:

• https://pynamodb.readthedocs.io/en/latest/settings.html#max-retry-attempts
• https://pynamodb.readthedocs.io/en/latest/settings.html#base-backoff-ms

[kamigerami]

yes - exponential backoff or auto increase limit (they support that in dynamo nowadays)

[parberge]

@mattiaslundberg With the move to using batch, have we tested this with big date ranges?

[mattiaslundberg]

Doesn't look any better :( We likely need to figure out something else... (Tried to add a range of 90 days and only ~25 are saved)