EmrStepSensor to auto detect stepid's #26809
Unanswered
pedresnyman
asked this question in
Ideas
Replies: 1 comment
-
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.
Beta Was this translation helpful? Give feedback.
All reactions