Is it possible to dynamically set dagster/concurrency_key
from partitions at run time?
#26540
Unanswered
chrishiste
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a job that runs against multiple Cloud regions, each backed by a separate data store. I want to enforce a concurrency limit such that, for each region, I never exceed 10 queries running at once in a specific
@op
.However, I don’t want one region’s concurrency limit to affect another’s—each region should be governed by its own limit.
What I’m trying to achieve:
I’d like to dynamically set the
dagster/concurrency_key
for each run based on the a partition value.My partitions are dynamic (
dagster.DynamicPartitionsDefinition
), and I can’t know or hardcode all the possibledagster/concurrency_key
values at definition time.Is there a supported way to achieve this behavior?
Beta Was this translation helpful? Give feedback.
All reactions