-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cuesubmit jobs from config file #1284
base: master
Are you sure you want to change the base?
Cuesubmit jobs from config file #1284
Conversation
|
9e85fe0
to
6cfd785
Compare
This branch is failing on CLA for some reason. @KernAttila I know it has been a while, but can you try following the CLA process again? |
47010d1
to
91483f0
Compare
It looks like some errors emerged after merging master into this branch. Unit tests are failing with:
|
@DiegoTavares no worries, I'm currently fixing those. It's a bit hard to come back to this code after so long ^^ but we'll get there ! |
…b `config_file` + error handling
I tested a lot of scenarios, everything seems up to expectation with this feature. IMPORTANT NOTE: I added the "Override Cores" feature in this branch as well, it goes with #1296 (rqd) and #1313 (cuebot) |
pyoutline/outline/backend/cue.py
Outdated
cores = None | ||
if layer.get_arg("overrideCores"): | ||
logger.info("%s is set to override service cores.", layer.get_name()) | ||
if layer.is_arg_set("cores") and layer.is_arg_set("threads"): | ||
logger.warning("%s has both cores and threads. Use cores.", layer.get_name()) | ||
sub_element(spec_layer, "cores", "%0.1f" % (layer.get_arg("cores"))) | ||
elif layer.get_arg("threads"): | ||
sub_element(spec_layer, "cores", "%0.1f" % (layer.get_arg("threads"))) | ||
cores = "%0.1f" % float(layer.get_arg("cores") or layer.get_arg("threads", 0)) | ||
sub_element(spec_layer, "cores", cores) | ||
else: | ||
logger.warning("%s will use service cores.", layer.get_name()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking back at this code, it introduces a mandatory argument "overrideCores" that will disrupt existing implementations relying only on pycue.
Another approach could be to test if "cores" is not None, meaning we explicitly override the job's cores value over the service one, thus making current codebases retro-compatible with this feature.
If "cores" is None, then we let the selected service handle that value.
What do you think @DiegoTavares @bcipriano @lithorus ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this change would break our pipeline today. I'd rather have it implemented as you suggested to make it retro-compatible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change is done and tested.
overrideCores
stays in cuesubmit.
I don't see a problem in keeping #1296 and #1313 on this PR, as long as they get approved and merged beforehand |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aside from the suggested change, this MR is ready to be merged.
pyoutline/outline/backend/cue.py
Outdated
cores = None | ||
if layer.get_arg("overrideCores"): | ||
logger.info("%s is set to override service cores.", layer.get_name()) | ||
if layer.is_arg_set("cores") and layer.is_arg_set("threads"): | ||
logger.warning("%s has both cores and threads. Use cores.", layer.get_name()) | ||
sub_element(spec_layer, "cores", "%0.1f" % (layer.get_arg("cores"))) | ||
elif layer.get_arg("threads"): | ||
sub_element(spec_layer, "cores", "%0.1f" % (layer.get_arg("threads"))) | ||
cores = "%0.1f" % float(layer.get_arg("cores") or layer.get_arg("threads", 0)) | ||
sub_element(spec_layer, "cores", cores) | ||
else: | ||
logger.warning("%s will use service cores.", layer.get_name()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this change would break our pipeline today. I'd rather have it implemented as you suggested to make it retro-compatible.
**Link the Issue(s) this Pull Request is related to.** Fixes #1297 **Summarize your change.** As in many render engines, we should be able to set a negative core requirement. minCores=8 > reserve 8 cores minCores=0 > reserve all cores minCores=-2 > reserve all cores minus 2 This PR addresses this feature by handling negative core requests. Cuebot will try to match this number against the number of cores on each host. The frame will be booked only if all cores are available in this scenario. If the host is busy (even slightly), the frame is **not** booked, to avoid filling the remaining cores. **Testing** I would need some guidance to create proper tests for cuebot. **Screenshot** ![negative_cores](https://github.com/AcademySoftwareFoundation/OpenCue/assets/5556461/d9c4400c-824a-40cc-9ba9-2f76a3fd8ceb) Update: There is now a "ALL" text for zero cores, or "ALL (-2)" for negative cores reservation. ![core_reservation](https://github.com/user-attachments/assets/88802b15-3ccd-4cb5-90b7-58e532523ae6) (cuesubmit feature in another PR #1284) --------- Signed-off-by: Kern Attila GERMAIN <[email protected]>
…g if "cores" is not None. If it has an explicit value (like in existing implementations), this means we want to override the value, which is the default behavior. Otherwise, we let the server assign the value of selected service.
…s" value is None.
…s" value is None.
Signed-off-by: Kern Attila GERMAIN <[email protected]>
… the args are set, because checking their value would be False for 0 (all cores)
…Wolf-Pipeline/OpenCue into cuesubmit-jobs-from-config-file
Link the Issue(s) this Pull Request is related to.
#1275
Summarize your change.
Note
Question
Screenshots