Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Pinned Hashes for JEDI Repositories #433

Merged
merged 28 commits into from
Oct 17, 2024

Conversation

asewnath
Copy link
Contributor

@asewnath asewnath commented Sep 23, 2024

Description

Adds two options for using pinned versions in swell experiments:

  1. Adds pinned_create jedi build method. Calling this method will clone jedi repositories with hashes in pinned_versions.yaml. Then it will build jedi and proceed as normal.
  2. Adds use_pinned_existing jedi build method. This will point to build and source directories in /discover/nobackup/projects/gmao/advda/jedi_bundles or /discover/nobackup/projects/gmao/advda/jedi_bundles_sles15 for milan nodes. This build is currently pinned to August 31st. The check_hashes script is called to check if commit hashes in this build lines up with pinned_versions.yaml. If it does not, it will abort. If it does, it will proceed as normal.
  3. Quick change for edit documents link

Note that none of these options are default and they must be indicated through the CLI.

Notes

  • Automatically building jedi in local directory if use_pinned_existing fails will be addressed in another PR
  • Github action will be addressed in different PR

Issues

Resolves #423
Addresses some concerns in #345

@asewnath asewnath marked this pull request as draft September 23, 2024 21:27
@asewnath asewnath marked this pull request as ready for review October 9, 2024 19:56
Copy link
Collaborator

@ashiklom ashiklom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

In addition to the stuff in my review, one more requested change (I couldn't add a suggestion because it's too far from any code you touched): In src/swell/tasks/task_questions.yaml line 466 (definition of jedi_build_method), we need to add the new valid options.

Also, if we do that, is there a way to enforce a particular set of options at runtime? I.e., When you go to set self.config.jedi_build_method, if it's set to something invalid, it'll fail immediately? Not sure if the string-drop-list or some other type will do that for us. In any case, if there's a way to easily do that, we should do it, because that will allow us to remove lots of if self.config.jedi_build_method() in (...) code blocks.

src/swell/tasks/build_jedi.py Show resolved Hide resolved
src/swell/test/platform_tests/__init__.py Outdated Show resolved Hide resolved
src/swell/utilities/build.py Show resolved Hide resolved
src/swell/tasks/clone_jedi.py Show resolved Hide resolved
Dooruk
Dooruk previously approved these changes Oct 10, 2024
Copy link
Collaborator

@Dooruk Dooruk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good!

We just need to ensure the consistency of pinned_versions.yaml and the following locations:

existing_jedi_build_directory:
default_value: /discover/nobackup/projects/gmao/advda/swell/JediBundles/fv3_soca_SLES15/build-intel-release
existing_jedi_source_directory:
default_value: /discover/nobackup/projects/gmao/advda/swell/JediBundles/fv3_soca_SLES15

@asewnath
Copy link
Contributor Author

Thanks, looks good!

We just need to ensure the consistency of pinned_versions.yaml and the following locations:

existing_jedi_build_directory:
default_value: /discover/nobackup/projects/gmao/advda/swell/JediBundles/fv3_soca_SLES15/build-intel-release
existing_jedi_source_directory:
default_value: /discover/nobackup/projects/gmao/advda/swell/JediBundles/fv3_soca_SLES15

I modified the platform task_question.yaml files to point to pinned version jedi bundles for SLES12 and SLES15.

@asewnath
Copy link
Contributor Author

Also, if we do that, is there a way to enforce a particular set of options at runtime? I.e., When you go to set self.config.jedi_build_method, if it's set to something invalid, it'll fail immediately? Not sure if the string-drop-list or some other type will do that for us. In any case, if there's a way to easily do that, we should do it, because that will allow us to remove lots of if self.config.jedi_build_method() in (...) code blocks.

In task_questions.yaml, I modified jedi_bundle_method to include the new valid options, and then I changed it to string-drop-list which works well for constraining the user to only valid build methods. If there are any remaining if self.config.jedi_build_method() in (...) code blocks, it's because the task itself is checking for specific build methods and aborts if other valid methods were provided. For example, BuildJediByLinking aborts if the user has selected either the create or pinned_create build method.

@asewnath asewnath merged commit 5f8f864 into develop Oct 17, 2024
2 checks passed
@asewnath asewnath deleted the feature/pinned_versions_support branch October 17, 2024 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(SE) "Edit documents" link is broken in docs
3 participants