-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running STEMMUS_SCOPE sensitivity analysis on Snellius with parallel computing #187
Comments
@Crystal-szj nice job in creating the issue 👍 , thanks. Here answers:
So, you need to move things around in |
|
Just to give you ideas about reading and writing config_file in python, here are some examples read_config and update_config. Use them as an example, you need to write your own functions. |
@SarahAlidoost Hi, Sarah, many thanks for your suggestions. I commented out the
Could you please help me to figure out what's wrong here? Please let me know if more information is needed. Thanks very much. |
@Crystal-szj there are several things to check:
|
@SarahAlidoost Many thanks for your advice.
import subprocess
subprocess.run(['.\exe\STEMMUS_SCOPE_SS.exe','.\config_file_snellius_sensitivity_analysis.txt']) or WSL terminal
Both of above commands work well. I think the exe file works. However, when I run |
The above problem may cause by the different operating systems (e.g. Linux and Windows). The documentation works well on the Linux system, but failed on WSL see here. In addition, the executable file generated by different systems may not be compatible. It's better to regenerate the executable file when run it on a new system. |
@SarahAlidoost Hi Sarah, many thanks for your advice. I installed a Linux system, and now the code works well. All the codes have been uploaded to EcoExtreML/STEMMUS_SCOPE_sensitivity_analysis repository. Here is some detailed information.
Please let me know if you need further information. Any insights or suggestions you can provide would be immensely helpful. Sincerely thanks for your time and support. |
I see that you commented out the |
Thanks for your prompt response and links. I understand your approach, where each site is assigned to a separate core for parallel execution. That enables the completion of 170 sites in six rounds, with 32 sites processed per round. However, considering the need for one task to run on a single core, as both you and Qianqian mentioned, I believe I should follow the 'parallel execution of serial programs' approach, where parallelism is not programmed into the STEMMUS_SCOPE model. According to this method, if I submit one task, only one CPU should be utilized, and if I submit ten tasks, ten CPUs should work concurrently. I noticed from the above screenshot that multiple cores were active, even though I just submitted only one task. Does this indicate the presence of parallelism within the executable file? My question is whether I should ensure "one task one CPU" or whether I can overlook this issue and proceed with using the loop for to run the 380 cases. Thanks again for your guidance and expertise. |
No, this is not the case except we tell the computer to run ten tasks on ten cores. It means that we should implement a method of parallelization, e.g. the loop for with the parameter |
your code is different than Qianqian's code and does not use many Python libraries. If you are just running stemmus_scope, it should use only one core except that your stemmus_scope is very different than the one in the main branch. If this is not the case, please check the code to build exe file and make sure that the argument |
@SarahAlidoost Hi Sarah, many thanks for your reply.
I would like to inquire whether this occasional CPU usage exceeding 100% and two cores activated for one case are common situations on a supercomputer?
I'm seeking your advice on any additional steps or considerations that should be taken before executing the 380 cases.Thanks again for your help and time. |
The STEMMUS_SCOPE version I used is based on version 1.1.9. And I added the plant hydraulics part as a separate function. I'd like to clarify that I have not utilized any parallel computer packages such as If you have any further questions or require more details, please let me know. Thanks for your support. |
@SarahAlidoost Hi Sarah,
I hope this message finds you well.
I want to do a sensitivity analysis on STEMMUS_SCOPE by setting different sets of parameters (run the model 380 times). I would like to utilize parallel computing to finish this part.
To begin, I have created a new executable file via STEMMUS_SCOPE_SS_exe.m that requires two input parameters: one for the config file (
config_file
) and the other for the parameters (parameter_setting_file
). The Matlab code portion has been completed.My intention is to utilize the existing run_STEMMUS_SCOPE_in_Snellius framework. If I understand correctly, I need to modify the
run_model.py
file to iterate through the input parameter file instead of the input forcing data for 170 sites.The pystemmusscope environment is activated. Now I have a couple of questions:
run_model.py
file, see here, we need to create an instance of the model. However, theparameter_file
is not an input for the StemmusScope class. Does this mean I need to modify the StemmusScope class in 'pystemmusscope' package and reinstall the package?run_model.py
requires the input ofjob_id
, is it possible for me to test this modified version on my local computer to ensure there are no bugs before submitting it to Snellius? I'm not sure how to solve this in the development phase.Please let me know if any information need be provided. I would greatly appreciate it if you could share your experience and provide guidance on how to address these questions.
Best regards,
Zengjing
The text was updated successfully, but these errors were encountered: