Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regression of cwltool performance by 5-7 minutes around 1st of june #1916

Open
azat-badretdin opened this issue Sep 29, 2023 · 9 comments
Open

Comments

@azat-badretdin
Copy link

Our PGAPX tool (ncbi/pgap on GitHub) uses cwltool as a CWL runner. We noticed that around 1st of June this year or TeamCity builds are running 5-7 minutes longer.

We tracked the issue to slow down in transition between calling a workflow and command line tool. For example:

[2023-09-29 01:38:51] DEBUG [job cluster_blastp_wnode_3] initial work dir {}
[2023-09-29 01:40:22] INFO [job cluster_blastp_wnode_3] /pgap/output/debug/tmp-outdir/kf1wjucm$ cluster_blastp_wnode \

See 1.5 min delay that used to be significantly smaller. The cwltool versions of "slow" and "fast" runs are correspondingly:

$ docker run -it ncbi/pgap-dev:2023-05-15.build4593 cwltool --version
/root/venv/bin/cwltool 3.1.20230513155734
$ docker run -it ncbi/pgap-dev:2023-06-06.build4641 cwltool --version
/root/venv/bin/cwltool 3.1.20230601100705

We directly tested that reversion of cwltool version to May value fixes the performance degradation issue.

@azat-badretdin
Copy link
Author

I have cwltool logs before and after and an Excel file comparing stage execution between the two, but I do not see a way to attach those to the ticket, please let me know how can I deliver them to you.

@mr-c
Copy link
Member

mr-c commented Sep 29, 2023

@azat-badretdin Make a zip archive, then you can drag-and-drop that into a comment here.

If you know that your workflow has no errors and is CWL v1.2 only, then --fast-parser should speed up the startup.

Do you have nodejs installed in your container?

@azat-badretdin
Copy link
Author

If you know that your workflow has no errors and is CWL v1.2 only, then --fast-parser should speed up the startup.

We did transition to CWL v1.2 some time ago. Thanks for the tip, Michael!

Do you have nodejs installed in your container?

Yes. In our May release we had

$ docker run ncbi/pgap:2023-05-17.build6771 nodejs --version   
v6.14.3

in our pending release we have

$ docker run ncbi/pgap-dev:2023-09-28.build4972 nodejs --version
v6.17.1

newer version.

Make a zip archive, then you can drag-and-drop that into a comment here.
cwl-issue-269.tar.zip

@azat-badretdin
Copy link
Author

Note that the suspect here is the volume of referenced input data, not parsing per se.

@azat-badretdin
Copy link
Author

We opened our internal JIRA ticket PGAPX-1208 to investigate cwltool --fast-parser suggestion

@mr-c mr-c transferred this issue from common-workflow-language/cwl-v1.2 Sep 29, 2023
@mr-c
Copy link
Member

mr-c commented Sep 29, 2023

We directly tested that reversion of cwltool version to May value fixes the performance degradation issue.

Everything else was the same? I'm seeing a lot of differences between the two docker images :-)

@azat-badretdin
Copy link
Author

The workflows inside docker image are the same. The binaries are different, not much, my point is that I do not see differences between [job X] execution times. If you filter the Excel file by job you won't see any large changes. All the time differences are on higher level.

@azat-badretdin
Copy link
Author

Here is a better attempt at naming steps, I also added the order of execution of stages (steps/workflows/jobs)

comparison.2b.xlsx

All increment happens in workflows/steps transitions, all job duration differences are within 1 sec

@azat-badretdin
Copy link
Author

Reminder: in case of government shutdown that might happen today at midnight, I probably won't be legally able to comment here (given that I am officially government at this account) and I will certainly not be able to actually use government equipment to answer questions here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants