Add mem for all tools with cores and rescale some memory based on usegalaxy.org usage
#76
Conversation
tools.yml
Outdated
| @@ -1428,12 +1395,14 @@ tools: | |||
| mem: 24 | |||
| toolshed.g2.bx.psu.edu/repos/iuc/iqtree/iqtree/.*: | |||
| cores: 10 | |||
| mem: 7 | |||
There was a problem hiding this comment.
This is low setting for mem. Max memory for Galaxy Australia's iqtree jobs is regularly in the 8-19Gb range (19 is the what we give it in TPV)
|
Thanks @mira-miracoli and @cat-bro! I think it should be good to go now. |
|
Oh, I guess I need to apply formatting. It removes all the comments though, which IMO can be useful - @nuwang you mentioned making roundtrip the default in galaxyproject/total-perspective-vortex#140, should I do that (or make it the method used by format, at least)? Or do we want to enforce no comments in the shared DB? |
tools.yml
Outdated
| toolshed.g2.bx.psu.edu/repos/iuc/fasta_stats/fasta-stats/.*: | ||
| mem: 11.4 | ||
| toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/.*: | ||
| cores: 4 | ||
| mem: |
| toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_interlacer/fastq_paired_end_interlacer/.*: | ||
| mem: 30.4 | ||
| toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/fastq_paired_end_joiner/.*: | ||
| mem: 11.4 | ||
| toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/.*: | ||
| cores: 8 | ||
| cores: min(max(int(input_size * 4), 1), 16) | ||
| mem: min(max(int(input_size * 7), 5.6), 64) |
There was a problem hiding this comment.
I 4GB input file would already get 16 cores? That seems to be a lot. We use 3 cores and never had complains so far.
| @@ -588,8 +536,8 @@ tools: | |||
| cores: 2 | |||
| mem: 7.6 | |||
| toolshed.g2.bx.psu.edu/repos/devteam/tophat2/tophat2/.*: | |||
| cores: 10 | |||
| mem: 90 | |||
| cores: 16 | |||
There was a problem hiding this comment.
half both cores and mem? tophat should not get any priority IMHO
| toolshed.g2.bx.psu.edu/repos/iuc/chira_merge/chira_merge/.*: | ||
| cores: 1 | ||
| mem: 60 |
There was a problem hiding this comment.
@mira-miracoli if we don't have this tool in our config, we use the memory from the shared DB isn't it? So we are using 60GB I assume?
@pavanvidem since this is your tool, any recommendation?
It would be great if we can preserve comments, at least when formatting I think. However, I just left a comment on that other thread about the impending pydantic transition. |
Some of this may be controversial so I'd certainly like to hear from other usegalaxy.* admins before this is merged. =)
fastqc is now scalable cores and memory based off input_size - probably I should fix this for compressed data a la my bwa-mem2 rule but perhaps TPV should have an
estimated_input_size?Data I used to make adjustments can be found in this gist. There is also a discussion about how to do a better job of this long term in #75.
Any tool I didn't have data for that has
coresbut nomemI left a comment on. fastp I don't know what to do with, since its usage is way outside of what I allocate for it, so I might try a more restricted query for that.