-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
capture basic performance data from jobs #283
Comments
@levinas For the input data size, is this measured in bytes or in number of reads ? For the time, I suppose this can be captured by the Python code, using the elapsed time between the For assembly/recipe, this is obviously already available in the Python code. Is this a string ? For memory usage: GNU time and tstime can report peak memory usage and other related metrics, but I don't know if they capture the information concerning the children of the main process. |
|
Can we grab the PID from the subprocesses and poll memory usage? Not sure if this is the best way. |
Can we implement something like a conditional pull for the compute nodes? If the data set is small, for example, the control node can tag it "small", and it could be consumed by a regular VM with 24GB memory. This is what Chris envisioned in the original architectural diagram. |
Yes, I'll have to double check, but the idea is that nodes can subscribe to On Wed Feb 11 2015 at 9:08:51 PM Fangfang Xia [email protected]
|
In the callback method in consume.py, the json payload is received. Does the tag need to be specified in channel.basic_consume ? |
Minimally a four-tuple for each assembly job:
This data will be used to prepare for regular worker nodes devoted to small jobs.
The text was updated successfully, but these errors were encountered: