Documentation about the metrics.

[josc-byte]

Hello jrmadsen,

first I want to say: really really great project! I found amazing use for it in optimizing or CI builds. Sadly I can not find any documentation about what the metrics represent exactly. Whats the difference between "virtual memory" and "peak_rss"?

    "peak_rss": {
        "value": 412.892,
        "repr": 412.892,
        "laps": 29,
        "unit_value": 1000000,
        "unit_repr": "MB"
       },
       "page_rss": {
        "value": 5.091328,
        "repr": 5.091328,
        "laps": 0,
        "unit_value": 1000000,
        "unit_repr": "MB"
       },
       "virtual_memory": {
        "value": 18.75968,
        "repr": 18.75968,
        "laps": 0,
        "unit_value": 1000000,
        "unit_repr": "MB"
       },

one of my logs created by the current master branch. If would have expected virtual_memory to be higher than peak_rss if this would be the VSZ, so its especially confusing to me.

much greetings!
Jonas

[jrmadsen]

first I want to say: really really great project! I found amazing use for it in optimizing or CI builds.

Thanks, glad to hear it helped.

Sadly I can not find any documentation about what the metrics represent exactly.

Thanks for bringing this to my attention. Most of the info about the metrics can be found in the timemory docs (I'll update the docs to link to this sometime soon). However, the [current] docs on virtual memory don't provide anything useful. You can find a more detailed description about virtual memory here but here is the relevant part:

Linux supports virtual memory, that is, using a disk as an extension of RAM so that the effective size of usable memory grows correspondingly. The kernel will write the contents of a currently unused block of memory to the hard disk so that the memory can be used for another purpose.

So, effectively, the virtual memory metric will be the max of how much memory was offloaded to disk (i.e. swap). The tendency to use swap is determined by the OS (via the swappiness setting). The "virtual memory" metric is calculated by sampling the value reported in the /proc/<PID>/statm file here.

Whats the difference between "virtual memory" and "peak_rss"?

• virtual memory: high-water mark of how much memory was offloaded into swap
• peak RSS: high-water mark of how much RAM memory was used

[jrmadsen]

One quick caveat: the peak RSS metric is provided by the OS and not sampled so it will always be accurate. The virtual memory could theoretically be inaccurate if the virtual memory usage surged and then decreased in-between the samples (however, in practice, this is quite unlikely to happen since the OS usually doesn't/can't move data in/out of swap this quickly).

[josc-byte]

Thanks for your fast reply and clear and detailed explanation, I really appreciate it. That exactly fits my use-case.

I am hoping I find the time and can split my code i am currently building in an appropriate way, to open source it, or parts at least. Its hard as it is deeply integrated into our CI. But maybe I can do a little write-up on how we use your tool in our toolchain if you are interested. It works great on our project. 👍

[jrmadsen]

But maybe I can do a little write-up on how we use your tool in our toolchain if you are interested. It works great on our project

Sure. Could probably go in a Wiki page or something.

FWIW, here are some examples of how it is used elsewhere:

• timemory CTP config
• timemory CTP ctest
• pykokkos-base CTP config
• pykokkos-base CTP Jenkins Generation
• pykokkos-base CTP Jenkins Artifacts

Most CI systems (Travis, CircleCI, Appveyor, etc.) have a concepts of artifacts so, if you aren't already doing something similar, what I did with Jenkins could be similarly applied to most CI runners.