Issue with -p or --threads

[Darkyere]

No matter what i do or try i cant make it only use a certain amount of threads.

I am trying to make a test example for a Enhancement propotion for Proxmox sepecifically the BackUp part.

Now my command is as following

lrzip -L 9 -p 4 -o /Proxmox-BackUp/Manual-vzdump/102-Kubuntu-Plasma-BTRFS-RAID0-lrzip-L9-P4.lrzip

Now it creates between 8 and 13 threads.
I allso tried out.

lrzip -L 7 --threads 4 -o /Proxmox-BackUp/Manual-vzdump/102-Kubuntu-Plasma-BTRFS-RAID0-lrzip-L7-P4.lrzip

Same result. 8 to 16 Threads.
And hogging 800 % of CPU usage.
100% being one full core.

This is an 4 cores, 8 threads system.
So i would like it to preferable use 400% of the CPU.

I am trying to not make it not take to much CPU usage by setting a thread count, since Proxmox rely on ZFS and is a VM host platform.

It is a dire need to be able to set a Thread count to exactly what a person needs.

Is there any way you can help me with this ?

it is version

lrzip --version
lrzip version 0.651

Best regards,
Darkyere

[pete4abw]

Out of curiosity, try running as root or sudo. May be a permissions thing.

[Darkyere]

Funny enough I actually has elavated the terminal with sudo permission because it is needed for the Proxmox vzdump tool which I am piping to lrzip.
So I should already have sudo permissions.

Thank you for looking into it.

[pete4abw]

Try not piping the command. Use `lrzip` directly and see.

…

On August 8, 2023 11:13:34 AM PDT, Darkyere ***@***.***> wrote: Funny enough I actually has elavated the terminal with sudo permission because it is needed for the Proxmox vzdump tool which I am piping to lrzip. So I should already have sudo permissions. Thank you for looking into it. -- Reply to this email directly or view it on GitHub: #247 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[Darkyere]

I tried not piping just to get an idea if this would make a difference.
Might as well try i suppose, unfortunately it did not work.

I do not know how else to make this example then lrztar.
lrzip requires to pipe through tar and the idea was not to pipe it.

vzdump has no other way than piping the output.

And with both vzdump... | lrzip... it seemed to not use the p4 threads as my fist post.

In this example with lrztar
lrztar -L 9 -p2 -o /home/proxmox/Compress\ example/Test1-lrzip-L9-p4.lrzip /home/proxmox/Compress\ example/Syncthing/
As the image shows it did not use only 2 threads and 200% CPU usage.
It takes anything that is available to it.

This is done on two seperate debian 12 systems.
To make the test more interesting.
That is why it is with 4 and 2 threads.

Is there anything here that makes sence to you?

[pete4abw]

If you pipe, the source application will use threads, and lrzip will. Do not use lrztar. It's bloated and uses tar which uses threads and finally you pipe to lrzip. Try this to eliminate variables. Pipe your dump to tar. Then run `lrzip -p4 -L1 tarfile.tar` (L1 to save time) and monitor that use. If thread use is 5 (one thread is for rzip) or 4 then you cannot reliably pipe and control thread use. That's all I can do here. I maintain a different variant. Good luck.

…

On August 9, 2023 5:15:57 AM PDT, Darkyere ***@***.***> wrote: I tried not piping just to get an idea if this would make a difference. Might as well try i suppose, unfortunately it did not work.  I do not know how else to make this example then lrztar. lrzip requires to pipe through tar and the idea was not to pipe it. vzdump has no other way than piping the output. And with both `vzdump... | lrzip...` it seemed to not use the p4 threads as my fist post. In this example with lrztar `lrztar -L 9 -p2 -o /home/proxmox/Compress\ example/Test1-lrzip-L9-p4.lrzip /home/proxmox/Compress\ example/Syncthing/` As the image shows it did not use only 2 threads and 200% CPU usage. It takes anything that is available to it. This is done on two seperate debian 12 systems. To make the test more interesting. That is why it is with 4 and 2 threads. Is there anything here that makes sence to you? -- Reply to this email directly or view it on GitHub: #247 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[Darkyere]

I can see making a tar actually makes it respect the -p4 threads.

But i am fairly certain that is not gonne be possible for Proxmox.
I had dealy hoped so.
They dont create a tar, then compress it.
It seems they are using vzdump to copy the data to their compression module on the fly.

Would of course never have been certain they would have implemented it.
But i do feel that proxmox is lacking some space conserving compression and not a "fast as possible" compression.

[pete4abw]

vzdump 100 --compress 0 --stdout | tar -O -cf - | lrzip -L 9 -p4 -o /Proxmox-BackUp/Manual-vzdump/vzdump-tar-lrzip-L9-p4.lrzip

You completely missed the point and my directions. In order to restrain threads, I advised you to NOT pipe to lrzip. You continue to do so. What you need to do is this:

vzdump 100 --compress 0 --stdout | tar -O -cf vzdump.tar
then
lrzip -L 9 -p4 -o /Proxmox-BackUp/Manual-vzdump/vzdump-tar-lrzip-L9-p4.lrz vzdump.tar

If you do not understand pipes, then linux administration may not be right for you.

[pete4abw]

Ah, I just remembered, looking at the lzma code. Each backend in lrzip may use threads as well. lzma may use up to 32 threads. However, the -p option in lrzip will control how large the block of data sent to the backend will be. Example:

If a chunk of data is 90:

• And the number of threads is 9, then each data block sent to the backend will be 10 (90/9).
• And the number of threads is 4, then each data block sent to the backend will be 22.5 (90/4).
• And the number of threads is 2, then each data block sent to the backend will be 45 (90/2).

This, of course, if memory and sliding window permits. lrzip or lrzip-next cannot control the backend use of threads over and above what lrzip uses.

The larger the data block, the better backend compression may be because it has more data to look ahead at.

[Darkyere]

I know I started by making a mistake by misunderstanding how to pipe it the right way.
But I then edited the original thread yesterday after my mistake.

I did as you just wrote made an vzdump to a .tar file.
And the used lrzip to compress the tar file.

And it actually worked.
It respected the 4 threads as you made a theory about.

Plz. Have a look at the edited post above your last post.

I didn't write the command I used when I edited, maybe I should have.

But your idea worked.
I appreciate your time.
I just do not believe I could ever convince the proxmox team to implement it in this way.

But I still find lrzip as an excellent and useful compression tool.

Bedt regards,
Darkyere

[Darkyere]

Funny we wrote at the same time.

I will have a look at what u just posted later.
I am unable to sit down ATM. Since I am on a phone.

[pete4abw]

Yes. it appears that threads are uncontrolled when passed to a backend. This will be an interesting problem since we do not want to constrain the backend from doing its work. I am not sure if this is controllable atm. Good luck, I am done here.

[pete4abw]

Yes, I was right. If you really want to constrain threads while using lzma, you must use the make option -DZ7_ST (for lzma sdk 23.01 and lrzip-next), -D_7ZIP_ST for lrzip. This will force lzma to NOT use multi threading. So, lrzip/lrzip-next is performing as designed. However, if you do constrain backend threads, you will a) hinder performance, and b) maybe reduce compression.