Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Changes/Switch RPMs to zstd compression #11

Closed
marxin opened this issue Jun 3, 2019 · 31 comments · May be fixed by #26
Closed

[RFC] Changes/Switch RPMs to zstd compression #11

marxin opened this issue Jun 3, 2019 · 31 comments · May be fixed by #26

Comments

@marxin
Copy link
Contributor

marxin commented Jun 3, 2019

Our friends in Fedora are considering to use zstd for rpm:
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

I would like to start discussion whether we as SUSE want to do the same? For me, the most interesting point is very decompression of zstd and possibility of a parallel compression that can be useful for packages like Firefox.

@marxin
Copy link
Contributor Author

marxin commented Jun 3, 2019

@scarabeusiv

@mlschroe
Copy link
Member

mlschroe commented Jun 3, 2019

You'll need to add zstd support to deltarpm first.

@marxin
Copy link
Contributor Author

marxin commented Jun 3, 2019

You'll need to add zstd support to deltarpm first.

Sure, I'm willing to implement that if that's the only blocker.

@mlschroe
Copy link
Member

mlschroe commented Jun 3, 2019

(Looking at the fedora mailing list it's pretty unclear to me if they'll switch or stay with xz. It doesn't seem to be worth the trouble.)

@marxin
Copy link
Contributor Author

marxin commented Jun 3, 2019

As mention in the wiki page, they are considering to do it: Contingency deadline: 2019-10-01 (a week before the Final Freeze). But I agree with you, I would with for now.

@Conan-Kudo
Copy link
Member

I've already got a pending SR for getting zstd support enabled in rpm, in connection to the Fedora change: https://build.opensuse.org/request/show/706643

@mlschroe
Copy link
Member

(I've added zstd support to deltarpm.)

@Conan-Kudo
Copy link
Member

And apparently a different SR was accepted that turned on the zstd support in Base:System/rpm. 😢

Oh well, at least it's activated there. It just needs to be submitted to Factory now.

@Conan-Kudo
Copy link
Member

SR to Factory submitted: https://build.opensuse.org/request/show/709948

@Conan-Kudo
Copy link
Member

So FESCo approved the change proposal last week, and it's going to be implemented in a couple of weeks in Rawhide. The drpm library will be getting support for zstd DeltaRPMs soon, which unblocks zstd DeltaRPM handling in createrepo_c.

@mlschroe
Copy link
Member

(SUSE doesn't build createrepo_c with drpm support, but nevermind...)

@Conan-Kudo
Copy link
Member

@mlschroe Yes, I know SUSE ships createrepo_c crippled (despite my best efforts...), but those are the steps being taken now to switch things over in Fedora.

@Conan-Kudo
Copy link
Member

At this point, all the pieces should be in place to do this, if we want. Ideally, we'd need SUSE Linux 15 rpm to support zstd compression so that it can read and handle Factory/SUSE Linux 16 packages.

@mlschroe Can we get rpm in SLE 15 SP2 to support zstd RPMs?

@marxin
Copy link
Contributor Author

marxin commented Jan 2, 2020

I can confirm that Fedora really switches to zstd 19.

@marxin
Copy link
Contributor Author

marxin commented Jan 2, 2020

And adding @DimStar77 as the maintainer of Factory, what do you think about it?

@marxin
Copy link
Contributor Author

marxin commented Jan 5, 2020

And I've just noticed that Archlinux has done the same (zstd level 19):
https://lists.archlinux.org/pipermail/arch-announce/2020-January/000400.html

@DimStar77
Copy link
Contributor

The overall plan to switch to zstd sounds good. But before we do do in TW, we need to have at least the still supported leap versions support extracting those rpms, as otherwise we can't zypper dip from there.

Once we enable zstd, we will have to drop all zdup tests from older openSUSE versions - but that's a sour apple we will have to bite. Pity, but acceptable

@DimStar77
Copy link
Contributor

Oh, archs stats to get an increase by 0.8% for the rpms can give us some more fun with space restrictions on the DVD and live media (at least lives are always at their limits... So increasing will likely push them over the edge)

@marxin
Copy link
Contributor Author

marxin commented Jan 21, 2020

Based on the discussion with @mlschroe, the backport for Leap 15 in on the way.
I've just done SR to rpm where I changed the compression and level:
https://build.opensuse.org/request/show/766061

@mlschroe
Copy link
Member

Dominique, what compression level do you recommend?

@DimStar77
Copy link
Contributor

Dominique, what compression level do you recommend?

Really tough question - considering that even level 19 gives a 0.8% increase in size (which is likely going to blow up our ISOs) I can't imagine any lower compression level providing any better fun.

hence -> 19 (giving up all the speed gains we received by RPM 4.15 parallel compression when building LibreOffice; but the time won by install time is still worth it imho)

@marxin
Copy link
Contributor Author

marxin commented Jan 21, 2020

Really tough question - considering that even level 19 gives a 0.8% increase in size (which is likely going to blow up our ISOs) I can't imagine any lower compression level providing any better fun.

hence -> 19 (giving up all the speed gains we received by RPM 4.15 parallel compression when building LibreOffice; but the time won by install time is still worth it imho)

We should test a real impact in a staging project. That will tell more about ISO size.

@Conan-Kudo
Copy link
Member

Based on the discussion with @mlschroe, the backport for Leap 15 in on the way.
I've just done SR to rpm where I changed the compression and level:
https://build.opensuse.org/request/show/766061

grumbles that we should stop patching rpm for vendor changes and use this for it

@romulasry
Copy link

Update?

@marxin
Copy link
Contributor Author

marxin commented Jun 1, 2020

We're waiting for Leap 15.2 release and then we'll change it for openSUSE Factory.

@Conan-Kudo
Copy link
Member

Conan-Kudo commented Jun 1, 2020

Why are we waiting for Leap 15.2 to release to do it? We already know it's coming...

@marxin
Copy link
Contributor Author

marxin commented Jun 1, 2020

Yes because we guarantee that zypper dup from an older release will be updated to Tumbleweed. And 15.2 will be first release where zstd will be supported by rpm.

@romulasry
Copy link

Interesting. Post here when you get further information on this. Thank you.

@romulasry
Copy link

romulasry commented Jun 3, 2020

openSUSE Leap 15.2 is planned to be released in early July 2020, based on SLE 15 SP2. openSUSE Leap has a no Monday and Friday release policy (Alpha, Beta, RC, GA), pre-agreed day of week for releases is Thursday.

https://en.opensuse.org/openSUSE:Roadmap#Upcoming_openSUSE_Leap_Release

@romulasry
Copy link

Leap 15.2
Is Now Available

@marxin
Copy link
Contributor Author

marxin commented Jul 8, 2020

Leap 15.2
Is Now Available

The rpm changes landed to Base:System/rpm and there's a pending submit request to openSUSE:Factory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants