Hi Steve,

...

Who's to say that one doesn't have the same basic issue? Same with any other project in FOSS for that matter.

That's the idea I was trying to make. There are no guarantees are there? But you can minimize the social problems.

The 'basic issue' I see is the "one or two" developers, some that nobody knows in person, vis-à-vis "many" developers on a big project.

For me it's most important when the project is on a Distro critical- or security-path.

Cheers! Arnie

On Thursday, April 4th, 2024 at 9:41 AM, Steve Cossette farchord@gmail.com wrote:

I have definitely not read 75% of the comments and articles about the xz issues but I understand the general reason why this happened.

Issue here is, let's say we do switch to an alternative, whatever it is. Who's to say that one doesn't have the same basic issue? Same with any other project in FOSS for that matter.

I'd say keep using XZ if the maintainers are quick to fix issues and quick to respond to the community's issues, this one for example. Everyone does mistakes. It's fine as long as we learn from them.

On Thu, Apr 4, 2024 at 9:26 AM Arnie T via devel devel@lists.fedoraproject.org wrote:

...

Hi,

I just installed Fedora on 2 of my PCs a couple of weeks ago. One version of Fedora 39 release and one of Fedora 40 to see where things are going.

I learned about this XZ-hack from Ars Technica & The Economist.

I got to the Fedora Magazine article and wasn't really clear on that.

So I followed the discussion to this thread in this Development mailing list.

I read a lot of it but _still_ can't 100% figure out what the final solution is going to be.

I have a question about that.

I'm for sure OK that a responsibly developed FOSS project can contribute value and should be welcomed.

ISTM that if a package is used on critical-path or security-path by default in a Distro it needs a higher bar.

IIUC from this thread and online discussions about XZ & alternatives that

1] Lack of committer 'Real' identity confidence and verification is a problem. 2] Undetected differences source + packaging in repo vs tarballs are unchecked. 3] Under-resourced development creates risk; 'Many eyes' bench depth in development is needed. 4] XZ has a single, unsupported committer. 5] ZSTD is developed & used at Facebook. 6] ZSTD matches or outperforms XZ and most other compression in most metrics. 7] ZSTD is already used for default compression by Distros.

I get that there's never going to be 100% perfect solution.

But wouldnt' switching Fedora from using XZ to ZSTD by default fix a lot of the uncertainty around at least this current issue?

Is that being considered in Fedora? Or is the focus trying to fix XZ to continue to use it?

Thanks for any help to understand all this :-)

Cheers!

Arnie

---

devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

Hi,

See also an upstream GNU discussion on whether more GNU packages should start providing zstd, or even lzip, tarballs in addition to xz: https://lists.gnu.org/archive/html/bug-standards/2024-04/msg00032.html

I'm sure not going to tell any developers here something they don't know! But for anybody that's just starting to look at this I thought these were really helpful.

https://manishrjain.com/compression-algo-moving-data

https://linuxreviews.org/Comparison_of_Compression_Algorithms

From both of those I get that ZSTD is a pretty good option to consider.

Cheers!

Arnie

On Thu, Apr 04, 2024 at 01:45:45PM -0000, Daniel Alley wrote:

As long as there are existing xz-compressed files in the wild, Fedora will need to support consuming them - as long as there is software that expects xz compression, Fedora will need to support creating them. It's not going to disappear any time soon, and until then we're stuck with xz

There's also the issue that liblzma is widely used and offers specific features which zstd does not[1].

Rich.

[1] https://github.com/facebook/zstd/issues/395#issuecomment-535875379

On Thu, Apr 04, 2024 at 02:40:08PM +0000, Arnie T via devel wrote:

Hello Rich,

...

There's also the issue that liblzma is widely used and offers specific features which zstd does not[1].

[1] https://github.com/facebook/zstd/issues/395#issuecomment-535875379

Is that about this?

https://github.com/facebook/zstd/tree/dev/contrib/seekable_format

If that was part of zstd or even actively being looked at, then yes.

From a Distro decision viewpoint does an alternative like ZSTD have to solve all the problems XZ does to be considered?

There's no such thing as a "distro decision" on this one, as was explained in the thread already.

Rich.

On Thu, 4 Apr 2024 at 11:22, Arnie T via devel < devel@lists.fedoraproject.org> wrote:

Hi,

...

There's no such thing as a "distro decision" on this one, as was explained in the thread already.

I'm sure the 'explanation' is all clear to you and the other Developers.

I'm also sure that it's not all that clear to non-Developers.

If the explanation was clear and obvious to me, here or anywhere ales, I wouldn't be asking the question.

So, sorry, I guess.

I guess I don't understand how a Distro decision is different from a Distro IN-decision.

Linux distributions are generally 'collective anarchies' where most packages are up to the individual packagers to support how they want things. 'Collectively' they will elect some group who will work out various high level things like 'where should the files go?' 'how should the files be packaged', 'what should we call ourselves', etc. These decisions may override what the upstream (aka the people who write the compiler, kernel, compression libraries, graphics, etc) but again it is usually a compromise.

How a decision to drop xz for some other compression library for software would be a fairly slow process. First a person who is willing to do the work would come up with a proposal on why it should be done and how it could be done. They would be expected to also test to see how much trouble this would be (aka find all the packages which use xz and could be changed to another library, which ones couldn't and what the effects would be.) Once that is done, they would make a general proposal to be reviewed by whatever technical committee a distribution has (Fedora has one whose acronym is FESCO, Debian has another or multiple others, etc). This would be reviewed and if accepted it would go as a future release work with a staged plan where some packages are moved in X release, some in X+1, and some final plan for X+2 (or backed out completely for some reason before then). There would be some amount of software which would rely on xz no matter what because either the upstream has no interest in changing or it is meant to use xz period.

Usually this would take a key person to drive it who understands the problem and a team of people who would be interested in actually doing the work. Without that the change will move slowly over many releases like the licensing change to SPDR.

This is how a change would happen if it were decided to be done. At the moment the distributions are first trying to figure out a couple of more important things: 0. What happened? 1. What else might have been affected 2. What projects that are also in similar straights 3. What can be done to help these projects (is it inside our sphere of control or influence). 4. What is a list of things that need to change. ... N. Is moving off of these projects needed or possible?

Currently most groups are between 0 and 1. There are a lot of things which need to be looked at before moving off can be looked at as a goal to make sure we aren't making things worse.

I hope the above helps

For example from what I can read you were in contact with this Jia Tan 'person' during this story.

I hope that a Distro decision would support whatever it takes to give you the tools, time and support to make sure that this sort of thing doesn't sneak past you or anyone.

If there's no way to make that kind of decision then it seems to me Developers could use better support.

This all seems like a very big deal. Which is why I guess I am reading about it on The Economist. And why I'm hoping that the Distro has some options to take some actions.

Cheers!

Arnie

---

devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

On Thu, 4 Apr 2024 at 12:21, Arnie T via devel < devel@lists.fedoraproject.org> wrote:

Hello Stephen,

How a decision to drop xz for some other compression library for software would be a fairly slow process. First a person who is willing to do the work would come up with a proposal on why it should be done and how it could be done. They would be expected to also test to see how much trouble this would be (aka find all the packages which use xz and could be changed to another library, which ones couldn't and what the effects would be.) Once that is done, they would make a general proposal to be reviewed by whatever technical committee a distribution has (Fedora has one whose acronym is FESCO, Debian has another or multiple others, etc). This would be reviewed and if accepted it would go as a future release work with a staged plan where some packages are moved in X release, some in X+1, and some final plan for X+2 (or backed out completely for some reason before then). There would be some amount of software which would rely on xz no matter what because either the upstream has no interest in changing or it is meant to use xz period. ... Currently most groups are between 0 and 1. There are a lot of things which need to be looked at before moving off can be looked at as a goal to make sure we aren't making things worse.

I hope the above helps

Thanks, I understand more of your explanation of how it's done.

I don't know how much time was needed to decide for example an Arch Distro change

"Now using Zstandard instead of xz for package compression"

https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-com...

So that is an individual package choice a distribution maintainer(s) can make. In this case the pacman maintainers decided to use a different library for their packages. It doesn't change anything outside of that one tool though. It is also not getting rid of xz from Arch. They will need to keep xz around because older systems will have used the older compression and pacman and similar tools will need to 'read' that. It mainly means that newer packages will use zstandard versus xz.

A similar change in Fedora would be that rpm uses zstandard by default etc. However rpm would need to keep xz because of 10 years of using xz as a compression standard in various RPMs and people need to install older software.

OK, that's my mistake. I thought that moving to open source Linux OS Distro like Redhat-related Fedora would result big or important issues can be fixed more efficiently than at Microsoft.

Decisions are people issues and people issues move at people speeds. There are about 1600 packagers in Fedora and I think 22,000 packages. Changes take time to communicate, understand and implement. The worst thing to do in a security situation is actually move too fast because you think you are getting ahead of the attacker. I have seen too many times where the attacker was waiting for said move and it makes their life easier. In this case, a bit of time is needed to really get an idea of what else is screwed up and where we need to fix things.

I guess I'm learning that even important or wise choices (not saying _ this_ is) can't be done with taking a long time. Even if they are security related issues.

Thanks one more time for the nice explanation!

Cheers!

Arnie

---

devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

On 4/4/24 14:04, Arnie T via devel wrote:

Hi Stephen,

Thanks for the explanation.

I just caught up with the article at the New York Times,

Did One Guy Just Stop a Huge Cyberattack? https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html

And the comic that looks like it fits the problem I'm most noticing here!

https://xkcd.com/2347/

I have to admit that I still don't know what the best or most official "At least do this" instruction page is for a Fedora user. I don't see anything at the main https://fedoraproject.org/ website or its "News & Announcements" page.

TL;DR: as with most security issues, end users should update their systems.

I think you may be caught in some news exaggeration. Don't get me wrong, this hack was a huge thing, but it was discovered early enough that most (i'd guess almost all) fedora users wont' have to do anything.

For Fedora, the problem package was only in Fedora 40 Beta and Fedora Rawhide. If you are not running these packages, this isn't more than a "wow, that was a near miss" for the end user. If you are running either version, the xz maintainer has already rolled back the problem update, so if you use "dnf update" you are safe.

Because of a stroke of luck (finding this as early as we did) its as simple as that, we have an assumed good version that users can 'update' to, and beyond that, us developers need to verify that the assumed good version is actually good, and if it isn't, issue new updates.

In this thread its becoming about the details of the process.  But not yet about a solution.  All of which I get. And in private emails people are insisting on sending to me about how I'm unreasonable for asking the questions, and "should have" understood this or that.

So, with your discussion the best guess I can some up with is to make sure XZ is downgraded and just hope that one of this Jia Tan's 6000+ commits are still hidden in some other project with not enough eyes.  Or that the XKCD coming true doesn't happen again.

Cheers!

Arnie

-- _______________________________________________ devel mailing list --devel@lists.fedoraproject.org To unsubscribe send an email todevel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue

On Thu, Apr 04, 2024 at 05:04:20PM +0000, Arnie T via devel wrote:

Hi Stephen,

Thanks for the explanation.

I just caught up with the article at the New York Times,

Did One Guy Just Stop a Huge Cyberattack? https://www.nytimes.com/2024/04/03/technology/prevent-cyberattack-linux.html

And the comic that looks like it fits the problem I'm most noticing here!

https://xkcd.com/2347/

I have to admit that I still don't know what the best or most official "At least do this" instruction page is for a Fedora user. I don't see anything at the main https://fedoraproject.org/ website or its "News & Announcements" page.

The magazine article should cover this.

If you are using Fedora 38 or Fedora 39, nothing to do. The versions affected were never in there.

If you are using Fedora 40 (prerelease) or Rawhide you should urgently update. This will get you the clean version. If you wish to be extra cautious, you could reinstall from current nightly media.

In this thread its becoming about the details of the process. But not yet about a solution. All of which I get. And in private emails people are insisting on sending to me about how I'm unreasonable for asking the questions, and "should have" understood this or that. So, with your discussion the best guess I can some up with is to make sure XZ is downgraded and just hope that one of this Jia Tan's 6000+ commits are still hidden in some other project with not enough eyes. Or that the XKCD coming true doesn't happen again.

Lots of folks are scrutinizing those commits. There were some minor things discovered, but nothing (at least that I know of right now) that affects Fedora.

There are changes coming in systemd, openssh and other places that would make this particular vector harder/impossible also.

kevin

Once upon a time, Simon Farnsworth simon@farnz.org.uk said:

Fedora made the same switch back in Fedora 31, and thus doesn't need to do anything about package compression right now.

About this... I was looking at RPMs and found there are a couple of packages that override _binary_payload in the SPEC to use xz: kernel and ceph. It appears they're doing it to get parallel threads (I guess not specifically to override the format itself), but should they be changed? The kernel SPEC actually mentions it might need to be changed if the global default is changed.

Also - if the reason to override it is to get threads, is there any reason to not always use threads? ceph is doing:

%global _binary_payload w7T%{_smp_build_ncpus}.xzdio

Assuming the zstd backend supports the threads option (I haven't looked), it seems like embedding T%{_smp_build_ncpus} might be a reasonable thing to always do (to go along with how make_build already uses %{?_smp_mflags}).

In the case of ceph, it's overriding _source_payload as well, which seems unwanted (feels like somebody just grepped and copied).

On Thu, Apr 4, 2024 at 8:00 PM Arnie T via devel devel@lists.fedoraproject.org wrote:

Hello Kevin,

...

I'm hopeful some things will come out of this as it's a chance for us to look at our processes and improve them.

I'm glad that's happening. It seems to me that improving those processes would be Distro decisions. Which I keep understanding don't really exist. At least not quickly. So I'm confused still. But glad.

...

...

1] Lack of committer 'Real' identity confidence and verification is a problem.

...

IMHO this isn't a problem. We don't have a right to demand anything from open source projects. We can ask, we can urge, we can contribute and change things, we can choose to not use something, or fork something.

I really don't suggest 'demanding' anything. I do think it's wise to make choices to avoid it. Like just my example of a critical-path XZ with one developer vs a critical-path ZSTD built & maintained in a Facebook FOSS project.

I know from just a business view I would never enter into a 'critical' contract without "knowing" the principal persons. Of course you must know what you need "knowing" to be.

Careful, for now you're approaching another scalability issue of the community development model. I mean, as chill as he is, even Daniel Stenberg [1] must have a limit on the amount of beers he can handle.

...

...

2] Undetected differences source + packaging in repo vs tarballs are unchecked.

Yeah, a lot of the discussion has been in this area.

I'm wondering if perhaps we shouldn't revisit source-git, or at least a variant of it where we keep the upstream sources in a branch always and apply packaging on top of that and build from there.

I'm not sure what the packaging tools and rules are here. It seems to me that repo source with an attested commit (signature? published hash?) can serve as the one source of truth. Then users can pull the commit or the on-demand API-generated tarball. I guess that could be subject to for example Github's or Gitlab's API tarball generators being hacked. But that seems less probable of a concern.

...

...

3] Under-resourced development creates risk; 'Many eyes' bench depth in development is needed.

Yep. I think also visibility of changes can be improved. So, maintainers know more about whats in a new version and how it works.

You can implement tools to increase the visibility for sure. And procedures.

Also just the "given enough eyeballs, all bugs are shallow" that comes with using a larger project helps.

Thanks for the discussion.

Cheers!

Arnie

[1] https://daniel.haxx.se

On Thu, Apr 04, 2024 at 08:11:42PM +0200, Leon Fauster via devel wrote:

One approach that would be at least bring some light into "weak" (non technical layer) components (albeit not sure how feasible it is), could be:

• Checking the resources of a packaged project. Resources in terms of man or firm power that backup the project

• Contribution activity of people

• General activity of the project

• Transparency of the workflow / tools

and that for all projects that the distribution includes.

Why? This would allow to plan internal review activities (or processes) more effectively. They would be directed to the "weak" components with higher priority (recurrent, actions).

Like the current process for checking the license (SPDX) of a project, it could also collect such metrics right away.

Well, as others have noted there is already OpenSSF scorecards.

I agree it's good to know this info, and for maintainers that have a ton of packages they maintain, it might be good to be able to look at this to remind them. For maintainers with fewer packages, they likely already just know this from interacting with the upstream project already.

I don't think we can or should use that for things like deciding if we allow packages into the collection or the like, there's a lot of ways a low score there could not matter or be non represenative of what the project is like.

kevin