Hi,
We run a diverse fleet of Linux laptops and desktops (at Facebook), and sometimes there are regressions that affect some of our fleet but not others.
To pick the latest example: - pulseaudio 1.3.99.1 (both -1 and -2) breaks Bluetooth support on a Dell XPS 15: https://bugzilla.redhat.com/show_bug.cgi?id=1814556 - but it fixes HDA audio input on ThinkPad T490s and X1 Carbon: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3e19f5098
We've had similar issues with kernel regressions (e.g. 5.4 kernels had issues with Intel GPUs, and on ThinkPads with Nvidia GPUs).
(Ideally we catch all this before they land -- over the medium term I'm trying to find a way to encourage our users to help test updates)
Would it be possible to keep 2 or 3 versions of the same package in the updates repo, so we can easily keep some of our fleet at a previous version known to work on that particular hardware? And is there a process for proposing this (e.g. file a ticket on Pagure)?
Our workaround right now is to check in the older versions in our internal repo.
Thanks,
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
Vít
Dne 24. 03. 20 v 1:12 Michel Alexandre Salim napsal(a):
Hi,
We run a diverse fleet of Linux laptops and desktops (at Facebook), and sometimes there are regressions that affect some of our fleet but not others.
To pick the latest example:
- pulseaudio 1.3.99.1 (both -1 and -2) breaks Bluetooth support on a Dell XPS 15: https://bugzilla.redhat.com/show_bug.cgi?id=1814556
- but it fixes HDA audio input on ThinkPad T490s and X1 Carbon: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3e19f5098
We've had similar issues with kernel regressions (e.g. 5.4 kernels had issues with Intel GPUs, and on ThinkPads with Nvidia GPUs).
(Ideally we catch all this before they land -- over the medium term I'm trying to find a way to encourage our users to help test updates)
Would it be possible to keep 2 or 3 versions of the same package in the updates repo, so we can easily keep some of our fleet at a previous version known to work on that particular hardware? And is there a process for proposing this (e.g. file a ticket on Pagure)?
Our workaround right now is to check in the older versions in our internal repo.
Thanks,
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
usually will do the trick (adjust for arch, obviously). For stable releases, all packages that were ever made stable updates are kept around, AIUI - they are exempt from garbage collection. For Rawhide I think things get garbage collected after a while, but they're usually there for several weeks first.
Dne 24. 03. 20 v 16:30 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
While this might sound useful, it is not that useful when your system is borked and you want to get it up and running again.
IOW this is strawman to the original request and my reasoning for the +1.
Vít
usually will do the trick (adjust for arch, obviously). For stable releases, all packages that were ever made stable updates are kept around, AIUI - they are exempt from garbage collection. For Rawhide I think things get garbage collected after a while, but they're usually there for several weeks first.
On Tue, 2020-03-24 at 18:59 +0100, Vít Ondruch wrote:
Dne 24. 03. 20 v 16:30 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
While this might sound useful, it is not that useful when your system is borked and you want to get it up and running again.
IOW this is strawman to the original request and my reasoning for the +1.
In what situation could you do 'dnf downgrade' but not that?
Dne 24. 03. 20 v 19:52 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 18:59 +0100, Vít Ondruch wrote:
Dne 24. 03. 20 v 16:30 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
While this might sound useful, it is not that useful when your system is borked and you want to get it up and running again.
IOW this is strawman to the original request and my reasoning for the +1.
In what situation could you do 'dnf downgrade' but not that?
In a situation I don't have other computer to explore Koji, in a case that for example Gnome was updated and it is not just about downgrading one package due to dependencies.
Vít
On Wed, 2020-03-25 at 10:09 +0100, Vít Ondruch wrote:
Dne 24. 03. 20 v 19:52 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 18:59 +0100, Vít Ondruch wrote:
Dne 24. 03. 20 v 16:30 Adam Williamson napsal(a):
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
+1
I would love this for Rawhide. This would also allow `dnf downgrade` to work, which would be very useful when things go south. In stable releases, you could in theory downgrade from version in `updates` repository to version from `fedora` repository`, but that is not possible in Rawhide :/
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
While this might sound useful, it is not that useful when your system is borked and you want to get it up and running again.
IOW this is strawman to the original request and my reasoning for the +1.
In what situation could you do 'dnf downgrade' but not that?
In a situation I don't have other computer to explore Koji, in a case that for example Gnome was updated and it is not just about downgrading one package due to dependencies.
You don't need another computer to 'explore koji', you can do it all with the koji CLI. 'koji list-tag-history' to find previous builds of a given package, for e.g. Admittedly it's a bit clunkier than the web UI, but it's all there. 'koji help' lists all the commands...
On Tue, 2020-03-24 at 10:03 +0100, Vít Ondruch wrote:
It's pretty easy to do this with the Koji CLI:
koji download-build --arch=x86_64 --arch=noarch (NVR) dnf downgrade *.rpm
We'll probably need to write some Chef resources (equivalent to Ansible modules) to handle installing packages from Koji then. That way we can automate fixes based on hardware types, instead of asking individually affected users to first install Koji's CLI and then either learn it or (worse) copy paste some commands :)
Taking Kevin's answer into account too -- yeah, in case where there are security updates I could imagine managing multiple versions of updates could be even more problematic.
Thanks everyone! I'll probably hold off on taking it to -devel or to FESCo just yet, and explore automating Koji via Chef first.
On Tue, Mar 24, 2020 at 12:12:29AM -0000, Michel Alexandre Salim wrote:
Hi,
We run a diverse fleet of Linux laptops and desktops (at Facebook), and sometimes there are regressions that affect some of our fleet but not others.
To pick the latest example:
- pulseaudio 1.3.99.1 (both -1 and -2) breaks Bluetooth support on a Dell XPS 15: https://bugzilla.redhat.com/show_bug.cgi?id=1814556
- but it fixes HDA audio input on ThinkPad T490s and X1 Carbon: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3e19f5098
We've had similar issues with kernel regressions (e.g. 5.4 kernels had issues with Intel GPUs, and on ThinkPads with Nvidia GPUs).
(Ideally we catch all this before they land -- over the medium term I'm trying to find a way to encourage our users to help test updates)
Would it be possible to keep 2 or 3 versions of the same package in the updates repo, so we can easily keep some of our fleet at a previous version known to work on that particular hardware? And is there a process for proposing this (e.g. file a ticket on Pagure)?
Our workaround right now is to check in the older versions in our internal repo.
So I tilted at this windmill for a while (although from the perspective of rawhide, not updates).
The things that come up:
* It's actually really hard to know what the last 2-3 versions of a package are. koji has no concept of versions, it just has tags. In an ideal world they would be in a nice order in the tag, but there's lots of things that cause this to not be the case. ie, you can get say the last 3 kernels tagged into a tag, but those are always the last 3 version wise. At one point this was very difficult for pungi to do, but might be easier now.
* Keeping 2-3 more packages increases space a great deal. Both updates space and repodata space.
* Keeping 2-3 more packages increases the threat surface about insecure updates. ie, now you could trick someone into downgrading or installing something insecure from the base repo, with this you have 2-3x the chance with all the versions in the updates repo.
Anyhow, I guess this would be something to propose to FESCo (althought they might want discussion on devel list first).
kevin
infrastructure@lists.fedoraproject.org
-
Adam Williamson -
Kevin Fenzi -
Michel Alexandre Salim -
Vít Ondruch