Hey folks.
I wonder if we couldn't add in some (re)testing of upgrades after a release is 'go' but before it's actually released. We hit at least two issues I am aware of with f34 due to multilib. ;(
first: pipewire.i686 is in the base x86_64 repo and users had/have it installed.
pipewire.i686 is pulled into the x86_64 base repo by mutter (mutter.i686 is there and requires pipewire.i686).
In any case it is not in the updates repo, so once an update of them is pushed (which it has been), upgrades fail due to the missing packages.
I have modified the updates pungi config to whitelist pipewire for multilib and it's in f34-updates now.
second: There's still an issue with iptables. See https://bugzilla.redhat.com/show_bug.cgi?id=1953178 basically the version in the base repo has a 'iptables' package. The one in updates Obsoletes this for a iptables-compat, but yet it's not doing things correctly as users are getting dnf errors.
Anyhow, I think it might be good to perhaps schedule some re-resting of upgrades after the 0 day updates repo is populated to try and catch these and fix them before release day.
We can't test this fully before there is an updates repo (I mean we kind of can with updates-testing, but it's not the exact same repo).
kevin
I wonder what the best way to improve the situation would be. There are two issues as a I see it: a) Some packages (pipewire.i686) are not in the default installation and therefore our automated checks don't catch it. The manual checks are often done with default installations as well, but someone with a real-world machine can stumble on it. b) Some packages (I think this is the case of iptables) are in a default installation, but are in updates-testing, which we ignore during upgrade testing. On one hand, it makes sense, we want to make sure the stable repo is functional, and updates-testing often includes broken stuff which could prevent us from running the upgrade test completely. On the other hand, on the release day, a lot of packages get pushed from updates-testing to updates, and it might result in broken upgrades, while we had no knowledge about it. To complicate it even further, it's a constantly moving target, i.e. the set of packages which are "pending stable" (to be pushed to updates when the freeze is over) can change literally every minute.
The true fix here for both cases would be a static test which would check the installability of packages, and wouldn't allow pushing updates which don't pass. Something similar to rpmdeplint which we used to run in Taskotron (rpmdeplint itself had many issues, it would need to get much improved or written differently, to be able to make it a reliable gating test). But we don't have it and likely won't have it for a long time.
So what can we do which is easier and at least partly covers the issues? For a) I think we need to depend on community reporting, because we can hardly test a random combination of installed packages. But for b), we could extend our upgrade testing matrices with two variants - one would test with stable updates, the other would test with updates-testing enabled. It is not perfect, because any broken update in updates-testing can obscure other issues present ("dnf --exclude" is not always the remedy), so we still can miss important problems. We can't force package maintainers to unpush problematic updates just because we want to test the rest of the repo. And even if we could, refreshing the repo takes at least a day, so there's a very slow turnaround. However, it gives us more chance to spot potential issues and make sure that they are reported and that they don't get pushed stable (by giving them negative karma). Thoughts?
We can also test upgrades once the updates repo gets populated, as you mentioned. It's pretty late to detect problems at this stage :-/ , but better late than never, I guess. I wonder if we could use openQA for this [1]. Adam?
[1] Which reminds me, we should make sure to run "dnf system-upgrade" with "--best", to catch these issues.
On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
Hey folks.
I wonder if we couldn't add in some (re)testing of upgrades after a release is 'go' but before it's actually released. We hit at least two issues I am aware of with f34 due to multilib. ;(
I'm asking this for years and to not stress and delay to much the release, I proposed do a second release in this case it would be 34.1
first: pipewire.i686 is in the base x86_64 repo and users had/have it installed.
pipewire.i686 is pulled into the x86_64 base repo by mutter (mutter.i686 is there and requires pipewire.i686).
In any case it is not in the updates repo, so once an update of them is pushed (which it has been), upgrades fail due to the missing packages.
I have modified the updates pungi config to whitelist pipewire for multilib and it's in f34-updates now.
second: There's still an issue with iptables. See https://bugzilla.redhat.com/show_bug.cgi?id=1953178 basically the version in the base repo has a 'iptables' package. The one in updates Obsoletes this for a iptables-compat, but yet it's not doing things correctly as users are getting dnf errors.
Anyhow, I think it might be good to perhaps schedule some re-resting of upgrades after the 0 day updates repo is populated to try and catch these and fix them before release day.
We can't test this fully before there is an updates repo (I mean we kind of can with updates-testing, but it's not the exact same repo).
kevin _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Apr 28, 2021 at 10:47:39AM +0100, Sérgio Basto wrote:
On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
Hey folks.
I wonder if we couldn't add in some (re)testing of upgrades after a release is 'go' but before it's actually released. We hit at least two issues I am aware of with f34 due to multilib. ;(
I'm asking this for years and to not stress and delay to much the release, I proposed do a second release in this case it would be 34.1
That would be prohibive on resources and there would be a lot of process that we do now once per 6 months that would need to be done in days/weeks. :( So no, I dont think thats a good answer unless we can't avoid it.
kevin
On Wed, 2021-04-28 at 10:14 -0700, Kevin Fenzi wrote:
On Wed, Apr 28, 2021 at 10:47:39AM +0100, Sérgio Basto wrote:
On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
Hey folks.
I wonder if we couldn't add in some (re)testing of upgrades after a release is 'go' but before it's actually released. We hit at least two issues I am aware of with f34 due to multilib. ;(
I'm asking this for years and to not stress and delay to much the release, I proposed do a second release in this case it would be 34.1
That would be prohibive on resources and there would be a lot of process that we do now once per 6 months that would need to be done in days/weeks. :( So no, I dont think thats a good answer unless we can't avoid it.
I had an idea and if instead of calling it stable, we call it RC1 (release candidate 1) and only after adding all the updates in the base version and (re) testing, we finally call it Fedora 34 stable ?
kevin _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, May 04, 2021 at 01:25:50AM +0100, Sérgio Basto wrote:
I had an idea and if instead of calling it stable, we call it RC1 (release candidate 1) and only after adding all the updates in the base version and (re) testing, we finally call it Fedora 34 stable ?
Yeah, we could add a few days in and require the updates repo, but...
It makes it really hard in some cases. Like, for example, say we have a RC1, we make the updates repo, and we find some release blocking bug. We make an update for it, but it needs to be in the base repo... so that just 1 update has to compose into the base repo, but all other updates have to go to updates-testing. ie, we have to do more things and it becomes more complex. Bodhi isn't setup to handle both populating a updates/updates testing repos and leting a branched compose compose just some updates into the base repo. More states could be added of course, but it's more work.
kevin
On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
Hey folks.
I wonder if we couldn't add in some (re)testing of upgrades after a release is 'go' but before it's actually released. We hit at least two issues I am aware of with f34 due to multilib. ;(
first: pipewire.i686 is in the base x86_64 repo and users had/have it installed.
pipewire.i686 is pulled into the x86_64 base repo by mutter (mutter.i686 is there and requires pipewire.i686).
In any case it is not in the updates repo, so once an update of them is pushed (which it has been), upgrades fail due to the missing packages.
I have modified the updates pungi config to whitelist pipewire for multilib and it's in f34-updates now.
second: There's still an issue with iptables. See https://bugzilla.redhat.com/show_bug.cgi?id=1953178 basically the version in the base repo has a 'iptables' package. The one in updates Obsoletes this for a iptables-compat, but yet it's not doing things correctly as users are getting dnf errors.
Anyhow, I think it might be good to perhaps schedule some re-resting of upgrades after the 0 day updates repo is populated to try and catch these and fix them before release day.
We can't test this fully before there is an updates repo (I mean we kind of can with updates-testing, but it's not the exact same repo).
We do sort of have this already, because openQA tests all updates, and one of the tests it runs on them is an upgrade test.
Now, the iptables thing is interesting, because the logs of that test on an F34 update show the issue:
2021-04-28T04:30:39-0400 WARNING Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and iptables-libs-1.8.7-6.fc34.x86_64 - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = 1.8.7-3.fc34, but none of the providers can be installed - cannot install the best update candidate for package iptables-libs-1.8.5-6.fc33.x86_64 - cannot install the best update candidate for package iptables-1.8.5-6.fc33.x86_64
but it's only logged as a WARNING and does not prevent the upgrade running. So the test passes. It doesn't even need to use `-- allowerasing` so it doesn't register as a soft fail.
For the pipewire issue, we won't catch it with automation because the package isn't installed in any default F32 or F33 package sets.
Doing some manual testing might be possible, but then we were already a bit tired from validating RC2 in five hours. People might have wanted a break. And of course, signing RC2 off on Friday meant there was virtually no office time for RH employees to do it in...
On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson adamwill@fedoraproject.org wrote:
We do sort of have this already, because openQA tests all updates, and one of the tests it runs on them is an upgrade test.
Now, the iptables thing is interesting, because the logs of that test on an F34 update show the issue:
2021-04-28T04:30:39-0400 WARNING Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and iptables-libs-1.8.7-6.fc34.x86_64
- package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) =
1.8.7-3.fc34, but none of the providers can be installed
- cannot install the best update candidate for package
iptables-libs-1.8.5-6.fc33.x86_64
- cannot install the best update candidate for package
iptables-1.8.5-6.fc33.x86_64
but it's only logged as a WARNING and does not prevent the upgrade running. So the test passes. It doesn't even need to use `-- allowerasing` so it doesn't register as a soft fail.
I assume that's because iptables from the main (fedora) repo are used instead. But the upgrade would fail if we added "--best" (I tested it). Here's a thought. What if we modified the openQA test to first try with "--best". If there's any error, log it as a softfail, and continue without it (and if if fails again, log it as a hard error). This way, we can catch *some* broken updates (it's still possible that one broken update obscures a second one and we'll not know about the latter - but the benefit is definitely there).
If we did it this way, what exactly happens with the softfailed result? Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing it soon enough?
As for manual testing, what do you think about my previous proposal?
we could extend our upgrade testing matrices with two variants - one
would test with stable updates, the other would test with updates-testing enabled
Gmail doesn't display it when you put the quote below your text, let me quote it differently: As for manual testing, what do you think about my previous proposal? "we could extend our upgrade testing matrices with two variants - one would test with stable updates, the other would test with updates-testing enabled"
On Thu, 2021-04-29 at 10:09 +0200, Kamil Paral wrote:
On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson adamwill@fedoraproject.org wrote:
We do sort of have this already, because openQA tests all updates, and one of the tests it runs on them is an upgrade test.
Now, the iptables thing is interesting, because the logs of that test on an F34 update show the issue:
2021-04-28T04:30:39-0400 WARNING Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and iptables-libs-1.8.7-6.fc34.x86_64 - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = 1.8.7-3.fc34, but none of the providers can be installed - cannot install the best update candidate for package iptables-libs-1.8.5-6.fc33.x86_64 - cannot install the best update candidate for package iptables-1.8.5-6.fc33.x86_64
but it's only logged as a WARNING and does not prevent the upgrade running. So the test passes. It doesn't even need to use `-- allowerasing` so it doesn't register as a soft fail.
I assume that's because iptables from the main (fedora) repo are used instead. But the upgrade would fail if we added "--best" (I tested it). Here's a thought. What if we modified the openQA test to first try with "--best". If there's any error, log it as a softfail, and continue without it (and if if fails again, log it as a hard error). This way, we can catch *some* broken updates (it's still possible that one broken update obscures a second one and we'll not know about the latter - but the benefit is definitely there).
If we did it this way, what exactly happens with the softfailed result? Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing it soon enough?
So I looked into this some more yesterday. openQA did in fact catch this bug, straight up: when the iptables update was initially submitted, an openQA upgrade test failed on exactly this problem. The upgrade succeeded, but the test failed because of the last-step check we do in openQA that, if any package from the update is installed, it is the version from the update that is installed. Because the upgrade pulled in the earlier -3 version and not the -6 version from the update, the test failed.
I mentioned this in Bodhi, even, but I noted that if the update was pushed stable *before* F34's final release, the bug should disappear, because the -3 builds would no longer be available as options and dnf would in that case choose to go with -6 and remove iptables (I tested this).
Unfortunately, the update was *not* pushed stable before final release, so the -3 builds are in the frozen 'fedora' repo forever now, and we have a more or less insoluble problem (probably the best option is just to put the iptables package back in F34, and re-do this better for F35).
See the bug report for more details: https://bugzilla.redhat.com/show_bug.cgi?id=1953178
Given this, I don't think we need to use your idea, since it's demonstrated that openQA *does* actually catch bugs like this already. We just need maintainers to pay more attention and follow up better.
As for manual testing, what do you think about my previous proposal?
we could extend our upgrade testing matrices with two variants - one
would test with stable updates, the other would test with updates-testing enabled
Sure, it's possible. It just adds more testing that either a human or a robot needs to do. And, again, it turns out our current testing did actually catch this problem just fine. If anything went wrong, it was the follow-up.
-
Adam Williamson -
Kamil Paral -
Kevin Fenzi -
Sérgio Basto