Re: post go upgrades (re)testing
On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson <adamwill(a)fedoraproject.org>
wrote:
We do sort of have this already, because openQA tests all updates,
and
one of the tests it runs on them is an upgrade test.
Now, the iptables thing is interesting, because the logs of that test
on an F34 update show the issue:
2021-04-28T04:30:39-0400 WARNING
Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and
iptables-libs-1.8.7-6.fc34.x86_64
- package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) =
1.8.7-3.fc34, but none of the providers can be installed
- cannot install the best update candidate for package
iptables-libs-1.8.5-6.fc33.x86_64
- cannot install the best update candidate for package
iptables-1.8.5-6.fc33.x86_64
but it's only logged as a WARNING and does not prevent the upgrade
running. So the test passes. It doesn't even need to use `--
allowerasing` so it doesn't register as a soft fail.
I assume that's because iptables from the main (fedora) repo are used
instead. But the upgrade would fail if we added "--best" (I tested it).
Here's a thought. What if we modified the openQA test to first try with
"--best". If there's any error, log it as a softfail, and continue without
it (and if if fails again, log it as a hard error). This way, we can catch
*some* broken updates (it's still possible that one broken update obscures
a second one and we'll not know about the latter - but the benefit is
definitely there).
If we did it this way, what exactly happens with the softfailed result?
Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing
it soon enough?
As for manual testing, what do you think about my previous proposal?
we could extend our upgrade testing matrices with two variants - one
would test with stable updates, the other would test with updates-testing
enabled
Attachments:
- attachment.html (text/html — 2.3 KB)