On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
It sounds to me like the problem is "how do we best use the
available
automated test resources?" so I'll answer accordingly. Ignore me if I
misunderstood ;-)
No, not really, sorry if I didn't explain clearly enough :D It's more
just a 'what's the best way to work what I want to do around the
existing definitions' question. What I want to do is have updates of
FreeIPA-related packages gated. The awkwardness is that right now the
definition of what gets gated is "critical path packages", and those
clearly don't meet the current definition of "critical path".
We currently have a small list of packages that are gated behind openQA,
and insufficient openQA resources to expand this list to all packages.
We also have a gating system based on karma, where users provide the QA
manually. At least, one hopes. The karma system has some
configurability for how much karma is needed, and for how long, etc.
Ah, no, again, I guess I should've explained that in more detail. When
I talk about "gating" I'm not talking about the manual karma mechanism.
I'm talking about the mechanism that prevents updates being pushed
stable if they fail the automated tests. That's a separate, parallel
thing which does not involve karma or manual feedback at all, it's
entirely automated.
What about a merger of those systems, where the openQA results count as
a type of user, contributing to the status? Give each package a "QA
Priority" that ranges from "full gating" to "five second rule",
where
the openQA resources and the user work together to prioritize and test
packages according to need:
* full gating requires all openQA tests to pass AND meet karma
requirements, openQA does these first.
* partial gating is like above, but either one (openQA or karma) can be
sufficient if enough time passes.
* reject only allows an openQA failure to reject an update, but the lack
of openQA success need not stop it if it has enough karma. (glibc uses
this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
pass adds nothing to the consensus needed)
... and so on down the list. Each user, including openQA, can "vote" a
pass or fail, and the rules say how all those passes and fails result in
the update's overall pass or fail.
This would allow the openQA resources to prioritize updates that it
knows *needs* openQA results, which allocating spare cycles to packages
that could benefit from testing but don't require it. It also means
that additional openQA resources can be put to use immediately, while
still allowing for growth in critical path size, without being wasted on
unseen results.
This is an interesting idea, but rather complex and doesn't solve any
problem I have right now. :P Sorry for the confusion!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net