On Tue, 2016-12-20 at 10:48 -0600, Michael Catanzaro wrote:
On Tue, 2016-12-20 at 14:27 +0000, Tom Hughes wrote:
> Surely it's more likely that it just delays the discovery of the
> botched
> update?
I don't think updates-testing should be batched. Testers should of
course still get all test updates ASAP.
> The only way it reduces the risk of releasing a botched update is
> the
> the updates somehow get more testing just by staying in the testing
> channel longer.
...and actual QA, from the professionals and volunteers on the QA team,
who are very good at finding bugs pre-release but currently do zero QA
on our updates because it's an unmanageable rolling stream of a
bazillion separate updates.
This is an exaggeration. We do test updates. We could always test
everything *better*, and that applies to updates, but it is not true to
say we 'do zero QA' on them.
With batched updates, you can test a batch
with the same overall criteria used for releases to see if it's
botched. That's the advantage of batching over simply extending the
amount of time spent in updates-testing.
This is also an exaggeration, or at least it's a long way from
proven. I don't think we could say that just batching updates would
suddenly allow us to QA them as extensively as we QA a release. Release
testing involves a lot of work by a lot of people; especially desktop
validation is rather onerous. If we're talking about *weekly* batched
updates, no, it is not at all practical to assume we'll magically be
able to find the time to do release-validation level testing of each
update batch every week.
We could in theory apply what automated functional testing we have to
batched updates, but it's not at all a simple thing to do, and we could
in fact apply it to *non*-batched updates too. It's something I've been
wanting to do for a while, just have not had time for yet.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net