3:43 p.m.
On Thu, 2010-09-30 at 05:50 -0400, Kamil Paral wrote:
This ticket:
https://fedorahosted.org/autoqa/ticket/228
and this patch:
https://fedorahosted.org/pipermail/autoqa-devel/2010-September/001195.html
forced me to do some thinking.
We have some tests that are really timing-critical - depcheck and upgradepath.
We can't run them in an arbitrary time as we can do with rpmlint and similar.
We must run them just before the packages are pushed - not earlier, not later.
In other words, we are given set of packages to examine and a repository
in some well-defined state. We examine the packages and the repo and say
"OK" or "not OK". Someone (Bodhi) must guarantee that neither the
repo nor
the set of packages change between our examination and actual push of those
packages. There must be no steps in between. Otherwise our result would be
misleading.
For upgradepath we have even stricter requirements - *no* repo must change
between our examination and actual push.
Now that we know the requirements, we can ask some questions:
As for the ticket:
Can we really rely on checking koji -pending tags, instead of communicating
with Bodhi? I don't know how the -pending tags are implemented. But can we
be sure that Bodhi takes exactly same set of packages in the exactly same
order as we do, when it tries to push them?
This is a valid concern.
I *think* wwoods discussed this scenario with me the other day. I've
cc'd him for confirmation. If I understand correctly, the
'bodhi-pending-update' (or whatever we call the updated watcher
addressed inticket#228), would trigger and request testing for a
specific set of tags. While the tests are queued and awaiting
execution, the -pending tag could be updated again. When the depcheck
test does run, it will only report pass/fail on the updates it was
originally scheduled for (and any updates that previously failed but now
pass). Remaining -pending updates will be queued for testing by the
next 'bodhi-pending-update' watcher.
Will, please sanity check me here.
But can we be sure that Bodhi takes exactly same set of packages in
the exactly same order as we do, when it tries to push them?
Release engineering does the pushes for this manually, perhaps someone
can confirm with them how the determine which updates to push. Does
order matter?
As for the patch:
I have a feeling that for these aforementioned tests there's not really any
sense in sending email reports. These tests either pass or fail, just before
push. If our test fails, then the push fails and maintainers will be probably
notified by Bodhi anyway, won't they?
Good observation! Also worth noting, I don't expect these tests (or
their results) to be in enforcing mode for the first few months. In the
meantime, as long as maintainers are notified about the results, we
should be fine. Whether that 'notification' takes place as the opt-in
emails, or by way of bodhi karma feedback. Your right, it doesn't make
sense to do both. Once we are able to submit bodhi karma feedback for
autoqa, we can probably turn off opt-in email notification. Or
recommend people opt-out :)
Until resultsdb is online, I'd like to keep autoqa-results@
notification. That seems to help us manage the flow of jobs through the
system, and serve as a historical record for test results.
And more importantly, we currently run upgradepath when packages are
*requested* for push, but the actual push might be some hours away, is that
right? So the result we produce is not really valid, in the meantime a lot
of things may change. That also speaks against sending some notifications.
But maybe I just got some stuff wrong. What do you think?
As Will always says, "darn this stuff is complicated!" :)
I don't mean for this to sound like too much of a hand-wave. I think
you correctly identified several exposures where our test results may be
invalidated before they are pushed by release engineering, and I *do*
think we need to explore these exposures, I think we're still be far
better off than things are now. I don't mean we shouldn't investigate
and think this through, just that I want to avoid a cycle of making it
perfect, but never releasing.
Getting ahead of myself here, but we may find that an additional
test-event (or test-run) is needed to validate the batch of updates
before rel-eng pushes the changes. Alternatively, I wonder if depcheck
needs to include 1) the target repo (updates or updates-testing), 2) the
proposed -pending updates, *and* 3) the previously tested but not pushed
-pending updates. It may already do this, I don't know.
Thanks,
James