On 05/02/2011 09:54 AM, Kamil Paral wrote:
Can we just send a single email after all tests have passed? - Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure, we could code up some stuff to determine what packages are supposed to pass what tests, but this seems to be quite a bit of overlap with PUATP and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we don't have ResultDB (which will take a while) and we need some reasonable workaround till that time. Of course some of that work will be nicely usable even for ResultDB (like splitting one bug test output into multiple per-item logs). But some stuff is just temporary fix. Therefore I wouldn't really be afraid of a few unclean hacks that will be removed once we have ResultDB.
I think that we're of the same mind on this. I'm not completely against not-completely-clean hacks for the short term but I also think that it would be wise to think about the level of effort we're putting in. In my mind, we're shooting for somewhat short-term gains on some of this and after a certain point, energy devoted to hacking stuff together could be better directed towards finishing ResultsDB.
So I thought about this. Could we send an email only after all tests have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
If there are that few, we could hard code them into the bodhi commenting code. Not sure that is the greatest idea ever, though. Another option would be to document the issue and contact the affected maintainers.
So our code may default to email=False and change to email=True only when all tests have been executed (counting also the current one that is to be sent). That alone decreases the email load by 66%.
That works for me in the short term (the hardcoding part).
Let's go further. If all tests are completed and PASSED, I wouldn't send an email. If everything went fine, why bother the maintainer?
All tests completed and PASSED might be worth sending an email for as a notification that everything is working but I see your point. We might want to get some input from maintainers on that one.
Let's go further. Currently we re-send FAILED results after at least 3 days from the first failure. I wouldn't send email notifications for them. The purpose is to provide a fresh failure log, not the notification (the maintainer already knows about it).
That makes sense to me unless we do really want to pester maintainers with packages that are failing repeatedly. Maybe up it from 3 to 5 or 7 days?
Then there remains the question of PASS->FAIL and FAIL->PASS transitions. I would probably send notifications for them. They seem rather important.
At the very least, transitions are important. Agreed on this one.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages. About 20 packages is 32bit only: http://fpaste.org/BRwA/ and 4 packages is 64bit only: http://fpaste.org/WVBO/ I'm willing to sacrifice that, for now. We have much higher test crash ratio anyway.