Since this has almost nothing to do with the log output of depcheck upgradepath, I'm splitting this conversation off into another thread.
As Kamil pointed out, my initial proposal was naive and needs more thought before implementing anything. We will soon have the ability to disable emails for comments in Bodhi and we need to figure out how to best make use of that ability.
Lots of questions here, I'll save my thoughts for replying to myself.
Tim
How would we deal with tests that change state between PASS and FAIL? - Either 'PASS -> FAIL' or 'FAIL -> PASS'
What emails do we want to get rid of? - Just certain 'PASS' messages? - All results messages?
How do we expect maintainers to learn of AutoQA results if we turn off email notifications?
Can we do this in a way such that it is configurable by maintainer? - If so, should we attempt this? - What is the benefit? How big is it?
Can we just send a single email after all tests have passed? - Again, if so, should we?
Is there another approach that would be better?
On 04/29/2011 03:10 PM, Tim Flink wrote:
How would we deal with tests that change state between PASS and FAIL?
- Either 'PASS -> FAIL' or 'FAIL -> PASS'
I think that both of these cases are important and I'm not sure how to do them without at least some parsing and comprehension of update comments as a whole.
This could get really complicated, really quick but I have some ideas based on parsing comments, creating sequences of test result order and making more holistic decisions.
I tried putting my thoughts into words, but it wasn't working. I'm not convinced that my idea here is actually going to work with a reasonable amount of work.
I'll see if I can spike something together for a rough concept. It might not get done today, though.
What emails do we want to get rid of?
- Just certain 'PASS' messages?
- All results messages?
I hadn't put a whole lot of thought into the details of this, but my initial thought on the most important messages we could send would be: - Update PASS/FAIL status transition - Package PASS/FAIL status transition
For now, I'm wondering if package state transition would be good enough.
How do we expect maintainers to learn of AutoQA results if we turn off email notifications?
I can't really think of anything here. I think that some pestering is a good thing, just not too much and that turning off emails completely would leave us with not enough pestering.
Can we do this in a way such that it is configurable by maintainer?
- If so, should we attempt this?
- What is the benefit? How big is it?
This is a hard one. I'm sure that we could do it but I'm more interested in the should (at least for the near future).
It would be a decent amount of work and require some new concepts in AutoQA - interfacing with FAS (at least indirectly), defining various knobs to turn for update emails and looking that up before or during commenting.
I'm thinking that this would be a cool feature but unless we start getting requests for it, I'd vote for not now.
Can we just send a single email after all tests have passed?
- Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure, we could code up some stuff to determine what packages are supposed to pass what tests, but this seems to be quite a bit of overlap with PUATP and ResultsDB to me.
I'm inclined to say wait on ResultsDB for this one. I think that it might be a good idea to continue to use Bodhi comments to notify maintainers of update test status after we implement ResultsDB but I can't think of an easy way to do this until then.
Is there another approach that would be better?
I'm probably the wrong person to answer this question but I'm interested in hearing what the alternatives would be.
I like this approach because it gives us the ability to determine when emails are getting sent. Unless that functionality is already in bodhi or elsewhere, I'm not sure it makes a whole lot of sense to do the filtering outside AutoQA.
Can we just send a single email after all tests have passed?
- Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure, we could code up some stuff to determine what packages are supposed to pass what tests, but this seems to be quite a bit of overlap with PUATP and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we don't have ResultDB (which will take a while) and we need some reasonable workaround till that time. Of course some of that work will be nicely usable even for ResultDB (like splitting one bug test output into multiple per-item logs). But some stuff is just temporary fix. Therefore I wouldn't really be afraid of a few unclean hacks that will be removed once we have ResultDB.
So I thought about this. Could we send an email only after all tests have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
So our code may default to email=False and change to email=True only when all tests have been executed (counting also the current one that is to be sent). That alone decreases the email load by 66%.
Let's go further. If all tests are completed and PASSED, I wouldn't send an email. If everything went fine, why bother the maintainer?
Let's go further. Currently we re-send FAILED results after at least 3 days from the first failure. I wouldn't send email notifications for them. The purpose is to provide a fresh failure log, not the notification (the maintainer already knows about it).
Then there remains the question of PASS->FAIL and FAIL->PASS transitions. I would probably send notifications for them. They seem rather important.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages. About 20 packages is 32bit only: http://fpaste.org/BRwA/ and 4 packages is 64bit only: http://fpaste.org/WVBO/ I'm willing to sacrifice that, for now. We have much higher test crash ratio anyway.
On 05/02/2011 09:54 AM, Kamil Paral wrote:
Can we just send a single email after all tests have passed? - Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure, we could code up some stuff to determine what packages are supposed to pass what tests, but this seems to be quite a bit of overlap with PUATP and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we don't have ResultDB (which will take a while) and we need some reasonable workaround till that time. Of course some of that work will be nicely usable even for ResultDB (like splitting one bug test output into multiple per-item logs). But some stuff is just temporary fix. Therefore I wouldn't really be afraid of a few unclean hacks that will be removed once we have ResultDB.
I think that we're of the same mind on this. I'm not completely against not-completely-clean hacks for the short term but I also think that it would be wise to think about the level of effort we're putting in. In my mind, we're shooting for somewhat short-term gains on some of this and after a certain point, energy devoted to hacking stuff together could be better directed towards finishing ResultsDB.
So I thought about this. Could we send an email only after all tests have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
If there are that few, we could hard code them into the bodhi commenting code. Not sure that is the greatest idea ever, though. Another option would be to document the issue and contact the affected maintainers.
So our code may default to email=False and change to email=True only when all tests have been executed (counting also the current one that is to be sent). That alone decreases the email load by 66%.
That works for me in the short term (the hardcoding part).
Let's go further. If all tests are completed and PASSED, I wouldn't send an email. If everything went fine, why bother the maintainer?
All tests completed and PASSED might be worth sending an email for as a notification that everything is working but I see your point. We might want to get some input from maintainers on that one.
Let's go further. Currently we re-send FAILED results after at least 3 days from the first failure. I wouldn't send email notifications for them. The purpose is to provide a fresh failure log, not the notification (the maintainer already knows about it).
That makes sense to me unless we do really want to pester maintainers with packages that are failing repeatedly. Maybe up it from 3 to 5 or 7 days?
Then there remains the question of PASS->FAIL and FAIL->PASS transitions. I would probably send notifications for them. They seem rather important.
At the very least, transitions are important. Agreed on this one.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages. About 20 packages is 32bit only: http://fpaste.org/BRwA/ and 4 packages is 64bit only: http://fpaste.org/WVBO/ I'm willing to sacrifice that, for now. We have much higher test crash ratio anyway.
So I thought about this. Could we send an email only after all tests have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
I just realized that upgradepath is not run for updates proposed into updates-testing. Something to keep an eye on. Still it should be doable.
On 04/29/2011 03:10 PM, Tim Flink wrote:
Since this has almost nothing to do with the log output of depcheck upgradepath, I'm splitting this conversation off into another thread.
Are we good with what has been discussed so far or at least good enough to say we're done with the initial planning? I'm looking to get started on this today.
My plan would be to get some rough work done and propose it since there are still some TBD aspects of this.
Tim
Just as an aside, I wouldn't mind few emails from Bodhi, anyway. What I was thinking is if we could do a 'Daily Digest' if there has been any changes/comments in the previous 24 hours, rather than receive an individual email every time. It can get annoying if I get five emails, and each one is essentially copied in the last.
Am I making sense here?
John Dulaney jdulaney@fedoraproject.org j_dulaney
Date: Wed, 11 May 2011 09:16:57 -0600 From: tflink@redhat.com To: autoqa-devel@lists.fedorahosted.org Subject: Re: Decreasing Bodhi Comment Emails From AutoQA
On 04/29/2011 03:10 PM, Tim Flink wrote:
Since this has almost nothing to do with the log output of depcheck upgradepath, I'm splitting this conversation off into another thread.
Are we good with what has been discussed so far or at least good enough to say we're done with the initial planning? I'm looking to get started on this today.
My plan would be to get some rough work done and propose it since there are still some TBD aspects of this.
Tim
_______________________________________________ autoqa-devel mailing list autoqa-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/autoqa-devel
On 05/11/2011 12:13 PM, John Dulaney wrote:
Just as an aside, I wouldn't mind few emails from Bodhi, anyway. What I was thinking is if we could do a 'Daily Digest' if there has been any changes/comments in the previous 24 hours, rather than receive an individual email every time. It can get annoying if I get five emails, and each one is essentially copied in the last.
Am I making sense here?
I think so, but I'm wondering if getting emails immediately when a test fails would be helpful instead of waiting until the next digest was sent.
Do you think that receiving emails as a digest would be better than receiving emails deemed "important" as soon as they are generated and getting far fewer "PASSED" noise emails?
Tim
I think so, but I'm wondering if getting emails immediately when a test fails would be helpful instead of waiting until the next digest was sent.
Point.
Do you think that receiving emails as a digest would be better than receiving emails deemed "important" as soon as they are generated and getting far fewer "PASSED" noise emails?
Indeed. Getting six emails saying "Yes, this works." is rather useless, to me, especially when the sixth contains all the info in the previous five.
Example:
The email bellow is for Anaconda. It contains everything that was in the previous four emails I have received from Bodhi on this particular package. Just the one would have been fine.
The following comment has been added to the anaconda-15.31-1.fc15 update:
autoqa - 2011-05-11 05:45:40 (karma: 0) AutoQA: upgradepath test PASSED on noarch. Result log: http://autoqa.fedoraproject.org/results/95379-autotest/qa02.c.fedoraproject.... (results are informative only)
To reply to this comment, please visit the URL at the bottom of this mail
================================================================================ anaconda-15.31-1.fc15 ================================================================================ Update ID: FEDORA-2011-6638 Release: Fedora 15 Status: testing Type: bugfix Karma: 3 Request: stable Bugs: 702603 - FormatCreateError: ('format failed: 1', '/dev/mapper : /vg_kralik-LogVol01') : 699923 - livecd doesn't have /var/log/dmesg : 700085 - iSCSI login dialog allows unselecting all : devices and continuing - unable to login to any iSCSI : devices after Notes: This update fixes the memory requirement dialog and enforces the : minimum btrfs filesystem size on partitions of that : type. This update fixes a problem with reporting bugs : on live installs and a bug with iSCSI. Submitter: clumens Submitted: 2011-05-06 19:03:25 Comments: bodhi - 2011-05-06 19:03:49 (karma 0) This update has been submitted for testing by clumens.
bodhi - 2011-05-07 15:05:02 (karma 0) This update has been pushed to testing
filiperosset (proventesters) - 2011-05-07 23:33:45 (karma 0) Not tested.
jlaska (proventesters) - 2011-05-09 19:33:12 (karma 1) filiperosset - Note, no comment is needed if you don't test an update. I tested the proposed update with custom-built x86_64 boot.iso. Works good for installation here.
adamwill (proventesters) - 2011-05-09 20:40:41 (karma 1) straight through live install in a vm works.
bodhi - 2011-05-09 20:40:41 (karma 0) Critical path update approved
jdulaney (proventesters) - 2011-05-09 22:00:15 (karma 0) not tested
rhe (proventesters) - 2011-05-11 05:30:54 (karma 1) Default install passed.
bodhi - 2011-05-11 05:30:57 (karma 0) This update has reached the stable karma threshold and will be pushed to the stable updates repository
autoqa - 2011-05-11 05:45:40 (karma 0) AutoQA: upgradepath test PASSED on noarch. Result log: http://autoqa.fedoraproject.org/results/95379-autotest /qa02.c.fedoraproject.org/upgradepath/results/output.l og (results are informative only)
https://admin.fedoraproject.org/updates/anaconda-15.31-1.fc15
On 05/11/2011 01:00 PM, John Dulaney wrote:
I think so, but I'm wondering if getting emails immediately when a test fails would be helpful instead of waiting until the next digest was sent.
Point.
Do you think that receiving emails as a digest would be better than receiving emails deemed "important" as soon as they are generated and getting far fewer "PASSED" noise emails?
Indeed. Getting six emails saying "Yes, this works." is rather useless, to me, especially when the sixth contains all the info in the previous five.
I might be misunderstanding you, but it sounds like you're talking about the emails generated when people add comments to an update. If I'm right, that is part of Bodhi and something that we can't change. You could file an enhancement request for Bodhi or bring it up on devel@, though.
Example:
The email bellow is for Anaconda. It contains everything that was in the previous four emails I have received from Bodhi on this particular package. Just the one would have been fine.
I think that the current plans of sending emails when all tests are done or when there has been a state change between PASS and FAIL would take care of the extra email noise generated by AutoQA. More details are in trac [1].
Are we missing a case where this wouldn't be true?
Tim
[1] https://fedorahosted.org/autoqa/wiki/Planning05xEmailReduction
I might be misunderstanding you, but it sounds like you're talking about the emails generated when people add comments to an update. If I'm right, that is part of Bodhi and something that we can't change. You could file an enhancement request for Bodhi or bring it up on devel@, though.
Exactly, and will do.
I think that the current plans of sending emails when all tests are done or when there has been a state change between PASS and FAIL would take care of the extra email noise generated by AutoQA. More details are in trac [1].
+ 1
Are we missing a case where this wouldn't be true?
Don't think so off the top of my head.
John Dulaney jdulaney@fedoraproject.org j_dulaney
autoqa-devel@lists.fedorahosted.org
-
John Dulaney -
Kamil Paral -
Tim Flink