I really can't tell if this is a particularly bad pain point for me since I work remotely and my internet connection isn't great (frequent SSL timeouts on koji, package downloads take forever) but I was wondering how you all went about testing AutoQA.
For me, it really depends on what I'm trying to poke at. Most of the time, I'll run 'watch-koji-builds.py --verbose' if I'm trying to do some general testing. I keep track of the events that are called if I want to run something again ('autoqa post-bodhi-update-batch --targettag dist-f13-updates --arch x86_64 --arch i386 oxygen-gtk-1.0.4-1.fc13' as an example).
If I'm trying to poke at something very specific, I find myself manually cobbling together some one-off script that runs something specific, like depcheck or upgradepath.
Testing the interaction with Koji or Bodhi? I still haven't figured out a good way to do that. Thus far, I have been hacking in print statements into bodhi_utils or koji_utils but that doesn't quite cover everything.
I ask because speaking for myself, I'm human. The more of a PITA it is to test something, the more likely I am to not do it or limit the number of times that I do test it. I think that I've been pretty good about testing stuff before pushing to master or stable, but I'm bothered by the amount of time I'm wasting on setting up tests and figuring out ways that I can trick AutoQA into going down code paths I want to test.
I'm not saying that testing is a waste of time, just thinking that some of this test setup time could be much better spent on coding or additional testing.
This isn't meant to be aimless complaining. I have some ideas on how to make the testing of AutoQA easier but I wanted to know if I was missing something obvious before I went too far down that road.
Thanks,
Tim
I really can't tell if this is a particularly bad pain point for me since I work remotely and my internet connection isn't great (frequent SSL timeouts on koji, package downloads take forever)
That's also our case in Brno office. Our internet is painfully slow, it often downloads from Koji around 100kB/s. When working on depcheck I sometimes keep the work for home where I have 10-20 times faster access. So I know what you are talking about.
Testing bandwidth-intensive tests is certainly quite problematic without fast access to the main Fedora Infrastructure machines. I don't know how to improve that.
That reminds me, I want to implement package caching for depcheck (and maybe some other tests), that could help us enormously at least for subsequent runs.
I don't receive much timeouts however. I guess that means test abort, right? Maybe we could improve that in our library (connection retries).
but I was wondering how you all went about testing AutoQA.
For me, it really depends on what I'm trying to poke at. Most of the time, I'll run 'watch-koji-builds.py --verbose' if I'm trying to do some general testing. I keep track of the events that are called if I want to run something again ('autoqa post-bodhi-update-batch --targettag dist-f13-updates --arch x86_64 --arch i386 oxygen-gtk-1.0.4-1.fc13' as an example).
I usually do "watch-koji-builds.py --verbose --dryrun" to get correct command syntax and then run desired tests by appending "-t TEST (--local)" to the command.
If I'm trying to poke at something very specific, I find myself manually cobbling together some one-off script that runs something specific, like depcheck or upgradepath.
Testing the interaction with Koji or Bodhi? I still haven't figured out a good way to do that. Thus far, I have been hacking in print statements into bodhi_utils or koji_utils but that doesn't quite cover everything.
I ask because speaking for myself, I'm human. The more of a PITA it is to test something, the more likely I am to not do it or limit the number of times that I do test it. I think that I've been pretty good about testing stuff before pushing to master or stable, but I'm bothered by the amount of time I'm wasting on setting up tests and figuring out ways that I can trick AutoQA into going down code paths I want to test.
I understand that. If I want to run depcheck and all builds are accepted ATM, I change the username in fas.conf to consider all builds as unaccepted. It would be nice not have to do this tweaks by hand over and over again.
I'm not saying that testing is a waste of time, just thinking that some of this test setup time could be much better spent on coding or additional testing.
When I announce "please test" email, I suppose around two or three people will actually run something. Personally I ran several depcheck tests and several upgradepath tests (different target repositories). That's it, it's not much, I know. Just the basic testing.
It would be nice to alleviate this manual effort. Staging server could help us greatly. Also the testing framework you're preparing is a great thing (tm). I'm all for automated tests. There is only the question of the extent we want to go in. In the world where 90% of our code is a workaround of some kind (e.g. bodhi comments) and we're constantly re-writing architecture basics I don't see 100% code coverage as a viable approach. But I would love to see some basic tests that cover important areas of our code (e.g. Koji/Bodhi interaction) while requiring reasonable effort to write them.
This isn't meant to be aimless complaining. I have some ideas on how to make the testing of AutoQA easier but I wanted to know if I was missing something obvious before I went too far down that road.
I don't think you're missing anything. It is our project that is missing important parts (it's all question of manpower). Improvements welcome.
Thanks,
Tim
On 04/28/2011 02:46 AM, Kamil Paral wrote:
I really can't tell if this is a particularly bad pain point for me since I work remotely and my internet connection isn't great (frequent SSL timeouts on koji, package downloads take forever)
That's also our case in Brno office. Our internet is painfully slow, it often downloads from Koji around 100kB/s. When working on depcheck I sometimes keep the work for home where I have 10-20 times faster access. So I know what you are talking about.
Ouch, I thought I had it bad at 600kB/s. I'm envious of the faster internet at home, though. I'm currently at the fastest I can get for my area. There is a company offering FTTP but they aren't in my immediate area yet :(
Testing bandwidth-intensive tests is certainly quite problematic without fast access to the main Fedora Infrastructure machines. I don't know how to improve that.
That reminds me, I want to implement package caching for depcheck (and maybe some other tests), that could help us enormously at least for subsequent runs.
Unless I'm misunderstanding you here, I was thinking of something a little less intrusive on our code for the short term: - Modify at least rpm download code to support proxies - Implement a squid cache locally (http://www.squid-cache.org/)
Would our production systems benefit as much from rpm caching? I'm tempted to say leave that for a little later when we have a better idea about how exactly the production infrastructure is going to look and focus on making tests work well and correctly for now.
I don't receive much timeouts however. I guess that means test abort, right? Maybe we could improve that in our library (connection retries).
Yeah, I haven't looked into this much so I'm not really sure why I'm seeing so many timeouts but it's usually during watch-koji-builds which then kills the process.
but I was wondering how you all went about testing AutoQA.
For me, it really depends on what I'm trying to poke at. Most of the time, I'll run 'watch-koji-builds.py --verbose' if I'm trying to do some general testing. I keep track of the events that are called if I want to run something again ('autoqa post-bodhi-update-batch --targettag dist-f13-updates --arch x86_64 --arch i386 oxygen-gtk-1.0.4-1.fc13' as an example).
I usually do "watch-koji-builds.py --verbose --dryrun" to get correct command syntax and then run desired tests by appending "-t TEST (--local)" to the command.
'--dryrun' hadn't occurred to me but that's a good idea. I usually end up killing off all the rpmguard and rpmlint jobs when I'm running locally.
If I'm trying to poke at something very specific, I find myself manually cobbling together some one-off script that runs something specific, like depcheck or upgradepath.
Testing the interaction with Koji or Bodhi? I still haven't figured out a good way to do that. Thus far, I have been hacking in print statements into bodhi_utils or koji_utils but that doesn't quite cover everything.
I ask because speaking for myself, I'm human. The more of a PITA it is to test something, the more likely I am to not do it or limit the number of times that I do test it. I think that I've been pretty good about testing stuff before pushing to master or stable, but I'm bothered by the amount of time I'm wasting on setting up tests and figuring out ways that I can trick AutoQA into going down code paths I want to test.
I understand that. If I want to run depcheck and all builds are accepted ATM, I change the username in fas.conf to consider all builds as unaccepted. It would be nice not have to do this tweaks by hand over and over again.
I've actually started writing manual scripts to download all the packages for depcheck and run the command manually.
It doesn't cover all the code paths but it does allow me to muck with accepted/non-accepted and not have to download things every time.
I'm not saying that testing is a waste of time, just thinking that some of this test setup time could be much better spent on coding or additional testing.
When I announce "please test" email, I suppose around two or three people will actually run something. Personally I ran several depcheck tests and several upgradepath tests (different target repositories). That's it, it's not much, I know. Just the basic testing.
It would be nice to alleviate this manual effort. Staging server could help us greatly. Also the testing framework you're preparing is a great thing (tm). I'm all for automated tests. There is only the question of the extent we want to go in. In the world where 90% of our code is a workaround of some kind (e.g. bodhi comments) and we're constantly re-writing architecture basics I don't see 100% code coverage as a viable approach. But I would love to see some basic tests that cover important areas of our code (e.g. Koji/Bodhi interaction) while requiring reasonable effort to write them.
I'm all for automated testing but I think that we have a long ways to go before we get there. It also doesn't quite solve the problem of being able to induce conditions in the environment for development or debugging.
I guess I'm just wondering where developer pain falls on our priority list. I'm not in any way suggesting that we should drop everything and start making our lives easier but I am thinking about some stuff for farther down the road (might make sense to chip away at some as we go forward)
- Start thinking about better test integration with koji and bodhi - After looking at the bodhi 2.0 roadmap, I get the feeling that we might want to start testing more with bodhi in the staging environment. - Not sure if it makes sense to write a mock bodhi interface to help us induce error conditions. - Maybe work with the bodhi guys (not sure if it's just lmacken) to see what makes sense.
- Think about a more testable interface for AutoQA eventually - More ability to poke at stuff for debugging and devel - Reproducing specific tests, inducing specific error conditions - Allow for more paradigms than just 'download from koji, run tests' - I'm thinking in terms of pluggable modules but that is going to be a gradual process
But we have bigger fish to fry for the immediate future :) I should probably put these in the form of tickets if there is any interest.
This isn't meant to be aimless complaining. I have some ideas on how to make the testing of AutoQA easier but I wanted to know if I was missing something obvious before I went too far down that road.
I don't think you're missing anything. It is our project that is missing important parts (it's all question of manpower). Improvements welcome.
I'm all for improvements but I think this will be a gradual process (as I'm sure you're thinking). Food for thought, anyways.
Tim
That reminds me, I want to implement package caching for depcheck (and maybe some other tests), that could help us enormously at least for subsequent runs.
Unless I'm misunderstanding you here, I was thinking of something a little less intrusive on our code for the short term:
- Modify at least rpm download code to support proxies
- Implement a squid cache locally (http://www.squid-cache.org/)
Oh, that didn't occur to me. Anyway, too late! I'm polishing the patch and will send it for review soon.
Would our production systems benefit as much from rpm caching? I'm tempted to say leave that for a little later when we have a better idea about how exactly the production infrastructure is going to look and focus on making tests work well and correctly for now.
My current patch is targeted at developers more than at production machines. They usually have fast Internet access.
I don't receive much timeouts however. I guess that means test abort, right? Maybe we could improve that in our library (connection retries).
Yeah, I haven't looked into this much so I'm not really sure why I'm seeing so many timeouts but it's usually during watch-koji-builds which then kills the process.
In my patch I have enabled connection retries in urlgrabber. I didn't look into koji module calls, possible place for improvement.
'--dryrun' hadn't occurred to me but that's a good idea. I usually end up killing off all the rpmguard and rpmlint jobs when I'm running locally.
Very useful. Just don't forget to prune /var/cache/autoqa before that. It's not a good idea to ask Koji for packages of the last two weeks (instead of the last three hours). :)
- Start thinking about better test integration with koji and bodhi
- After looking at the bodhi 2.0 roadmap, I get the feeling that we
might want to start testing more with bodhi in the staging environment.
- Not sure if it makes sense to write a mock bodhi interface to help
us induce error conditions.
- Maybe work with the bodhi guys (not sure if it's just lmacken) to
see what makes sense.
- Think about a more testable interface for AutoQA eventually
- More ability to poke at stuff for debugging and devel
- Reproducing specific tests, inducing specific error conditions
- Allow for more paradigms than just 'download from koji, run tests'
- I'm thinking in terms of pluggable modules but that is going to be
a gradual process
But we have bigger fish to fry for the immediate future :) I should probably put these in the form of tickets if there is any interest.
Tickets welcome. However I wouldn't file them too far into the future, because usually the future is much far then expected and then the tickets stay Trac for ages (and the project evolves in different direction anyway) :-)
Thanks, Kamil
On 04/28/2011 10:47 AM, Kamil Paral wrote:
Unless I'm misunderstanding you here, I was thinking of something a little less intrusive on our code for the short term: - Modify at least rpm download code to support proxies - Implement a squid cache locally (http://www.squid-cache.org/)
Oh, that didn't occur to me. Anyway, too late! I'm polishing the patch and will send it for review soon.
If you've already got something almost working, I'm not going to fight you on the implementation :)
In my patch I have enabled connection retries in urlgrabber. I didn't look into koji module calls, possible place for improvement.
I'll start looking into koji retries soon since I seem to be the one hitting that particular issue. It might be a couple of days, though.
'--dryrun' hadn't occurred to me but that's a good idea. I usually end up killing off all the rpmguard and rpmlint jobs when I'm running locally.
Very useful. Just don't forget to prune /var/cache/autoqa before that. It's not a good idea to ask Koji for packages of the last two weeks (instead of the last three hours). :)
I learned my lesson after forgetting to do that a couple of times :) IIRC, 120 or so jobs were created on my development system and it took a LONG time to grab all the data from koji/bodhi.
- Start thinking about better test integration with koji and bodhi
- After looking at the bodhi 2.0 roadmap, I get the feeling that
we might want to start testing more with bodhi in the staging environment. - Not sure if it makes sense to write a mock bodhi interface to help us induce error conditions. - Maybe work with the bodhi guys (not sure if it's just lmacken) to see what makes sense.
- Think about a more testable interface for AutoQA eventually -
More ability to poke at stuff for debugging and devel - Reproducing specific tests, inducing specific error conditions - Allow for more paradigms than just 'download from koji, run tests' - I'm thinking in terms of pluggable modules but that is going to be a gradual process
But we have bigger fish to fry for the immediate future :) I should probably put these in the form of tickets if there is any interest.
Tickets welcome. However I wouldn't file them too far into the future, because usually the future is much far then expected and then the tickets stay Trac for ages (and the project evolves in different direction anyway) :-)
Good point, I'll hold off on them for a little bit. They might fit in well with some of the longer term things we want to do with supporting independent test development (not just the git idea we currently have for 0.5.0).
Tim
Late to the party ...
On Thu, 2011-04-28 at 09:53 -0600, Tim Flink wrote:
On 04/28/2011 02:46 AM, Kamil Paral wrote:
I really can't tell if this is a particularly bad pain point for me since I work remotely and my internet connection isn't great (frequent SSL timeouts on koji, package downloads take forever)
That's also our case in Brno office. Our internet is painfully slow, it often downloads from Koji around 100kB/s. When working on depcheck I sometimes keep the work for home where I have 10-20 times faster access. So I know what you are talking about.
Ouch, I thought I had it bad at 600kB/s. I'm envious of the faster internet at home, though. I'm currently at the fastest I can get for my area. There is a company offering FTTP but they aren't in my immediate area yet :(
Testing bandwidth-intensive tests is certainly quite problematic without fast access to the main Fedora Infrastructure machines. I don't know how to improve that.
That reminds me, I want to implement package caching for depcheck (and maybe some other tests), that could help us enormously at least for subsequent runs.
Unless I'm misunderstanding you here, I was thinking of something a little less intrusive on our code for the short term:
- Modify at least rpm download code to support proxies
- Implement a squid cache locally (http://www.squid-cache.org/)
Would our production systems benefit as much from rpm caching? I'm tempted to say leave that for a little later when we have a better idea about how exactly the production infrastructure is going to look and focus on making tests work well and correctly for now.
Yes they would! When we are able to do more automated testing against ISO images (netinst and DVD.iso), I was planning to invest some time to explore squid. Having each test system download the ISO images over and over again would be horrible.
Thanks, James
On 04/29/2011 08:20 AM, James Laska wrote:
Would our production systems benefit as much from rpm caching? I'm tempted to say leave that for a little later when we have a better idea about how exactly the production infrastructure is going to look and focus on making tests work well and correctly for now.
Yes they would! When we are able to do more automated testing against ISO images (netinst and DVD.iso), I was planning to invest some time to explore squid. Having each test system download the ISO images over and over again would be horrible.
How close are we to doing this? I'm pretty sure that Hongqing is working on this but I have no idea where he is on the work.
How many systems would we have deployed that took advantage of this cache? What is the advantage of having each system cache the DVDs instead of re-downloading them from koji if everything is in the same datacenter?
If we start going that route, it might be interesting to look into some sort of local distributed datastore (gluster, sheepdog, GFS2 ...) but we would have to think about whether it was worth the extra effort first.
Tim
On Fri, 2011-04-29 at 15:17 -0600, Tim Flink wrote:
On 04/29/2011 08:20 AM, James Laska wrote:
Would our production systems benefit as much from rpm caching? I'm tempted to say leave that for a little later when we have a better idea about how exactly the production infrastructure is going to look and focus on making tests work well and correctly for now.
Yes they would! When we are able to do more automated testing against ISO images (netinst and DVD.iso), I was planning to invest some time to explore squid. Having each test system download the ISO images over and over again would be horrible.
How close are we to doing this? I'm pretty sure that Hongqing is working on this but I have no idea where he is on the work.
Not tremendously close. Hongqing is exploring different technologies, while at the same time developing a roadmap for the test suite. Until we have the roadmap defined, I'd hesitate to set any development timelines.
How many systems would we have deployed that took advantage of this cache? What is the advantage of having each system cache the DVDs instead of re-downloading them from koji if everything is in the same datacenter?
We could certainly have each test system download the images, but that seems wasteful and time consuming (esp for 4G DVD images). I hadn't thought about the test systems caching the ISOs, although we could certainly explore that. I was thinking that the AutoQA server would be responsible for caching behind the scenes.
Getting a little ahead of myself, but an early idea was to have 2 watchers/events ... 1. The first event that detects the presence of new ISO images on serverbeach1 - I have a crude, but effective script that does this already. This watcher would trigger a test that locally caches the ISO's and presents them for use by all ISO-based install tests 2. A second event would then detect when new ISO images have been locally downloaded and are available for testing (either means just a faster link, or they are shared over NFS ... not sure).
Of course, I think we could probably avoid all this multi-event complexity by just using squid on the AutoQA server to do any caching behind the scenes?
If we start going that route, it might be interesting to look into some sort of local distributed datastore (gluster, sheepdog, GFS2 ...) but we would have to think about whether it was worth the extra effort first.
Shudder :) Open to the idea if there are any KISS solutions available. We'll see how the install automation roadmap progresses and what sort of timelines that demands for these ideas.
Thanks, James
On 05/02/2011 06:45 AM, James Laska wrote:
How close are we to doing this? I'm pretty sure that Hongqing is working on this but I have no idea where he is on the work.
Not tremendously close. Hongqing is exploring different technologies, while at the same time developing a roadmap for the test suite. Until we have the roadmap defined, I'd hesitate to set any development timelines.
OK, that makes sense. Just making sure that there wasn't anything imminent that I wasn't aware of.
How many systems would we have deployed that took advantage of this cache? What is the advantage of having each system cache the DVDs instead of re-downloading them from koji if everything is in the same datacenter?
We could certainly have each test system download the images, but that seems wasteful and time consuming (esp for 4G DVD images). I hadn't thought about the test systems caching the ISOs, although we could certainly explore that. I was thinking that the AutoQA server would be responsible for caching behind the scenes.
I thought that all the fp.o machines were in the same datacenter. Am I wrong here?
If the autoqa machines don't have to hit the WAN to get the ISOs, I'm not sure that doing a squid cache (that it also over the LAN) would make for much of an improvement.
I'm not sure how wise it would be to have the test machines caching too much, assuming that I understand our eventual HW roadmap (more virt, possibly throwaway systems at some point in the glorious future) and the storage resources available for each system.
As usual, it would be a balance between immediate effort, payoff and how far out the HW changes would be. Either way, probably not worth worrying about for the moment.
Getting a little ahead of myself, but an early idea was to have 2 watchers/events ... 1. The first event that detects the presence of new ISO images on serverbeach1 - I have a crude, but effective script that does this already. This watcher would trigger a test that locally caches the ISO's and presents them for use by all ISO-based install tests 2. A second event would then detect when new ISO images have been locally downloaded and are available for testing (either means just a faster link, or they are shared over NFS ... not sure).
Of course, I think we could probably avoid all this multi-event complexity by just using squid on the AutoQA server to do any caching behind the scenes?
We'd still have to detect the event and kick off tests, but a squid cache should make things easier - as long as the tests don't all kick off at the same time. I'm not sure how well squid handles multiple requests for the same large file at the same time - it might post-cache, leaving all the downloads to run across the WAN.
If we start going that route, it might be interesting to look into some sort of local distributed datastore (gluster, sheepdog, GFS2 ...) but we would have to think about whether it was worth the extra effort first.
Shudder :) Open to the idea if there are any KISS solutions available. We'll see how the install automation roadmap progresses and what sort of timelines that demands for these ideas.
I'd still lean more toward some sort of shared filesystem, but I agree that it would have to be a balance between functionality and effort (both setup and maintenance). I also agree that it would be silly to plan this out more now - way too far in the future at this point.
Tim
On Mon, 2011-05-02 at 12:01 -0600, Tim Flink wrote:
On 05/02/2011 06:45 AM, James Laska wrote:
How close are we to doing this? I'm pretty sure that Hongqing is working on this but I have no idea where he is on the work.
Not tremendously close. Hongqing is exploring different technologies, while at the same time developing a roadmap for the test suite. Until we have the roadmap defined, I'd hesitate to set any development timelines.
OK, that makes sense. Just making sure that there wasn't anything imminent that I wasn't aware of.
How many systems would we have deployed that took advantage of this cache? What is the advantage of having each system cache the DVDs instead of re-downloading them from koji if everything is in the same datacenter?
We could certainly have each test system download the images, but that seems wasteful and time consuming (esp for 4G DVD images). I hadn't thought about the test systems caching the ISOs, although we could certainly explore that. I was thinking that the AutoQA server would be responsible for caching behind the scenes.
I thought that all the fp.o machines were in the same datacenter. Am I wrong here?
They are, but our tests can be run from anywhere. Early on, it was a requirement that no test assume it's running inside
If the autoqa machines don't have to hit the WAN to get the ISOs, I'm not sure that doing a squid cache (that it also over the LAN) would make for much of an improvement.
I honestly don't know how the actual bits flow, but if you are right ... it certainly wouldn't make sense to use squid here. Although, perhaps that's an enhancement we leave (or document) for folks running their own AutoQA "labs".
I'm not sure how wise it would be to have the test machines caching too much, assuming that I understand our eventual HW roadmap (more virt, possibly throwaway systems at some point in the glorious future) and the storage resources available for each system.
I agree, I hadn't really thought much about test client caching until it was raised in this thread.
As usual, it would be a balance between immediate effort, payoff and how far out the HW changes would be. Either way, probably not worth worrying about for the moment.
Yeah, I agree.
Getting a little ahead of myself, but an early idea was to have 2 watchers/events ... 1. The first event that detects the presence of new ISO images on serverbeach1 - I have a crude, but effective script that does this already. This watcher would trigger a test that locally caches the ISO's and presents them for use by all ISO-based install tests 2. A second event would then detect when new ISO images have been locally downloaded and are available for testing (either means just a faster link, or they are shared over NFS ... not sure).
Of course, I think we could probably avoid all this multi-event complexity by just using squid on the AutoQA server to do any caching behind the scenes?
We'd still have to detect the event and kick off tests, but a squid cache should make things easier - as long as the tests don't all kick off at the same time. I'm not sure how well squid handles multiple requests for the same large file at the same time - it might post-cache, leaving all the downloads to run across the WAN.
Okay
If we start going that route, it might be interesting to look into some sort of local distributed datastore (gluster, sheepdog, GFS2 ...) but we would have to think about whether it was worth the extra effort first.
Shudder :) Open to the idea if there are any KISS solutions available. We'll see how the install automation roadmap progresses and what sort of timelines that demands for these ideas.
I'd still lean more toward some sort of shared filesystem, but I agree that it would have to be a balance between functionality and effort (both setup and maintenance). I also agree that it would be silly to plan this out more now - way too far in the future at this point.
Definitely. Good/fun to explore options, and I definitely have a better sense about the benefits/drawbacks of the squid approach from this thread. It'll be good to kick the tires on this thread as the auto-install development roadmap matures and we get to the point of scheduling media-based install tests.
Thanks, James
autoqa-devel@lists.fedorahosted.org
-
James Laska -
Kamil Paral -
Tim Flink