autoqa-devel April 2010

autoqa-devel@lists.fedorahosted.org

7 participants
65 discussions

bodhi-2.0 as a resultsdb front-end for package acceptance
by James Laska 20 Apr '10

20 Apr '10

Greetings gang, In discussion with wwoods today, he made an interesting point regarding the upcoming bodhi rewrite. Seemed like it might be good to raise for discussion to a larger audience. The quick version, Bodhi will be a resultsdb front-end for package acceptance test results [1]. I think this mostly has an impact on ticket#145 [2]. We still may need some *basic* front-end to validate data display/entry, this seemed like a good topic to send out for wider review. Thanks, James [1] https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptan… [2] https://fedorahosted.org/autoqa/ticket/145

1 0

2010-04-14 - Bodhi and AutoQA recap
by James Laska 19 Apr '10

19 Apr '10

# Bodhi Updates and AutoQA Discussion # Date: 2010-04-14 # Time: 14:00 UTC (11:00 EDT, 15:00 CET) # Participants: lmacken, kparal, jlaska, jkeating, wwoods = Discussion Topics = == Goals == * Do *not* add more steps for package maintainers (unless the autoqa test fails) == Background: depcheck == * triggered on post-bodhi-update hook * inputs: update NVRs, target repo, [other pending packages] ** Why "other pending packages"? *** Can't handle interdependent packages without it == Current Bodhi == * What states/stages do updates go through? ** See workflow below * How do we recognize them through the API? ** See workflow below == Review bodhi update workflow == Package built by maintainer | koji tag dist-fxx-updates-candidate V Bodhi update created by maintainer | koji tag dist-fxx-updates-candidate | bodhi (status: pending, request: None) V Bodhi update requested by maintainer (for stable or testing) | koji tag dist-fxx-updates-candidate | bodhi (status: pending, request: ['testing' or 'stable']) | bodhi tags the package with the *-pending tag, sends message to bus V pending | could be koji tag dist-fxx-updates(-testing)-pending | koji uses newRepo to make repos V Wait for newRepo <---+ | | | FAIL (remains in pending) V | Acceptance testing ---+ | PASS - autoqa gives +1 to update (future work: autoqa sets 'test-ok' tag/field) | V Available for pushing (still in dist-fxx-updates(-testing)-pending) | V Pushed by releng (package moved from -candidate/-testing to -updates(-testing) and drops -pending tag) (-testing updates can now go to "update requested" with request: 'stable') == What does depcheck/autoqa need? == * Needs a repo of pending packages * Package sanity requirement: https://fedorahosted.org/autoqa/ticket/139 ** Multiple tests are needed for complete package acceptance, the autoqa +1 should happen after all have completed * un-depsolved packages = the *-pending tags * depsolved packages (eligible for push) = *-pending tag & +1 from autoqa = Open Questions = Q. Are we going to need a lot more koji tags? A. The tag could live on the bodhi side, just need a way to determine which builds are eligible for pushing to requested repo (stable, testing) Q. How to note that AutoQA has passed acceptance tests? A. Agreement that there should be AutoQA specific karma in bodhi. This lets maintainers and other testers see that the response is from an automated tool Q. Re-test everything once a maintainer retracts an update request? A. Yes - any time the -pending set changes, we need to re-run the depcheck/acceptance test Q. What happens when things change and an update which once passed acceptance no longer does? A. Q. What about the tests beyond depcheck, individual package acceptance tests A. This process would only relate to package acceptance testing in accordance with the criteria (https://fedoraproject.org/wiki/Package_update_acceptance_criteria) Functional testing will occur once the packages/updates are moved to updates-testing. Q. How to handle packages that failed testing, and then a new NVR comes in a new update request? A. Q. If a maintainer revokes a push request what now? A. Need to remove the -pending tag Q. What do we do when a firefox security update is headed out, but breaks other packages? A. Q. What to do when acceptance test fails A. Don't add negative karma, add karma 0 with a comment in the bodhi request Q. We have 4 planned automated acceptance tests, how to make sure all run before approving the update A. = Action Items = [ ] - Update the bodhi-1.0 watcher to detect builds (quick'n'dirty method to watch for koji *-pending tags) (wwoods) [ ] - Integrate use of mash into acceptance tests (jkeating && wwoods) [ ] - Create new koji tags for *-pending (jkeating) [ ] - Bodhi-1.0 changes? (lmacken) [ ] - Add -pending tag when update is requested [ ] - Magic handling for autoqa's +1 karma (only show autoqa-approved stuff) [ ] - Bodhi-2.0 roadmap changes? - Tickets in bodhi trac, against 2.0 milestone [ ] - Move package acceptance test plan out of Draft (kparal && jlaska) = Next Meeting = 2010-04-28 - Another 'touch-base' meeting

2 1

2010-04-16 - AutoQA resultsdb meeting recap
by James Laska 16 Apr '10

16 Apr '10

# AutoQA ResultsDB Discussion # Date: 2010-04-16 # Time: 14:00 UTC (10:00 EDT, 15:00 CET) # Attendees: kparal, jskladan, wwoods, jlaska = Previous action items = * [jskladan+kparal] Create updated DB schema according to the discussion ** https://fedoraproject.org/wiki/AutoQA_resultsdb_schema#Result -> rename 'Test' table to 'TestCase' -> type of the test is not stored in the DB, add it somewhere, or use tags? We must have the mandatory keyvals stored in the database (cached? from wiki) to have fast access to the keyval pairs ->-> we don't have to keep this data inside DB if we are just happy with that result not displaying in the frontend; DB itself doesn't care about invalid data * [wwoods?] Study MediaWiki RPC mechanism * [kparal] Define default (common) key-values for basic test classes (installation tests, repo tests, package tests) ** https://fedoraproject.org/wiki/AutoQA_resultsdb_schema#Default_key-values_f… -> install key/val pairs exist in .treeinfo (http://download.fedoraproject.org/pub/fedora/linux/development/13/i386/os/.…) * [jskladan] Communicate with Autotest developers how to get Job ID in autotest client ** https://fedorahosted.org/pipermail/autoqa-devel/2010-March/000326.html * [jlaska] Try to set up a qpid instance and send some test data through it (from provider to listener) (jkeating (Oxf13) is a good person to talk to about this!)UI ** https://fedorahosted.org/autoqa/ticket/132#comment:1 = ResultsDB API = * 'Input' interface <https://fedoraproject.org/wiki/AutoQA_resultsdb_API> * Is this satisfactory? Do we any other methods for pushing data into the ResultsDB * Tagging - kparal and I discussed, if there could be any reasonable tag, which one would like to use _from inside_ the testrun. (All the tags I could think of were either about Tests, not Testruns, or they'd make sense to be applied after a real person went through the results.) * 'Output' interface * Fedora Message Bus (FMB) * What's the state? -> In progress, Bodhi will may soon talk over the bus, * Will we set up some testing instance? -> There should be running Qpid instance somewhere -> talk to infrastructure guys about current state and if/how can we use it * My understanding of using the FMB is, that we'll just post new 'events' to the FMB, and whoever wants to will consume that information, is that right? If so, will the consumers be responsible for storing the data, for their further use, or will they be able to query ResultsDB? -> Mostly we'll send events, that will inform users that 'new data is in the resultsDB' * Querying ResultsDB * What is the best approach 1) Set of predefined functions 2) Possibility to execute SQL queries -> not secure, if needed we should add API 3) Combination of both * Predefined functions: * What functions will we provide at the beginning? -> we'll create a set of functions _we_ need at the beginning (we'll find out, while creating the front-end) * Will we add more if somebody asks? * How to tell, if the demand is valid? = Open Questions = Q. Whether or not front-ends will be message bus consumers, or query the resultsdb directly? A. Q. Are we at the point where we can add a built-in unit test suite for resultsdb? A. Jskladan noted there is a ticket for this already (see https://fedorahosted.org/autoqa/ticket/144) Q. How to proceed with prototyping? A. Jskladan plans to experiment with a python xmlrpc to get started. Wwoods recommended moving to TurboGears when the time is right. Q. Next steps? A. Prototype the API ... use this to get data in and out of resultsdb. Wwoods noted that it can be helpful to think of a specific use case first (installation, package acceptance). Q. What front-end should we focus on first. A. Wwoods+jlaska felt the current draft package acceptance test plan (https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptan…) was best to start with. Q. What to do with 'Is rawhide broken' test plan? A. One suggestion was to rename it to 'Is branched broken'. = Action Items = * [all] *Keep It Simple* * [ ] - talk to Fedora Infrastructure to figure out current msgbus state and how to use it * [ ] - ask lmacken || toshio whether TurboGears2 is still prefered Fedora solution

1 0

2010-04-16 - AutoQA resultsdb meeting agenda
by James Laska 16 Apr '10

16 Apr '10

# AutoQA ResultsDB Discussion # Date: 2010-04-16 # Time: 14:00 UTC (10:00 EDT, 15:00 CET) # Where: gobby Hey folks, Several of you have requested a gathering to review progress and discuss any challenges/roadblocks related to the resultsdb milestone [1]. Other than checking in on previous tasks [2], I don't have any specific topics to cover. Please feel free to reply with key points you'd like to discuss. We can try using gobby [3] and Fedora Talk [4] again. Let's plan to meet on #fedora-qa and get things started. Thanks, James [1] https://fedorahosted.org/autoqa/milestone/Resultdb [2] https://fedorahosted.org/pipermail/autoqa-devel/2010-March/000318.html [3] https://fedoraproject.org/wiki/Gobby [4] http://talk.fedoraproject.org/

2 1

Re: [PATCH] Ticket 65, 130: getting autotest test-ID to the client
by Kamil Paral 14 Apr '10

14 Apr '10

----- "Josef Skladanka" <jskladan(a)redhat.com> wrote: > Added new option to autoqa --autotest-server which can be used > to set hostname of the autotest server, which will hold the results > of the test. Since our infrastructure has Autotest and AutoQA > on the same machine, default value for this option is the hostname > of current machine (socket.gethostname()). Maybe it would be nice also to support this option in /etc/autoqa.conf, so in case we need to use it we don't have to specify the command line option all the time.

3 3

[PATCH] Ticket 65, 130: getting autotest test-ID to the client
by Josef Skladanka 14 Apr '10

14 Apr '10

Added new option to autoqa --autotest-server which can be used to set hostname of the autotest server, which will hold the results of the test. Since our infrastructure has Autotest and AutoQA on the same machine, default value for this option is the hostname of current machine (socket.gethostname()). Also the templates of control were updated, so they propagate the basic URL to the Autotest results directory. Test_class_templates were updated, so they create the whole URL to the results. --- autoqa | 6 ++++++ hooks/post-koji-build/control.template | 3 ++- hooks/post-koji-build/test_class_template.py | 5 ++++- hooks/post-repo-update/control.template | 3 ++- hooks/post-repo-update/test_class_template.py | 5 ++++- hooks/post-tree-compose/control.template | 3 ++- hooks/post-tree-compose/test_class_template.py | 5 ++++- 7 files changed, 24 insertions(+), 6 deletions(-) diff --git a/autoqa b/autoqa index efb94e3..9f41c00 100755 --- a/autoqa +++ b/autoqa @@ -27,6 +27,7 @@ import optparse import tempfile import StringIO import urlgrabber +import socket from ConfigParser import * from subprocess import call @@ -139,6 +140,9 @@ parser.add_option('--local', action='store_true', dest='local', help='Do not schedule jobs - run test(s) directly on the local machine') parser.add_option('-l', '--list-tests', action='store_true', dest='listtests', help='list the tests for the given hookname - do not run any tests') +parser.add_option('--autotest-server', action='store', default=socket.gethostname(), + help='Sets the autotest-server hostname. Used for creating URLs to results.\ +Hostname of the local machine is used by default.') # Read and validate the hookname # Check for no args, or just -h/--help if len(sys.argv) == 1 or sys.argv[1] in ('-h', '--help'): @@ -197,6 +201,8 @@ for arch in opts.arch: # N.B. process_testdata may grow new keyword arguments if we add new autoqa # args that add another loop here.. testdata = hook.process_testdata(opts, args, arch=arch) + if not 'autotest_server' in testdata.keys(): + testdata['autotest_server'] = opts.autotest_server # XXX FIXME: tests need to be able to indicate that they do not require # any specific arch (e.g. rpmlint can run on any arch) for test in testlist: diff --git a/hooks/post-koji-build/control.template b/hooks/post-koji-build/control.template index b90260f..b7c5261 100644 --- a/hooks/post-koji-build/control.template +++ b/hooks/post-koji-build/control.template @@ -15,4 +15,5 @@ TEST_CATEGORY = 'Functional' # kojitag: koji tag applied to this package job.run_test('testclassname', name=name, envr=envr, kojitag=kojitag, - config=autoqa_conf) + config=autoqa_conf, + autotest_url="http://%s/results/%s/" % (autotest_server, job.tag)) diff --git a/hooks/post-koji-build/test_class_template.py b/hooks/post-koji-build/test_class_template.py index cf91865..462dc91 100644 --- a/hooks/post-koji-build/test_class_template.py +++ b/hooks/post-koji-build/test_class_template.py @@ -35,7 +35,10 @@ class post_repo_update_test_class_name(test.test): #utils.system('yum -y install yum-utils') pass - def run_once(self, envr, name, kojitag): + def run_once(self, envr, name, kojitag, autotest_url): + # results_url points to the directory on autotest-server with + #the sotred results + results_url = "%s%s/" % (autotest_url, self.__class__.__name__) parentlist = ' '.split(parents) cmd = 'test_binary --url %s' % baseurl # You can get stuff from the [test] section of autoqa.conf like this diff --git a/hooks/post-repo-update/control.template b/hooks/post-repo-update/control.template index 50ba58d..204dc69 100644 --- a/hooks/post-repo-update/control.template +++ b/hooks/post-repo-update/control.template @@ -19,4 +19,5 @@ TEST_CATEGORY = 'Functional' job.run_test('testclassname', baseurl=url, parents=parents, reponame=reponame, - config=autoqa_conf) + config=autoqa_conf, + autotest_url="http://%s/results/%s/" % (autotest_server, job.tag)) diff --git a/hooks/post-repo-update/test_class_template.py b/hooks/post-repo-update/test_class_template.py index e274f98..c0c0137 100644 --- a/hooks/post-repo-update/test_class_template.py +++ b/hooks/post-repo-update/test_class_template.py @@ -33,7 +33,10 @@ class post_repo_update_test_class_name(test.test): #utils.system('yum -y install yum-utils') pass - def run_once(self, baseurl, parents, reponame): + def run_once(self, baseurl, parents, reponame, autotest_url): + # results_url points to the directory on autotest-server with + #the sotred results + results_url = "%s%s/" % (autotest_url, self.__class__.__name__) parentlist = ' '.split(parents) cmd = 'test_binary --url %s' % baseurl # You can get stuff from the [test] section of autoqa.conf like this diff --git a/hooks/post-tree-compose/control.template b/hooks/post-tree-compose/control.template index e327799..ef8ea35 100644 --- a/hooks/post-tree-compose/control.template +++ b/hooks/post-tree-compose/control.template @@ -17,4 +17,5 @@ TEST_CATEGORY = 'Functional' job.run_test('testclassname', baseurl=url, treename=treename, - config=autoqa_conf) + config=autoqa_conf, + autotest_url="http://%s/results/%s/" % (autotest_server, job.tag)) diff --git a/hooks/post-tree-compose/test_class_template.py b/hooks/post-tree-compose/test_class_template.py index 0ffab9d..6084fb1 100644 --- a/hooks/post-tree-compose/test_class_template.py +++ b/hooks/post-tree-compose/test_class_template.py @@ -33,7 +33,10 @@ class post_tree_compose_test_class_name(test.test): #utils.system('yum -y install yum-utils') pass - def run_once(self, baseurl, treename): + def run_once(self, baseurl, treename, autotest_url): + # results_url points to the directory on autotest-server with + #the sotred results + results_url = "%s%s/" % (autotest_url, self.__class__.__name__) cmd = 'test_binary --url %s' % baseurl # You can get stuff from the [test] section of autoqa.conf like this email = self.config.get('test','result_email') -- 1.6.6.1

1 0

[AutoQA] #139: pst: Find a way how to test packages coming to updates-testing
by fedora-badges 14 Apr '10

14 Apr '10

#139: pst: Find a way how to test packages coming to updates-testing --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: Package Sanity Tests Component: tests | Version: 1.0 Keywords: pst | --------------------+------------------------------------------------------- We must find a way how to test package sanity of packages coming to updates-testing, that means those packages there are not yet in any repo. The problem here are the dependencies, which may already be present in updates-testing (ideal case), but often may not be. In this case the update author should have probably specified all depending packages in one "set" when creating the update in Bodhi. We must communicate with Bodhi, find out this set, detect all packages this set contains, download them, and install them together with our main package. Does Bodhi have API capable of this? -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/139> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

Re: [AutoQA] #65: Make israwhidebroken test results have links back to autotest results
by fedora-badges 13 Apr '10

13 Apr '10

#65: Make israwhidebroken test results have links back to autotest results ------------------------+--------------------------------------------------- Reporter: wwoods | Owner: jskladan Type: task | Status: new Priority: major | Milestone: Fedora 13 Component: reporting | Version: 1.0 Resolution: | Keywords: ------------------------+--------------------------------------------------- Changes (by jskladan): * owner: wwoods => jskladan Comment: I'm working on this in the resultsdb scope (#130), so i'm reasigning this ticket to myself. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/65#comment:4> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

[AutoQA] #142: ResultsDB: create resultsdb prototype - set up database
by fedora-badges 13 Apr '10

13 Apr '10

#142: ResultsDB: create resultsdb prototype - set up database ----------------------------+----------------------------------------------- Reporter: jskladan | Owner: jskladan Type: task | Status: new Priority: major | Milestone: Resultdb Component: infrastructure | Version: 1.0 Keywords: | ----------------------------+----------------------------------------------- Sub-task for #134 - set up database reflecting the schema from #133 -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/142> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #146: ResultsDB: Propose ResultsDB API - API for the frontends
by fedora-badges 13 Apr '10

13 Apr '10

#146: ResultsDB: Propose ResultsDB API - API for the frontends -----------------------+---------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: major | Milestone: Resultdb Component: docs/wiki | Version: 1.0 Keywords: | -----------------------+---------------------------------------------------- In #131, we have a API for the tests side of the resultsdb, i.e. API for storing the information. We should discuss the API we want to provide for digging information from the resultsdb - do we want the frontends to have limited set of functions, of do we want to give them a chance to run any select/join query they'd like? Binded with #145 -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/146> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel April 2010