#154: Tracker: critical path test case creation -------------------------+-------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Keywords: | -------------------------+-------------------------------------------------- = problem =
We need more specific per-package guidance for proven tester testing, to make sure the testing covers all the relevant critical path functionality of each package.
= enhancement recommendation =
We can create a group of test cases in the Wiki for each critical path package. Bodhi could integrate with this system and, in the future non- numeric karma system, provide checkboxes for each test case for packages which have test cases available. fedora-easy-karma could provide an interface to display and check off test cases from the Wiki.
This ticket can serve as a tracker for this process, which will be ongoing. To start with I will file tickets for each package which has so far been identified (on test and devel mailing lists) as an obvious candidate for such test cases, with a note of what should be covered for each package. Those tickets will block this one.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by adamwill):
* cc: jlaska (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by johannbg):
* cc: johannbg (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by fenris02):
* cc: fenris02 (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by masami):
* cc: masami (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by fcami):
* cc: fcami (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by rhe):
* cc: rhe (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by jlaska):
* milestone: => Fedora 15
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by ykopkova):
* cc: ykopkova (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
here's a draft of a sample test case, for mdadm:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_testcase_mdadm_package...
the production name would be QA:Testcase_mdadm_package_software , with QA:Testcase being our standard prefix for test cases (it's not up for debate within the context of this ticket, if we change it we must change it for all test cases), then 'mdadm' being the source package name, 'package' indicating this (that it's named based on a src.rpm name - perhaps it should come before the name rather than after?), and 'software' being the name of this particular test (referring to software RAID).
what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great. thanks.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:9 adamwill]:
here's a draft of a sample test case, for mdadm:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_testcase_mdadm_package...
the production name would be QA:Testcase_mdadm_package_software , with
QA:Testcase being our standard prefix for test cases (it's not up for debate within the context of this ticket, if we change it we must change it for all test cases), then 'mdadm' being the source package name, 'package' indicating this (that it's named based on a src.rpm name - perhaps it should come before the name rather than after?), and 'software' being the name of this particular test (referring to software RAID).
Seems sensible from a general best practices point of view (QA:Testcase_%{sourcerpm}_%{userdefined}). The "_package" seems extraneous, but I might be missing something. With regards to finding applicable tests for a certain update, the most important thing is the category structure, right? That's how other tools will locate applicable tests.
what does everyone think, particularly of the naming and the
categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great.
In the page you linked, that test is in three categories ... 1. Category:critpath_test_cases 2. Category:mdadm_test_cases 3. Category:mdadm_critpath_test_cases I don't yet see the big picture for how the Categories will be grouped/organized, can you provide a hierarchical example?
On 12/13/2010 01:28 PM, Fedora QA wrote:
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
here's a draft of a sample test case, for mdadm:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_testcase_mdadm_package...
the production name would be QA:Testcase_mdadm_package_software , with QA:Testcase being our standard prefix for test cases (it's not up for debate within the context of this ticket, if we change it we must change it for all test cases), then 'mdadm' being the source package name, 'package' indicating this (that it's named based on a src.rpm name - perhaps it should come before the name rather than after?), and 'software' being the name of this particular test (referring to software RAID).
what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great. thanks.
Seems straightforward. I assume it doesn't make any difference if bios raid devices are being used as well as mdadm created devices.
Regards, OldFart
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
the idea of the 'package' word is to clarify that the next word is a .src.rpm package name and not, well, anything else. I can see confusion arising with some .src.rpm names that are ambiguous and could be used in test case names in entirely different ways. but it's not strictly needed, no, with the categories.
"I don't yet see the big picture for how the Categories will be grouped/organized, can you provide a hierarchical example? "
that's one of the areas I'm not feeling totally sure of yet, but the idea of the organization in the example ticket is that *any* test case intended to aid critpath testing of any critpath package will be in 'critpath_test_cases', any test case relating to mdadm - even if it's not testing critpath functionality - will be in mdadm_test_cases, and just tests that are for mdadm and are for testing critpath functionality will be in mdadm_critpath_test_cases . The third set is the combination of the first two, basically. I suppose you can search for pages that are in both of two categories and then you wouldn't need the third set.
again, this is one i don't think is right yet, so better suggestions for doing the categorization are welcome.
On Mon, 2010-12-13 at 15:43 -0500, Clyde E. Kunkel wrote:
what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great. thanks.
Seems straightforward. I assume it doesn't make any difference if bios raid devices are being used as well as mdadm created devices.
actually, it does. Most BIOS RAID devices aren't handled by mdadm (only recent Intel BIOS RAID devices are), and there are different codepaths there, so theoretically, software RAID and Intel BIOS RAID could be in different states - one working and the other broken. The Grand Plan would be to have separate test cases for each.
But the content of this specific test case isn't really the important thing here, it's just an example to give a feel for how the whole critpath test case setup will look. I threw it together in half an hour or so, it wasn't carefully worked out (or, well, tested :>)
On 12/13/2010 04:06 PM, Adam Williamson wrote:
On Mon, 2010-12-13 at 15:43 -0500, Clyde E. Kunkel wrote:
what does everyone think, particularly of the naming and the categorization? I don't quite feel that this one's nailed down yet, if anyone has smart ideas that'd be great. thanks.
Seems straightforward. I assume it doesn't make any difference if bios raid devices are being used as well as mdadm created devices.
actually, it does. Most BIOS RAID devices aren't handled by mdadm (only recent Intel BIOS RAID devices are), and there are different codepaths there, so theoretically, software RAID and Intel BIOS RAID could be in different states - one working and the other broken. The Grand Plan would be to have separate test cases for each.
But the content of this specific test case isn't really the important thing here, it's just an example to give a feel for how the whole critpath test case setup will look. I threw it together in half an hour or so, it wasn't carefully worked out (or, well, tested :>)
Well, I have both setups on two different machines and mdadm works ok on both AFAICT.
Willing to test both!!
Regards, OldFart
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Awesome, I like how you've focused on writing the test case *once*, and distinguishing the content from how it's grouped/organized. That distinction will help us with a possible future TCMS migration.
We discussed on IRC, but following up here for a larger audience.... I wasn't sure if the 3 groups you listed (critpath_test_cases, mdadm_test_cases, mdadm_critpath_test_cases) were all needed. Certainly they help for convenience, but it sounds like the consumers of this hierarchy would only need to find 1. all mdadm test cases (''Category:mdadm_test_cases'') ... OR ... 2. All critical path mdadm test cases (intersection between ''Category:mdadm_test_cases'' and ''Category:Critpath_test_cases'').
I didn't initially think the intersection would be challenging to do. But thinking ahead to some remote API that consumers would use to find applicable tests, we may want to use the three categories as you suggest to avoid having each consumer defining logic for how it finds applicable tests. So in your example, we would treat ''Category:Critpath_test_cases'' as a way to denote test priority (where high == critpath, media, low). ''Category:mdadm_test_cases'' would be used to indicate membership to a test plan (the mdadm test plan). And finally, ''Category:Critpath_mdadm_test_cases'' would be our way to provide all *high* priority tests in the mdadm test plan. Does that make sense?
I know I'm getting too far ahead, but I'm trying to articulate our wiki organization in TCMS terms, and to reduce the requests consumers would have down to a simple API.
Using the 3 category suggestion, my ASCII-art attempt at a top-down view from ''Category:Test_cases'' would be ... {{{ Category:Test_Cases | +-> Category:mdadm_test_cases | | | +-> Category:Critpath_mdadm_test_cases (e.g. high priority tests) | | | | | +-> QA:Testcase_mdadm_important_use_case_1 | | +-> QA:Testcase_mdadm_important_use_case_2 | | +-> QA:Testcase_mdadm_important_use_case_3 | | | +-> QA:Testcase_mdadm_obscure_use_case_4 | +-> QA:Testcase_mdadm_obscure_use_case_5 | +-> QA:Testcase_mdadm_obscure_use_case_6 | +-> Category:kernel_test_cases | | | +-> Category:Critpath_kernel_test_cases (e.g. high priority tests) | | | +-> ... . . . }}}
And then the same view, but for just ''Category:Critpath_test_cases''. This is probably the easiest thing for consumers of this data to manage.
{{{ Category:Critpath_Test_Cases | +-> Category:Critpath_mdadm_test_cases (e.g. high priority tests) | | | +-> QA:Testcase_mdadm_important_use_case_1 | +-> QA:Testcase_mdadm_important_use_case_2 | +-> QA:Testcase_mdadm_important_use_case_3 +-> Category:Critpath_kernel_test_cases (e.g. high priority tests) | | | +-> QA:Testcase_kernel_important_use_case_1 | +-> QA:Testcase_kernel_important_use_case_2 | +-> ... . . . }}}
While it's an extra category for each src.rpm, this seems like a really sane approach.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
"I didn't initially think the intersection would be challenging to do. But thinking ahead to some remote API that consumers would use to find applicable tests, we may want to use the three categories as you suggest to avoid having each consumer defining logic for how it finds applicable tests. So in your example, we would treat Category:Critpath_test_cases as a way to denote test priority (where high == critpath, media, low). Category:mdadm_test_cases would be used to indicate membership to a test plan (the mdadm test plan). And finally, Category:Critpath_mdadm_test_cases would be our way to provide all *high* priority tests in the mdadm test plan. Does that make sense? "
Honestly, not really, no. =) The thinking behind creating the technically- not-needed combined category was 90% 'half-assing it as I was going along' and 10% for *non*-tool based cases: people just browsing the tests. It's easy via the API for tools to determine which test cases are members of both categories, but it's not easy for a user just browsing the Wiki to do it, AFAIK - there's no easy way to browse pages that are members of both one category and another category in the mediawiki user interface, that I know of.
But really, I'm not married to the idea. (That's why I like it, ba-dum tish. Tip your server.) I don't really see any problem with having tools derive the combination themselves - it's not like they'll all be implementing it in wildly different ways, I can't see how that would happen. I'm not sure manual tending of a combined category could produce results that would be different from doing it programmatically in any way other than the manual category occasionally getting screwed up, which *will* happen if we put squishy pink things in the loop.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:13 adamwill]:
Honestly, not really, no. =) The thinking behind creating the
technically-not-needed combined category was 90% 'half-assing it as I was going along' and 10% for *non*-tool based cases: people just browsing the tests. It's easy via the API for tools to determine which test cases are members of both categories, but it's not easy for a user just browsing the Wiki to do it, AFAIK - there's no easy way to browse pages that are members of both one category and another category in the mediawiki user interface, that I know of.
Correct, not without http://www.mediawiki.org/wiki/Extension:Multi- Category_Search
But really, I'm not married to the idea. (That's why I like it, ba-dum
tish. Tip your server.) I don't really see any problem with having tools derive the combination themselves - it's not like they'll all be implementing it in wildly different ways, I can't see how that would happen.
I can and have, I'd like to stay away from that. We're talking about a *small* set of tools at this point so the fallout would be minimal and manageable, agreed. My main point is, let's do our best to keep test cases and related test case metadata within the realm of TCMS (whether it's wiki or nitrate).
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
Fair enough. I guess the best approach would be to have it automatically generated 'server'-side - i.e. wherever we're storing the test cases, i.e., right now, the Wiki. I don't know if mediawiki can do that, but if we're looking at the Glorious Future where we have a TCMS, presumably the TCMS could do it?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
sigh, I keep getting ready to send this out to lists and then hitting one more snag. =)
so, in discussion with jlaska, we both think it makes most sense to only create two new types of category: one for associating test cases with specific packages, one for marking them as critical path test cases.
Critical path test cases should be in the category Critical_path_Test_Cases . Note that this means *any test case which is primarily concerned with verifying critical path test cases* should be in this category. It doesn't mean any test case associated with a critical path package should be in the category, because critical path packages can have non-critical path uses and hence non-critical path test cases: a test case to test, say, tape drive functionality in the kernel is associated with a critical path package (the kernel) but is not a critical path test.
The category name for specific source packages should be:
Package_(srcrpmname)_Test_Cases
same reasoning as my initial proposal for the test case name: we need to specify that (srcrpmname), in this context, is specifically a source RPM name. I can imagine, for instance, we might want to create a generic category for Python test packages, called Category:Python_Test_Cases, and put test cases into that category which are not actually test cases for the Python .src.rpm. So we'd also have a Category:Package_python_Test_Cases . That could probably be a member of Category:Python_Test_Cases , but not vice versa.
I think the page name of each test case can be a bit more flexible and we don't really need to worry about defining it, the categories are the crucial thing.
Do we also want to suggest a category for each .src.rpm along the lines of:
Category:Package_python_core_Test_Cases
? Rationale: I'm thinking about how we want the bodhi or fedora-easy-karma listing for each package to look. I'm thinking of something like:
-------------------------------------------------------
Update: python-2.7.9-blahblah
This update does blah blah foo foo whee whee poop.
The following test cases can be used to check the functionality of this package.
'''Critical path tests'''
These tests verify critical path functionality (link to critical path process page) and should take highest priority:
* links to all tests in Category:Package_python_Test_Cases *and* Category:Critical_path_Test_Cases
'''Core functionality tests'''
These tests verify core functionality of the package and should take highest priority after critical path tests:
* links to all tests in Category:Package_python_core_Test_Cases (but not Category:Critical_path_Test_Cases)
'''Remaining tests'''
These tests verify non-critical path, non-core functionality:
* links to all other tests (in Category:Package_python_Test_Cases but not Category:Package_python_core_Test_Cases or Category:Critical_path_Test_Cases)
------------------------------------------------------ ------------------------------------------------------
The thinking being that some packages will grow fairly large sets of tests (abrt already has, for instance) and it would help to isolate core functionality tests for rapid sanity checking.
random side observation: our naming of test case pages and categories currently stinks.
next big concern: where do we effectively document this? All we currently have on test cases is:
https://fedoraproject.org/wiki/Category:Test_Cases
I can add some overview and stuff to that page but it's not a terribly visible page, really. I guess the best approach is to edit that page and add some links to it from strategic other places, especially developer instruction pages. Thoughts?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
looking through existing test cases I found that abrt already has a good set of test cases we can use as seeds and examples:
https://fedoraproject.org/wiki/Category:ABRT_Test_Cases
we just need to adjust them to the planned system.
I also found an interesting illustration of the point about using the package_ descriptor - we have a set of 'bootchart test cases':
https://fedoraproject.org/wiki/Category:Bootchart_Test_Cases
but these aren't test cases for checking the functionality of the bootchart srpm; they're various boot scenarios that you time *using bootchart*. the idea isn't to test bootchart, but to use bootchart to test boot speed...
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:16 adamwill]:
The category name for specific source packages should be:
Package_(srcrpmname)_Test_Cases
That works fine. I think the "Package_" prefix is not needed, but it certainly doesn't hurt anything to have it (just longer wiki page names "Category:Package_mobile-broadband-provider-info_Test_Cases" vs "Category :Mobile-broadband-provider-info_Test_Cases", but that's certainly not horrible).
same reasoning as my initial proposal for the test case name: we need to
specify that (srcrpmname), in this context, is specifically a source RPM name. I can imagine, for instance, we might want to create a generic category for Python test packages, called Category:Python_Test_Cases, and put test cases into that category which are not actually test cases for the Python .src.rpm. So we'd also have a Category:Package_python_Test_Cases . That could probably be a member of Category:Python_Test_Cases , but not vice versa.
I think the page name of each test case can be a bit more flexible and
we don't really need to worry about defining it, the categories are the crucial thing.
Definitely!
Do we also want to suggest a category for each .src.rpm along the lines
of:
Category:Package_python_core_Test_Cases
? Rationale: I'm thinking about how we want the bodhi or fedora-easy-
karma listing for each package to look. I'm thinking of something like:
Update: python-2.7.9-blahblah
This update does blah blah foo foo whee whee poop.
The following test cases can be used to check the functionality of this
package.
'''Critical path tests'''
These tests verify critical path functionality (link to critical path
process page) and should take highest priority:
- links to all tests in Category:Package_python_Test_Cases *and*
Category:Critical_path_Test_Cases
'''Core functionality tests'''
These tests verify core functionality of the package and should take
highest priority after critical path tests:
- links to all tests in Category:Package_python_core_Test_Cases (but not
Category:Critical_path_Test_Cases)
'''Remaining tests'''
These tests verify non-critical path, non-core functionality:
- links to all other tests (in Category:Package_python_Test_Cases but
not Category:Package_python_core_Test_Cases or Category:Critical_path_Test_Cases)
The thinking being that some packages will grow fairly large sets of
tests (abrt already has, for instance) and it would help to isolate core functionality tests for rapid sanity checking.
The way I read your organization above is that the front-ends will present the tests by priority (critical path, core, *). I really like this visualization, but wonder if it is outside the scope for the first pass. Meaning, I think I'd want to see more focus on just getting critpath tests defined for as many critpath packages, rather than spending time on non- critpath or non-core stuff. With the setup you've designed, it doesn't exclude people from creating non-critpath tests.
An aside, I think the previously discussed Category: organization would lend better to allowing maintainers to prioritize tests themselves. For example ...
bodhi and f-e-k will look for wiki pages under Category:Package_%{sourcerpm}_Test_cases. Any tests in that category would be listed, any tests in sub-categories of that page will be presented as you demonstrate above. So to achieve your desired output, you might organize tests like ...
{{{ Category:Package_python_Test_Cases | +--> Category:Critpath_python_Test_Cases | | | +-> QA:Testcase_python_do_something_1 [1] | +-> QA:Testcase_python_do_something_2 [1] | +-> QA:Testcase_python_do_something_3 [1] | +--> Category:Core_python_Test_Cases | | | +-> QA:Testcase_python_do_something_4 | +-> QA:Testcase_python_do_something_5 | +--> Category:Low_python_Test_Cases | | | +-> QA:Testcase_python_do_something_6 | +-> QA:Testcase_python_do_something_7 | |--> QA:Testcase_python_do_something_8 |--> QA:Testcase_python_do_something_9
[1] Could also exist in Category:Critpath_test_cases, but not required }}}
This would allow maintainers to group the tests as they see fit, but holds that everything must be anchored out of Category:Package_python_Test_Cases. We would still focus on prioritizing development of critpath tests, but the support exists for maintainers/packagers to organize as they feel is appropriate. Also, the mediawiki API provides the needed methods to find and present the information listed above.
Another something to keep in mind, any heavy/complex wiki organization we do may require extra work when migrating data to another system (which is a possible mid-term future scenario).
random side observation: our naming of test case pages and categories
currently stinks.
Sure, page naming wasn't as important when we weren't attempting to integrate it with other tools so I gather some wiki renaming may be required. Or did it stink in another regard?
next big concern: where do we effectively document this? All we
currently have on test cases is:
I can add some overview and stuff to that page but it's not a terribly
visible page, really. I guess the best approach is to edit that page and add some links to it from strategic other places, especially developer instruction pages. Thoughts?
Agreed. Do we need an SOP/HOWTO for each of the use cases where proventesters, or maintainers, will interact with this setup? Such as ... 1. How to create a test case 2. How to organize test cases 3. How to post results for test cases 4. ... ?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
Thanks. I want to document the non-critpath usage of this system because I think it'll happen quite fast (in fact, we already have a set of abrt tests that should be converted to use the new system, and abrt isn't critpath...) and if we don't have the usage planned out ahead of time, people will do it however feels right to them and we'll wind up with a mess.
I think I've mostly got everything straight in my head now; I'm going to start drafting up changes to Wiki pages and linking to them here.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
here's a draft of a test case creation SOP:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_test_case_creation
it's not really anything specific to this ticket, but it is a prerequisite. I plan to write a (short) SOP on package-specific test cases, linking to this more generic SOP, and then edit the test case category page to link to both (and possibly the test case template page), and then edit strategic pages in the packaging instructions section to point out to these pages. sound good?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_... <--- draft SOP for package test plan creation.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
I tweaked https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_... somewhat and https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_test_case_creation a small amount. I think they're ready to go out for review now. I'll mail the lists for feedback.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
I converted over one package's test cases to serve as an example of the new system: the test cases for the Radeon driver, xorg-x11-drv-ati . This was a good example as there's a lot of them, and they split neatly into critpath, core and extended to illustrate the optional advanced categorization system.
So, see:
https://fedoraproject.org/wiki/Category:Package_xorg-x11-drv- ati_test_cases
and note that one is also in:
https://fedoraproject.org/wiki/Category:Critical_path_test_cases
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by fcami):
Thank you Adam and James! This is excellent work. Two comments: The extended test cases (at least, the 3D ones) should be run on Intel also, and maybe on Nouveau (do we support 3D there?), so we might need a way to share these across drivers. We also probably need to define a list of hardware we want the tests executed on, as a lot of the bugs we encounter are hardware-dependent, hitting different paths in the radeon/mesa stack. Any idea how to handle this?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
fcami: we have similar test cases for Intel and Nouveau already. I have not converted those to the new scheme because it's just a draft at present; I converted the Radeon test cases simply to serve as an example.
any elaboration of the Radeon test cases is really a separate topic from this process. The Radeon test cases themselves are not particularly significant to the package-specific test case process, they're merely examples. Please discuss things specific to the Radeon test cases elsewhere (file a new ticket, or mail the list). thanks!
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by fcami):
Sorry, I should have used more generic terms in my explanation. Let me try again :)
1. we should share test cases between all supported drivers and not duplicate them - because otherwise, editing a test case relevant to multiple hardware drivers means editing multiple pages.
2. how do we define success/failure (for a go/nogo meeting for instance) when some people manage to run the test cases and others don't, depending on hardware?
Note that 2. is probably more a "see what it's bugzilla" than a "see the test case results" question, so it's moot.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
again, neither of those is really relevant to this effort. I really want to keep this ticket focused on the project it's for: a way to organize test cases for specific packages. Please can we discuss your thoughts in another ticket or on the mailing list? Thanks.
1 does suggest one point I've thought of in relation to this process but not spelled out, I suppose - there's no reason we can't write test cases to apply to more than one package (in this case, multiple video drivers) and add them to the categories for multiple packages. That's certainly acceptable under the scheme, yes.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by bruno):
* cc: bruno (added)
Comment:
I have tried to create a test for squashfs-tools at: https://fedoraproject.org/wiki/QA:Testcase_squashfs-tools_compression
Does it look reasonable? Most of it is running a test script and making sure it doesn't print any "failed" messages.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:28 bruno]:
I have tried to create a test for squashfs-tools at:
https://fedoraproject.org/wiki/QA:Testcase_squashfs-tools_compression
Does it look reasonable? Most of it is running a test script and making
sure it doesn't print any "failed" messages.
Ooh, nothing speaks better than examples! :) Some comments...
1. Bruno, where is upstream for this test script? Does it live anywhere else? If not, I'd recommend uploading that test script to the wiki as a file. Either way, you might want to link to it for download, rather than including it inline. I've made a few changes along these lines, you just need to upload and link to the script now. 2. I suspect you've done so already, but a determination whether this is critpath for squashfs-tools, or core. This would impact which Category you use. 3. I'm not super familiar with squashfs-tools, so perhaps a bit more wording in the ''description'' on how this test fits into the big picture for squashfs-tools? I've re-organized some of your content along these lines, feel free to adjust as needed. 4. Perhaps include some examples of good test results, and/or an example of bad test results in the ''results'' section?
As noted above, I made a few adjustments to the test case. Of course, feel free to adjust/revert as you see fit.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by bruno):
There is no upstream. I needed to make one so that I would have a reasonable test before putting a development version of squashfs-tools in rawhide. (Kyle asked me not to break things :-) I'll take a look at doing the upload and if I think it is as simple as it looks at first glance I'll do it shortly.
This is mostly crit path. squashfs-tools is used in live images. If uncompressing doesn't work then the images won't work. Arguably only gzip needs to work now, but hopefully we'll also need xz for F15. Also the unsquashfs path isn't used by live images, so in theory that could be skipped as well. But it seemed convenient to test these together. If it runs too long, the test data size could be cut back and the test would probably be as good. In my tests it runs reasonably quick on an older machine.
In theory there could be other core tests to make sure extended attributes and special files work correctly. But those features aren't really used by live images currently.
A more complete test, would be to test live images, but there could be other things blocking a live image test and they would take longer to do.
I can add a good results example. The only bad results I saw were when I had bugs in my test script.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by bruno):
I fixed up the test case page for squashfs-tools.
One other note about this is that in the past I think testing of squashfs- tools by proventesters was slow because there wasn't a test case and people didn't know what to test. Eventually nirik ended up doing when I asked him about it. Hopefully this won't be as big of an issue in the future.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:31 bruno]:
I fixed up the test case page for squashfs-tools.
Nice, looks good to me!
One other note about this is that in the past I think testing of
squashfs-tools by proventesters was slow because there wasn't a test case and people didn't know what to test. Eventually nirik ended up doing when I asked him about it. Hopefully this won't be as big of an issue in the future.
That's one of the big reasons Adam is proposing the wiki structure discussed in this ticket, so that we now have a place for people to document repeatable test procedures, and a consistent way to find them. Yay, thank you Adam!
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
I love it when a plan comes together!
Thanks for the test case Bruno. It looks like you should add it to the category Category:Package_squashfs-tools_test_cases and add that category to Category:Test_Cases . Did you not find that the instructions made this clear enough? If not, I'll have to revise them...
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by bruno):
I think when I first read them I wasn't sure it was required. And since I just had the one test case, I wasn't sure if it was worthwhile. I might have also seen an early version of the instructions as they were being worked on at least somewhat after I did the test case. I'll take care of the category change.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by bruno):
I reviewed the SOP again and I think it should stress making a package category for each package more strongly. Also it should mention the trick used to get the package in the right alphabetical position instead of having everything end up under P.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
thanks for the feedback, I'll update the page soon. yes, you should create a category even for just one test case, because it is still useful to help both people and programs locate it. If there isn't a category, tools like f-e-k and bodhi will not have a way to figure out that the test case should be associated with the package.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
okay, updated - see https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_QA_SOP_packa... . changes look good?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by bruno):
That looks good. I found the real problem in my case, is that I did not realize that there were two documents and I was going mostly off the test case document, rather than the test plan document. I think I ran across both at various times, but when actually writing up the test case I was looking at the test case document. There is a link back to the test plan document, but I didn't check at that time.
I still like the changes to the test plan document. The sort trick I only knew about from the conversation about the radeon test cases. I'd be less likely to have made the mistake of not having a package category if I had been referring to both pages at the time I was actually making up the test case, but with the current wording I would have been sure to have made the package category.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
I posted a python script ([http://jlaska.fedorapeople.org/multi-cat- search.py multi-cat-search.py]) that can be used to demonstrate using the mediawiki API to find tests that apply for a particular source rpm name. See the --help for the command, it offers several options, including ''-c|--critpath'' to limit results to only critical path tests.
I'm happy to adjust the script as needed by f-e-k or bodhi. This was just a proof-of-concept to demonstrate sifting through wiki using the API and user-input.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
I think we can put a fork in the initial proof-of-concept of this ticket. Is there anything else you wanted to track here before closing?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
well, the ticket actually got sidetracked: if you read the initial post, the intention was to use it to track the creation of the test cases themselves, not the process. So we could still use it for that.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Changes (by athmane):
* cc: athmane (added)
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Replying to [comment:41 adamwill]:
well, the ticket actually got sidetracked: if you read the initial post,
the intention was to use it to track the creation of the test cases themselves, not the process.
How goes the progress? What's remaining?
So we could still use it for that.
It's too much for a single ticket, imo. But you choose ... I'm just cleaning up *stale* tickets.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Assuming I'm using my own script correctly ([http://fedorapeople.org/gitweb?p=jlaska/public_git/scripts.git;a=blob;f =get-mediawiki-data get-mediawiki-data]), we have the following packages with wiki test cases.
{{{ Category:Package abrt test cases Category:Package anaconda test cases Category:Package biosdevname test cases Category:Package clamav test cases Category:Package Django test cases Category:Package evince test cases Category:Package firefox test cases Category:Package gnome-color-manager test cases Category:Package gnome-disk-utility test cases Category:Package gammu test cases Category:Package gnome-shell test cases Category:Package httping test cases Category:Package imsettings test cases Category:Package ibus test cases Category:Package mysql test cases Category:Package mysql-libs test cases Category:Package mysql-server test cases Category:Package NetworkManager test cases Category:Package nmap test cases Category:Package nikto test cases Category:Package openvas-scanner test cases Category:Package openvas-libraries test cases Category:Package openvas-client test cases Category:Package preupgrade test cases Category:Package report test cases Category:Package rkhunter test cases Category:Package squashfs-tools test cases Category:Package smolt test cases Category:Package wireshark test cases Category:Package xorg-x11-drv-ati test cases Category:Package xorg-x11-drv-nouveau test cases Category:Package xorg-x11-drv-intel test cases }}}
Combining the list above, with a [http://kojipkgs.fedoraproject.org/mash/rawhide-20110610/logs/critpath.txt recent critpath log], the following critpath packages have wiki test cases:
{{{ anaconda NetworkManager report squashfs-tools xorg-x11-drv-ati xorg-x11-drv-nouveau xorg-x11-drv-intel }}}
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
Just created new test case for openssh (=> critpath):[[BR]]
https://fedoraproject.org/wiki/QA:Testcase_OpenSSH
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
I've categorized and tested two test-cases for cronie and perl (they are on critpath category) so they'll show up in Bodhi updates.
https://fedoraproject.org/wiki/QA:Testcase_Perl_sanity
http://fedoraproject.org/wiki/QA:Testcase_crontabs
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
I've categorized dracut tests cases:
https://fedoraproject.org/wiki/Category:Package_dracut_test_cases
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
Fixed <pre> tag issue on:
https://fedoraproject.org/wiki/QA:Testcase_Dracut_root%3Dpartition
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
I've just created a test-case for rsyslog (=> critpath):
https://fedoraproject.org/wiki/QA:Testcase_rsyslog
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
Athmane pointed out that the linked critpath.log shows binary packages, not src.rpms. I've attached a src.rpm list to this ticket for review. The list is taken form the 0610 rawhide critpath.log.
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by jlaska):
As requested during QA meeting, the following 13 critical path packages have wiki tests. There are a total of 432 src critical path packages. {{{ anaconda cronie dracut gnome-disk-utility NetworkManager perl report rsyslog squashfs-tools xorg-x11-drv-ati xorg-x11-drv-nouveau xorg-x11-drv-intel firstboot }}}
How do others feel about narrowing the scope of this ticket to a more attainable short-term goal? Perhaps all user-facing Desktop packages?
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
I've just created a test-case for yum (=> critpath):
https://fedoraproject.org/wiki/QA:Testcase_Yum_basics
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by athmane):
I've just created a test-case for sendmail (=> critpath):
https://fedoraproject.org/wiki/QA:Testcase_Sendmail
#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: closed Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: fixed | Keywords: --------------------------+------------------------------------------------- Changes (by adamwill):
* status: new => closed * resolution: => fixed
Comment:
Looking at this again, I've had a change of heart and I agree with Past James and not Past Adam: a single ticket for 'create test cases for 400+ critpath components' isn't much use, especially given trac doesn't let you have tickets depend on each other. So let's close it. That doesn't mean we should stop working on critpath test cases, though, this is just bureaucracy. :) Thanks athmane for your work so far on writing and categorizing test cases!