#154: Tracker: critical path test case creation --------------------------+------------------------------------------------- Reporter: adamwill | Owner: adamwill Type: enhancement | Status: new Priority: major | Milestone: Fedora 15 Component: Wiki | Version: Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by adamwill):
sigh, I keep getting ready to send this out to lists and then hitting one more snag. =)
so, in discussion with jlaska, we both think it makes most sense to only create two new types of category: one for associating test cases with specific packages, one for marking them as critical path test cases.
Critical path test cases should be in the category Critical_path_Test_Cases . Note that this means *any test case which is primarily concerned with verifying critical path test cases* should be in this category. It doesn't mean any test case associated with a critical path package should be in the category, because critical path packages can have non-critical path uses and hence non-critical path test cases: a test case to test, say, tape drive functionality in the kernel is associated with a critical path package (the kernel) but is not a critical path test.
The category name for specific source packages should be:
Package_(srcrpmname)_Test_Cases
same reasoning as my initial proposal for the test case name: we need to specify that (srcrpmname), in this context, is specifically a source RPM name. I can imagine, for instance, we might want to create a generic category for Python test packages, called Category:Python_Test_Cases, and put test cases into that category which are not actually test cases for the Python .src.rpm. So we'd also have a Category:Package_python_Test_Cases . That could probably be a member of Category:Python_Test_Cases , but not vice versa.
I think the page name of each test case can be a bit more flexible and we don't really need to worry about defining it, the categories are the crucial thing.
Do we also want to suggest a category for each .src.rpm along the lines of:
Category:Package_python_core_Test_Cases
? Rationale: I'm thinking about how we want the bodhi or fedora-easy-karma listing for each package to look. I'm thinking of something like:
-------------------------------------------------------
Update: python-2.7.9-blahblah
This update does blah blah foo foo whee whee poop.
The following test cases can be used to check the functionality of this package.
'''Critical path tests'''
These tests verify critical path functionality (link to critical path process page) and should take highest priority:
* links to all tests in Category:Package_python_Test_Cases *and* Category:Critical_path_Test_Cases
'''Core functionality tests'''
These tests verify core functionality of the package and should take highest priority after critical path tests:
* links to all tests in Category:Package_python_core_Test_Cases (but not Category:Critical_path_Test_Cases)
'''Remaining tests'''
These tests verify non-critical path, non-core functionality:
* links to all other tests (in Category:Package_python_Test_Cases but not Category:Package_python_core_Test_Cases or Category:Critical_path_Test_Cases)
------------------------------------------------------ ------------------------------------------------------
The thinking being that some packages will grow fairly large sets of tests (abrt already has, for instance) and it would help to isolate core functionality tests for rapid sanity checking.
random side observation: our naming of test case pages and categories currently stinks.
next big concern: where do we effectively document this? All we currently have on test cases is:
https://fedoraproject.org/wiki/Category:Test_Cases
I can add some overview and stuff to that page but it's not a terribly visible page, really. I guess the best approach is to edit that page and add some links to it from strategic other places, especially developer instruction pages. Thoughts?