Hi - we (David Kutalek, Izidor Matusov & myself) thought of creating ABRT test suite and broaden and formalize overall testing efforts. We came out with mind map (both source XML and bitmap attached) describing what we have and what we'd like to have in the future. Basic idea we believe in is that testing in some specific field should fit the underlying environment. So giving an example: Testing in Fedora should use Fedora's own infrastructure, that means e.g. running internal test suite as part of Koji build and AutoQA tests on Fedora-specific regressions and sanity matters. Let me describe the mind map in detail, branch by branch: * Fedora + Fedora Test Day - Global public testing by Fedora community, should focus on both Features (e.g. Retrace Server), infrastructure compatibility (e.g. reporting to Bugzilla) and desktop integration. + Fedora AutoQA Tests - By use of Fedora AutoQA infrastructure we should test only sanity integration and regressions specific to Fedora only (e.g. bugzilla plugin worked with mozilla.org Bugzilla but not with the Red Hat instance). + Release Criteria - Update to Fedora stable release will be pushed only iff FedoraQA tests passed & upstream test suite passed in Koji. * RHEL - Not intended to be discussed publicly. * Internal Test Suite + Unit Tests - Some folks generally think that writing unit tests *after* the actual code is written is more or less useless because the test may match broken code. Although I understand possible issues it's far superb to what we have now. Also considering ABRT being still more or less rewritten to enhance it's design, it's quite a good chance we do it the right way. We'd like to see unit tests in framework of your choice for both core and plugins. + Regression Tests - Having wide range of regression tests in sources needs some time, it may be a pain and it may make you feel like you are coding tests and not developing the useful code instead. However, in my opinion, it saves you the time you have to invest in fixing regressions. Also with regression tests you may slightly less fear breakage and anger from users. You may also view it as a step further in providing more stable code; for example code review was a positive step in that direction. + BuildBot - BuildBot builds ABRT when new commits we the other day, runs tests, results sends to the mailing list and perhaps even copies resulting RPMs to fedorapeople repo for testers consumption. * Infrastructure - Just that ABRT should be installed by default in Fedora and enable in the GUI. Similar tasks for RHEL and RH infrastructure. More on the implementation details: For Fedora, AutoQA should be the choice. Upstream unit tests - not sure, up to you. Upstream regression and sanity tests - we'd prefer Beaker/beakerlib, since we are really familiar with it and is easy to play with. Let me know, what you think guys about the mind map, your frameworks of choice, anything. Thanks, Michal