[PATCH] pretty logs

This patch involves tickets: https://fedorahosted.org/autoqa/ticket/315 - Create per-item logs for multi-item tests https://fedorahosted.org/autoqa/ticket/316 - depcheck: extract possible cause of failure https://fedorahosted.org/autoqa/ticket/318 - Provide access to test documentation https://fedorahosted.org/autoqa/ticket/298 - test.py - split postprocess_iteration reporting into standalone methods https://fedorahosted.org/autoqa/ticket/319 - Create HTML log output if possible and it is the result of my, jskladan's and vhumpa's work. I'll the describe the patch in detail here, but I decided not to attach it nor to post it into reviewboard, because it is simply too large. Also I expect that it will receive a little more polish in the following days, so it's not absolutely finalized yet. But if you see anything you would like to change, please, send your comments. You can display the patch by command: $ git fetch $ git diff origin/master..origin/pretty -M Most important changes: 1. TestDetail class All the test's outcomes are now stored in a single class TestDetail. It works as a simple container for test result, summary, output, highlights and a few additional information. It is easier now to call methods that operate on these information, you just pass them your TestDetail instance. By default every test has a default "main" TestDetail instance created in self.detail. You can access self.detail to change the values of result, summary, etc. For a simple single-item tests (like rpmlint), this main TestDetail instance is all they need. For complex multi-item tests (like upgradepath) you need to create a TestDetail instance for every "report" you want to create. In the case of upgradepath and depcheck this means creating separate TestDetail for every Bodhi update tested. 2. TestDetail.update_result() method TestDetail class remembers list of result keywords sorted by importance. If you use TestDetail.update_result(result) method, it will set the test result only if your provided test result is more important (i.e. "worse") then the current test result. This way you can easily handle all error states inside your script and only as the last line include: self.detail.update_result('PASSED') 3. self.log() method We used to do a pretty complicated tasks when printing and saving some output. E.g. you would print in out on the stderr, and then you would append it to self.outputs and maybe even self.highlights. Now you just use: self.log(message, stderr=True, highlight=True) For multi-item tests (like upgradepath) you can also use self.log() to print to several TestDetail instances at once. 4. automatic log creation, different terminology We now have several types of logs (debug, output.log and pretty log). We want to have them transparent and simple for test authors. Debug log is just for us, the developers, so we don't mention it. The former output.log was renamed to full.log and we mention it as little as possible, preferably no at all. And the pretty log is referenced simply as "log" or "test log" and that's the only one we hope test authors will work with. Full log (full.log) is created automatically at the end of every test run. It contains everything that has been logged via self.log() method (provided that @param printout==True). It does not contain output of "print foo" messages (that's just in the debug log). Test log (aka pretty log) is also created automatically at the end of every test run and it's populated from main TestDetail instance (self.detail). Except for the case when the test author has created some test log already manually, then we don't create another one automatically. Test log is created in HTML and this file is used for reporting results everywhere (it is either sent or linked). 5. self.post_results() method There is now a single method for reporting results. It takes care of everything - creating log file, sending email, sending opt-in email and sending Bodhi comment. The first two are mandatory and the last two are optional - you have to provide correct parameters. By default the results are posted automatically at the end of the test run. It means for simple tests where you don't need anything fancy (like Bodhi comments or opt-in emails), you don't have to care about reporting results, everything is handled for you. For more advanced uses you have to call the self.post_results() method. For multi-items tests like upgradepath or depcheck you'll need to call self.post_results(test_detail) for every "item" (Bodhi update in our case) you want to report results for. 6. Test logs are in HTML Test logs are now in HTML. It will allow us to do pretty formatting, nice highlighting, etc. When sending as an email, only a plaintext overview header and a link to proper HTML log is included in the email body. Example: http://kparal.fedorapeople.org/autoqa/upgradepath2.html 7. All tests modified All tests were re-written to the new architecture. Most of them don't take any advantage of the new log features yet. rpmlint example: http://kparal.fedorapeople.org/autoqa/rpmlint.html Upgradepath and depcheck received more love and they should creater nice per-update logs with just the relevant info included. Depcheck now tries to filter out irrelevant messages and keep only the interesting ones in the log. upgradepath example: http://kparal.fedorapeople.org/autoqa/upgradepath.html depcheck example: http://kparal.fedorapeople.org/autoqa/depcheck.html We were working on it hard even today. Some code may be still rough. All comments welcome. Thanks, Kamil