pairing static analysis results with known crashes

Hi devel, it was suggested to me to repost here two messages originally sent to firehose-devel list. Hopefully someone might find them interesting. Martin ----- Forwarded message from Martin Milata <mmilata(a)redhat.com&gt; ----- Date: Fri, 18 Oct 2013 15:14:57 +0200 To: firehose-devel(a)lists.fedoraproject.org Subject: correlating static analysis results with known crashes Hello, I've implemented a proof-of-concept of an analysis that tries to pair static analysis results with known crashes based on the source code locations, as outlined in [1]. The code extends David Malcolm's mock-with-analysis and is available at [2]. The machinery for generating static analysis results is unchanged apart from a few fixes needed for it to run on Fedora 19. The script make-simple-report.py was extended to accept second argument with crash reports from FAF server, and the matching crashes are referenced in the generated reports. There is currently no way to obtain the file with crashes automatically, I got it from the server administrator. I ran the analysis on following packages: * tracker-0.16.2-1.fc19 * evolution-3.6.4-3.fc18 * gnome-shell-3.6.3.1-1.fc18 * nautilus-3.6.3-4.fc18 * python-2.7.3-13.fc18 * rhythmbox-2.98-4.fc18 Tracker was chosen arbitrarily, the rest of the builds are those that have the highest number of distinct crashes. The results can be viewed at [3] and given that they were obtained from packages with the highest number of collected crashes, they don't seem to be very encouraging. There are only three [4,5,6] matches that are not obvious false positives. All the data needed to reproduce this should be available at [7]. There are two main causes of false positives: * The code considers all static analysis results, not only those from tests for behaviour that would result in a crash at runtime. * It considers all stack frames in a crash, not just the topmost one. As a side note, all three matches come from the clang static analyzer, which for some reason fails for quite a lot of source files. What do you think? Thanks, Martin [1] https://lists.fedoraproject.org/pipermail/firehose-devel/2013-October/000... [2] https://github.com/mmilata/mock-with-analysis/tree/crash-correlation [3] http://mmilata.fedorapeople.org/firehose-crash-correlation/ [4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc... [5] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc... [6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources... [7] http://mmilata.fedorapeople.org/firehose-crash-correlation.tar.xz ----- End forwarded message ----- ----- Forwarded message from Martin Milata <mmilata(a)redhat.com&gt; ----- Date: Tue, 22 Oct 2013 12:26:06 +0200 To: firehose-devel(a)lists.fedoraproject.org Subject: Re: correlating static analysis results with known crashes I uploaded the clang-analyzer-generated html reports for the three "interesting" cases that the script found and took a further look at them. * nautilus 1 [1], clang-analyzer report [2] The trace from the static analyzer consists of nautilus_file_operations_copy_move calling nautilus_file_operations_move which then segfaults. This agrees with the backtraces. Unfortunately there is no BZ ticket associated probably due to too few people affected by this bug * nautilus 2 [3], clang-analyzer report [4] Only nautilus_file_operations_copy_move is in the static analyzer trace. There's bugzilla ticket [5] with full backtrace corresponding to this problem. * python [6], clang-analyzer report [7] The trace consists of PyObject_Unicode calling PyObject_GetAttr, which is not the case of the linked backtrace, making this pair a false positive. The trace from clang-analyzer describes a real bug though, one that has been already fixed [8][9]. Didn't know clang-analyzer can do inter-procedural analysis, that's nice. [1] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc... [2] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-... [3] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc... [4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-... [5] https://bugzilla.redhat.com/show_bug.cgi?id=860109 [6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources... [7] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/scan-bu... [8] http://bugs.python.org/issue16839 [9] http://hg.python.org/cpython/rev/0012d4f0ca59 ----- End forwarded message -----