Hello,
I've implemented a proof-of-concept of an analysis that tries to pair static analysis results with known crashes based on the source code locations, as outlined in [1].
The code extends David Malcolm's mock-with-analysis and is available at [2]. The machinery for generating static analysis results is unchanged apart from a few fixes needed for it to run on Fedora 19. The script make-simple-report.py was extended to accept second argument with crash reports from FAF server, and the matching crashes are referenced in the generated reports. There is currently no way to obtain the file with crashes automatically, I got it from the server administrator.
I ran the analysis on following packages: * tracker-0.16.2-1.fc19 * evolution-3.6.4-3.fc18 * gnome-shell-3.6.3.1-1.fc18 * nautilus-3.6.3-4.fc18 * python-2.7.3-13.fc18 * rhythmbox-2.98-4.fc18
Tracker was chosen arbitrarily, the rest of the builds are those that have the highest number of distinct crashes. The results can be viewed at [3] and given that they were obtained from packages with the highest number of collected crashes, they don't seem to be very encouraging. There are only three [4,5,6] matches that are not obvious false positives. All the data needed to reproduce this should be available at [7].
There are two main causes of false positives: * The code considers all static analysis results, not only those from tests for behaviour that would result in a crash at runtime. * It considers all stack frames in a crash, not just the topmost one.
As a side note, all three matches come from the clang static analyzer, which for some reason fails for quite a lot of source files.
What do you think?
Thanks, Martin
[1] https://lists.fedoraproject.org/pipermail/firehose-devel/2013-October/000065... [2] https://github.com/mmilata/mock-with-analysis/tree/crash-correlation [3] http://mmilata.fedorapeople.org/firehose-crash-correlation/ [4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/... [5] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/... [6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71... [7] http://mmilata.fedorapeople.org/firehose-crash-correlation.tar.xz
I uploaded the clang-analyzer-generated html reports for the three "interesting" cases that the script found and took a further look at them.
* nautilus 1 [1], clang-analyzer report [2]
The trace from the static analyzer consists of nautilus_file_operations_copy_move calling nautilus_file_operations_move which then segfaults. This agrees with the backtraces. Unfortunately there is no BZ ticket associated probably due to too few people affected by this bug
* nautilus 2 [3], clang-analyzer report [4]
Only nautilus_file_operations_copy_move is in the static analyzer trace. There's bugzilla ticket [5] with full backtrace corresponding to this problem.
* python [6], clang-analyzer report [7]
The trace consists of PyObject_Unicode calling PyObject_GetAttr, which is not the case of the linked backtrace, making this pair a false positive. The trace from clang-analyzer describes a real bug though, one that has been already fixed [8][9].
Didn't know clang-analyzer can do inter-procedural analysis, that's nice.
[1] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/... [2] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-bui...
[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/... [4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-bui... [5] https://bugzilla.redhat.com/show_bug.cgi?id=860109
[6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71... [7] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/scan-build... [8] http://bugs.python.org/issue16839 [9] http://hg.python.org/cpython/rev/0012d4f0ca59
On Tue, Oct 22, 2013 at 12:26:06PM +0200, Martin Milata wrote:
I uploaded the clang-analyzer-generated html reports for the three "interesting" cases that the script found and took a further look at them.
- nautilus 1 [1], clang-analyzer report [2]
The trace from the static analyzer consists of nautilus_file_operations_copy_move calling nautilus_file_operations_move which then segfaults. This agrees with the backtraces. Unfortunately there is no BZ ticket associated probably due to too few people affected by this bug
- nautilus 2 [3], clang-analyzer report [4]
Only nautilus_file_operations_copy_move is in the static analyzer trace. There's bugzilla ticket [5] with full backtrace corresponding to this problem.
- python [6], clang-analyzer report [7]
The trace consists of PyObject_Unicode calling PyObject_GetAttr, which is not the case of the linked backtrace, making this pair a false positive. The trace from clang-analyzer describes a real bug though, one that has been already fixed [8][9].
Didn't know clang-analyzer can do inter-procedural analysis, that's nice.
Thrilling stuff, nice work!
I'll soon have a corpus of checks being run against Debian packages, I'll be sure to forward you data points (if y'all have the same source/version pair in Fedoraland)
Keep up the great work, Paul
On Tue, Oct 22, 2013 at 11:27:45 -0400, Paul Tagliamonte wrote:
Thrilling stuff, nice work!
I'll soon have a corpus of checks being run against Debian packages, I'll be sure to forward you data points (if y'all have the same source/version pair in Fedoraland)
Thanks! It would be interesting to run the analysis on those, though the possible differences between Debian and Fedora sources could pose a problem. E.g. a patch in one package and not the other might cause the line numbers to disagree. I have no idea how often this is the case.
Martin
On Tue, 2013-10-29 at 16:05 +0100, Martin Milata wrote:
On Tue, Oct 22, 2013 at 11:27:45 -0400, Paul Tagliamonte wrote:
Thrilling stuff, nice work!
I'll soon have a corpus of checks being run against Debian packages, I'll be sure to forward you data points (if y'all have the same source/version pair in Fedoraland)
Thanks! It would be interesting to run the analysis on those, though the possible differences between Debian and Fedora sources could pose a problem. E.g. a patch in one package and not the other might cause the line numbers to disagree. I have no idea how often this is the case.
That in itself might be something we could track using firehose, perhaps? i.e. have an <info> element that says that the code is patched downstream by a particular distribution. Then the UI can render those elements (though which version of the source would you render in such a situation), and one can run a query showing patches across multiple distros and packages.
(Not sure if this is a good idea, but I thought I'd share it)
Dave
On Tue, Oct 29, 2013 at 11:10:32AM -0400, David Malcolm wrote:
That in itself might be something we could track using firehose, perhaps? i.e. have an <info> element that says that the code is patched downstream by a particular distribution. Then the UI can render those elements (though which version of the source would you render in such a situation), and one can run a query showing patches across multiple distros and packages.
Uhm, so the broader need here is the ability to correlate different distro-specific versions with one another or, in fact, to the respective upstream version. We can do that via external databases, but it would add a pretty heavy infrastructure dependency.
IMO it would be better to pursue one of the following two solutions (or even both):
- add a new sub-element to <sut> which mentions the *upstream* version; once we have that we can correlate reports from different distros via the upstream version (if there are significant differences, that should come from the distro-specific patching)
- add a new <context> or <excerpt> sub-element to failure/info/etc. that can be used to add snippets of code around the location the static analysis tool is pointing at. The idea of this is the same of contexts for textual diffs: by comparing them we will be able to understand if we're talking about the same code or, due to patched, significantly different parts of it.
Of course it's an approximated solution, as the failure might descend from patches far far away in the code base, but if it works for diff, I think it'd be good enough for us as well. (And if we also have the upstream version, we can always lookup the distro-specific patches by external means and compare those.)
Just my 0.02 EUR, Cheers.
crash-catcher@lists.fedorahosted.org
-
David Malcolm -
Martin Milata -
Paul Tagliamonte -
Stefano Zacchiroli