test errors with gcc 4.6.1
by Denys Duchier
Here are the diffs I used on Gentoo to test with gcc 4.6.1:
and here are the messages from the tests:
Cheers,
--Denys
12 years, 9 months
ANN: gcc-python-plugin 0.5
by David Malcolm
gcc-python-plugin is a plugin for GCC 4.6 onwards which embeds the
CPython interpreter within GCC, allowing you to write new compiler
warnings in Python, generate code visualizations, etc.
Tarball releases are available at:
https://fedorahosted.org/releases/g/c/gcc-python-plugin/
Prebuilt-documentation can be seen at:
http://readthedocs.org/docs/gcc-python-plugin/en/latest/index.html
Project homepage:
https://fedorahosted.org/gcc-python-plugin/
High level summary of the changes since the initial announcement:
- new contributors
- lots of bug fixes and compatibility fixes (e.g. for Python 3): the
selftest suite now works for me with all eight different
combinations of:
- optimized vs debug builds of Python
- Python 2.7 vs Python 3.2
- i686 and x86_64
building against gcc-4.6.1 (also tested with gcc-4.6.0)
- new example scripts; see:
http://readthedocs.org/docs/gcc-python-plugin/en/latest/examples.html
- if PLUGIN_PYTHONPATH is defined at build time, hardcode the value
into the plugin's sys.path, allowing multiple builds to be
independently packaged
- more documentation
- work-in-progress on detecting reference-count errors in C Python
extension code. Although this usage example can now detect errors,
it isn't yet ready for general use. It can generate HTML
visualizations of those errors; see
http://dmalcolm.livejournal.com/6560.html
for examples.
- numerous other improvements (see below)
- new dependency: the "six" module is required at both build time and
run-time, to smooth over Python 2 vs Python 3 differences:
http://pypi.python.org/pypi/six/
I've also packaged the plugin in RPM form for Fedora 16 onwards; see:
https://fedoraproject.org/wiki/Features/GccPythonPlugin
Enjoy!
Dave
Detailed change notes follow
Version 0.5
===========
David Malcolm (7):
Override all locale information with LC_ALL=C when running
selftests
Revamp support for options in selftests
Add note about ccache
Improvements to the example scripts
Split up examples within the docs
Fix gcc.Pass.__repr__
Version 0.4
===========
David Malcolm (10):
add explicit BR on gmp-devel (rhbz#725569)
Make the test suites be locale-independent
Suppress buffering of output in Python 3
run-test-suite.py: support excluding tests from a run
Python 3 fixes to testcpychecker.py
Add 'str_no_uid' attribute to gcc.Tree and gcc.Gimple; use it to
fix a selftest
Fix segfault seen on i686 due to erroneous implementation of
'pointer' attribute of gcc.TypeDecl
Selftest fixes and exclusions for 32-bit builds
Fix the test for 32/64-bit in selftests so that it works with both
Python 2 and 3
Version 0.3
===========
David Malcolm (3):
If PLUGIN_PYTHONPATH is defined at build time, hardcode it into
the plugin's sys.path
Python 3 fixes
Add the beginnings of a manpage for gcc-with-python
Version 0.2
===========
Alexandre Lissy (2):
fix: permerror() misusage
Using Freedesktop standard for image viewing
David Malcolm (98):
Introduce gcc.Parameter and gcc.get_parameters()
Introduce a compatibility header file
Document the 'basic_blocks' attribute of gcc.Cfg
Fix a mismatch between gccutils.pformat() and the API docs
Add note about debugging
Move the debugging information to be more prominent, and reword
Automatically supply the correct header search directory for
selftests that #include <Python.h>
Set up various things in sys, including sys.path
Cope with calls to function pointers in the arg checker
Remove stray import
Fix issue with PyArg_ParseTuple("K") seen compiling gdb
Fix erroneous error messages for the various "s" and "z" format
codes
Format codes "U" and "S" can support several different argument
types
Add a way of turning of const-correctness for "const char*"
checking
Fix breakage of the various "es" and "et" format codes introduced
in last commit
Implement verification of the "O&" format code (converter
callback, followed by appropriate arg)
Use newlines and indentation to try to make the PyArg_ error
messages more readable
Add the example from my blog post
( http://dmalcolm.livejournal.com/6364.html )
Remove redundant (and non-functioning) selftest for "O&" format
code
Add 'local_decls', 'start', 'end', 'funcdef_no' to gcc.Function
Add 'arguments' and 'result' to gcc.FunctionDecl
Fix typos in docs
Add 'operand' to gcc.Unary; check that the keywords table to
PyArg_ has a NULL terminator
Add 'location' to more tcc types; fill out more documentation
Start building out examples of C syntax vs how it's seen by the
Python API
Fix the behavior of the various "#" format codes.
Add Alexandre Lissy to contributors
The various "e" codes can accept NULL as the encoding
Add support for PyNumberMethods; use this to support int() on
gcc.IntegerCst
Introduce gcc.Option, wrapping GCC's command-line options
Add gcc.warning() and gcc.error()
Implement gcc.Option.is_enabled; allow -Wno-format to work, adding
related workaround to gcc.warning()
Explicitly set -Wno-format in the selftest that requires it
If pygments is available, use it to colorize the source code in
graphviz visualizations. Add line numbers.
Add (disabled) option to generate CFGs as .svg rather than .png
Minor tweaks to dot generation
Support edge connections to individual statements in CFG
renderings (via graphviz 'port=')
Add an optional name to CFG renderings, along with other tweaks
Work-in-progress on refcount checker
Disable the checker by default again (accidentally enabled in last
commit)
Disable non-functioning tests for C stdio checker
Remove the now-redundant SSA support from location-tracking
Move generic interp code from refcounts.py to absinterp.py
Eliminate superfluous StateEdge class, adding 'src' field to
Transition
Add test coverage for gcc.inform(); clarify docs (the output is to
stderr)
CFG renderer: cope with pygments styles that have a "bold" prefix
Implement gcc.Tree.__hash__
Add a high-level overview of the insides of GCC
Add "operand" field to gcc.MemRef; document gcc.MemRef, along with
gcc.FieldDecl, gcc.ComponentReference
Add a Table class to gccutils, to make it easier to generate
tabular data
Make gccutils.Table's output be a valid .rst table
Expose more of the pass machinery, and use this to include the
full pass tree within the docs
Fix broken image link for cfg docs
Add more information on the different representations
Finish the table of gcc.PROP_ flags
Add gcc.get_gcc_version() and gcc.get_plugin_gcc_version()
Add a note about the GCC version to the pass documentation
Add an autogenerated SVG diagram of GCC's passes
Syntax-highlight Gimple within CFG dumps
Rewrite of state tracking within libcpychecker (work-in-progress)
Eliminate the DataState class, merging it all into State
Report erroneous refcounts returned from functions, and errors
Highlight new and changing values in State Graph visualizations
Fix lookup of ob_refcnt on PyObject* for instances that were never
modified
Suggest the use of Py_RETURN_NONE when detecting
_Py_NoneStruct.ob_refcnt is too low
Detect reference leaks for values that aren't the return value
Ensure that the checker can cope with calls to functions it
doesn't know about
Fix the folding of gcc.GimpleCond where we can know the value of
the boolean
Remove dead code
Set ob_size in PyList_New
Set up ob_type on objects within the analyser
Add "dump_traces" option to check_refcounts, to make it easier to
write selftests
Avoid hardware dependent results in the tests/plugin/parameters
selftest
Improvements to error reporting
Add support for stolen references
Report on where objects were allocated when reporting on an
ob_refcnt problem
Generate HTML reports on errors (in addition to text on stderr)
Try to make the plugin compile against gcc 4.5
libcpychecker cleanups
Warn when a function returns NULL without setting the per-thread
exception state
Beginning implementing Py_DECREF
Warn about code that accesses deallocated memory
Warn about functions that return pointers to deallocated memory
Don't analyse traces with loops, as a primitive means of ensuring
that the analysis terminates
Improve error handling within the checker
Rigidly distinguish between l-values and r-values in cpychecker
Add 'array' and 'index' fields to gcc.ArrayRef
minor docs tweak
Support arrays, and more pointer handling
Fix the build (and many selftests) against Python 3; tweak
exception handling
Better support for building against debug builds of Python
Comment out the "Other callback events" section for now
Add explicit requirements section to the documentation
Clean the docs during a 'make clean'
Minor tweak to Makefile
Tom Tromey (2):
make html from top-level
expose 'inform' function
12 years, 9 months
New dependency: "six"
by David Malcolm
I've been fixing python 3 support within the plugin, so that it's
buildable and usable with both Python 2 and Python 3. [1]
The plugin now builds against Python 3, and most of the selftests now
pass (I'm testing against 3.2, fwiw).
(It still builds against Python 2, and all of the selftests pass, on my
machine, at least, with 2.7).
My aim is to support both Python 2 and Python 3 from a single source
tree, without needing to run 2to3 on the .py sources. (The plugin will
need to be built twice, once for each python version, and only one build
of the plugin can be loaded into gcc at once).
To make this easier, I'm using the compatibility module named "six":
http://pypi.python.org/pypi/six/
This is now needed at both buildtime, and at runtime.
I'm assuming this isn't too onerous a dependency (it's just one .py
file; fwiw, I've packaged it for Fedora)
Dave
[1]
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=7...
12 years, 9 months
compilation problems with gcc 4.5.2
by Denys Duchier
I cloned the git repo today and tried to compile the plugin.
Apparently, includes have changed a bit: by adding in <coretypes.h> and
<toplev.h> I went a little further, but now I am stuck on an error about
all_translation_units (indeed, it does seem to be undeclared). any
idea?
Cheers,
--Denys
12 years, 9 months
gcc-python-plugin status update: static analysis, and HTML reports
by David Malcolm
I've written up a status update on the gcc-python-plugin on my blog
here:
http://dmalcolm.livejournal.com/6560.html
Summarizing that blog post, I've revamped the internals of how my
checker script so that it can detect various kinds of CPython reference
count bug, and it can now render bug reports as both regular text on
stderr, and in HTML form:
http://fedorapeople.org/~dmalcolm/blog/2011-07-15/refcount_demo-refcount-...
(you may need a modern browser to see the control flow annotations in
the HTML, as it uses JavaScript and the <canvas> element; there are
screenshots in the blog post)
Detecting the all-too-common: "return Py_None;" bug:
http://fedorapeople.org/~dmalcolm/blog/2011-07-15/losing_refcnt_of_none-r...
Detecting a (contrived) reference leak:
http://fedorapeople.org/~dmalcolm/blog/2011-07-15/object_leak-refcount-er...
Detecting a stray Py_INCREF that makes the reference count too high, or
segfaults python, depending on what happened earlier within the
function:
http://fedorapeople.org/~dmalcolm/blog/2011-07-15/too_many_increfs-refcou...
Some of that rendering code might be of use to other users of the
plugin; has anyone else rendered HTML reports from within GCC?
I didn't discuss implementation details in the blog post, but given the
audience on these lists, it's probably on-topic to mention here:
It works by a rather crude implementation of abstract interpretation,
tracking all possible paths through the function, generating traces of
execution. (It currently will fail horribly when it encounters a loop;
I suspect I need widening/narrowing operators, and to read a lot more
papers on the subject :) )
Internally, the analyser models the memory state using a method similar
to the one used by LLVM's clang static analyser, as described in:
"A Memory Model for Static Analysis of C Programs" (Zhongxing Xu, Ted
Kremenek, and Jian Zhang)
thus handling pointers and dereferences (though my implementation is
very much an early prototype as this stage).
I've added a new (as far as I know) class of abstract value representing
"all reference counts that the function owns" (i.e. an integer value,
relative to the unknown refcount owned by all of the rest of the
program).
Hope this is helpful/interesting
Dave
12 years, 9 months
Fixed a hardware dependency in the selftests
by David Malcolm
When I implemented gcc.Parameter, out of all the parameters I could have
chosen for the selftest [1] I managed to pick the two parameters that
vary based on the hardware on the test machine, thus embedding an
expected value that only worked with a specific certain amount of RAM.
Specifically, 'ggc-min-expand' and 'ggc-min-heapsize' are set up by
gcc's init_ggc_heuristics() to values based on the amount of physical
memory on the test machine.
So the selftest suite has been working fine for me, but probably failing
for anyone else.
I've pushed a fix for this, so hopefully the test suite is no longer
hardware dependent, and "make" should now run to completion.
Sorry about this. Please let me know if you see any other test failures
running "make".
Dave
[1] "tests/plugin/parameters"
12 years, 9 months
A visualization of GCC's passes, as a subway map
by David Malcolm
For fun over the weekend I wrote a python script (using my
gcc-python-plugin[1]) to render an SVG diagram of GCC's optimization
passes (or, at least, based on my understanding of them).
This diagram shows the various GCC optimization passes, arranged
vertically, showing child passes via indentation.
The lifetime of the various properties that they maintain is shown,
giving the pass that initially creates the data (if any), the pass that
destroys it (if any), and each pass that requires a particular property
(based on the PROP_* flags). I've attempted to make it look a little
like a subway train schematic map, where the properties are train lines,
and the passes are the stations they stop at; the initial/final passes
are the termini.
The resulting SVG can be seen at the top of:
http://readthedocs.org/docs/gcc-python-plugin/en/latest/tables-of-passes....
Direct link to SVG, in case the above gets mangled on your browser:
http://readthedocs.org/docs/gcc-python-plugin/en/latest/_images/passes.svg
(550 x 3302 pixels; about 250k in size).
The script can be seen here (it uses pycairo to abstract the drawing
operations):
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=generat...
Hope this is fun/helpful (and that I'm correctly interpreting the data!)
Dave
[1] https://fedorahosted.org/gcc-python-plugin/
12 years, 9 months
