20.2. Building and Using a Debug Version of Python

There is a spectrum of debug builds of Python that you can create. This chapter describes how to create them.

20.2.1. Building a Standard Debug Version of Python

Download check and and unpack the Python source into the directory of your choice:

 1$ curl -o Python-3.13.2.tgz https://www.python.org/ftp/python/3.13.2/Python-3.13.2.tgz
 2# Get the Gzipped source tarball md5 from:
 3# https://www.python.org/downloads/release/python-3132/ "6192ce4725d9c9fc0e8a1cd38410b417"
 4$ md5 Python-3.13.2.tgz | grep 6192ce4725d9c9fc0e8a1cd38410b417
 5MD5 (Python-3.13.2.tgz) = 6192ce4725d9c9fc0e8a1cd38410b417
 6# No output would be a md5 missmatch.
 7$ tmp echo $?
 80
 9# 1 would be a md5 missmatch.
10$ tar -xzf Python-3.13.2.tgz
11$ cd Python-3.13.2

Then in the source directory create a debug directory for the debug build:

$ mkdir debug
$ cd debug
$ ../configure --with-pydebug
$ make
$ make test

20.2.2. Specifying Macros

They can be specified at the configure stage, this works:

$ ../configure CFLAGS='-DPy_DEBUG -DPy_TRACE_REFS' --with-pydebug
$ make

However the python documentation suggests the alternative way of specifying them when invoking make:

$ ../configure --with-pydebug
$ make EXTRA_CFLAGS="-DPy_REF_DEBUG"

I don’t know why one way would be regarded as better than the other.

20.2.3. The Debug Builds

The builds are controlled by the following macros. The third column shows if CPython C extensions have to be rebuilt against that version of Python:

Debug Macros

Macro

Description

Rebuild EXT?

Py_DEBUG

A standard debug build. Py_DEBUG sets LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).

Yes

Py_REF_DEBUG

Turn on aggregate reference counting which will be displayed in the interactive interpreter when invoked with -X showrefcount on the command line. If you are not keeping references to objects and the count is increasing there is probably a leak. Also adds sys.gettotalrefcount() to the sys module and this returns the total number of references.

No

Py_TRACE_REFS

Turns on reference tracing. Sets Py_REF_DEBUG.

Yes

WITH_PYMALLOC

Enables Pythons small memory allocator. For Valgrind this must be disabled, if using Pythons malloc debugger (using PYMALLOC_DEBUG) this must be enabled. See: Python’s Memory Allocator

No

PYMALLOC_DEBUG

Enables Python’s malloc debugger that annotates memory blocks. Requires WITH_PYMALLOC. See: Python’s Memory Allocator

No

Here is the description of other debug macros that are set by one of the macros above:

Macro

Description

LLTRACE

Low level tracing. See Python/ceval.c.

20.2.4. Python’s Memory Allocator

A normal build of Python gives CPython a special memory allocator ‘PyMalloc’. When enabled this mallocs largish chunks of memory from the OS and then uses this pool for the actual PyObjects. With PyMalloc active Valgrind can not see all allocations and deallocations.

There are two Python builds of interest to help solve memory problems:

  • Disable PyMalloc so that Valgrind can analyse the memory usage.

  • Enable PyMalloc in debug mode, this creates memory blocks with special bit patterns and adds debugging information on each end of any dynamically allocated memory. This pattern is checked on every alloc/free and if found to be corrupt a diagnostic is printed and the process terminated.

To make a version of Python with its memory allocator suitable for use with Valgrind:

$ ../configure --with-pydebug --without-pymalloc
$ make

See Using Valgrind for using Valgrind.

To make a version of Python with its memory allocator using Python’s malloc debugger either:

$ ../configure CFLAGS='-DPYMALLOC_DEBUG' --with-pydebug
$ make

Or:

$ ../configure --with-pydebug
$ make EXTRA_CFLAGS="-DPYMALLOC_DEBUG"

This builds Python with the WITH_PYMALLOC and PYMALLOC_DEBUG macros defined.

20.2.4.1. Finding Access after Free With PYMALLOC_DEBUG

TODO:

Python built with PYMALLOC_DEBUG is the most effective way of detecting access after free. For example if we have this CPython code:

 1static PyObject *access_after_free(PyObject *Py_UNUSED(module)) {
 2    PyObject *pA = PyLong_FromLong(1024L * 1024L);
 3    fprintf(
 4        stdout,
 5        "%s(): Before Py_DECREF(0x%p) Ref count: %zd\n",
 6        __FUNCTION__, (void *)pA, Py_REFCNT(pA)
 7    );
 8    PyObject_Print(pA, stdout, Py_PRINT_RAW);
 9    fprintf(stdout, "\n");
10
11    Py_DECREF(pA);
12
13    fprintf(
14            stdout,
15            "%s(): After Py_DECREF(0x%p) Ref count: %zd\n",
16            __FUNCTION__, (void *)pA, Py_REFCNT(pA)
17    );
18    PyObject_Print(pA, stdout, Py_PRINT_RAW);
19    fprintf(stdout, "\n");
20
21    Py_RETURN_NONE;
22}

And we call this from the interpreter what we get is undefined and may vary from Python version to version. Here is Python 3.9:

 1# $ python
 2# Python 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15)
 3# [Clang 6.0 (clang-600.0.57)] on darwin
 4# Type "help", "copyright", "credits" or "license" for more information.
 5>>> from cPyExtPatt import cPyRefs
 6>>> cPyRefs.access_after_free()
 7access_after_free(): Before Py_DECREF(0x0x7fe55f1e2fb0) Ref count: 1
 81048576
 9access_after_free(): After Py_DECREF(0x0x7fe55f1e2fb0) Ref count: 140623120051984
101048576
11>>>

And Python 3.13:

 1# $ python
 2# Python 3.13.1 (v3.13.1:06714517797, Dec  3 2024, 14:00:22) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
 3# Type "help", "copyright", "credits" or "license" for more information.
 4>>> from cPyExtPatt import cPyRefs
 5>>> cPyRefs.access_after_free()
 6access_after_free(): Before Py_DECREF(0x0x102465890) Ref count: 1
 71048576
 8access_after_free(): After Py_DECREF(0x0x102465890) Ref count: 4333131856
 90
10>>>

And we call this from the (debug) Python 3.13 interpreter we get a diagnostic:

 1# (PyExtPatt_3.13.2_Debug) PythonExtensionPatterns git:(develop) $ python
 2# Python 3.13.2 (main, Mar  9 2025, 11:01:02) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
 3# Type "help", "copyright", "credits" or "license" for more information.
 4>>> from cPyExtPatt import cPyRefs
 5>>> cPyRefs.access_after_free()
 6access_after_free(): Before Py_DECREF(0x0x10a984e00) Ref count: 1
 71048576
 8access_after_free(): After Py_DECREF(0x0x10a984e00) Ref count: -2459565876494606883
 9<refcnt -2459565876494606883 at 0x10a984e00>
10>>>

20.2.4.2. Getting Statistics on PyMalloc

If the environment variable PYTHONMALLOCSTATS exists when running Python built with WITH_PYMALLOC``+``PYMALLOC_DEBUG then a (detailed) report of pymalloc activity is output on stderr whenever a new ‘arena’ is allocated.

$ PYTHONMALLOCSTATS=1 python

I have no special knowledge about the output you see when running Python this way which looks like this:

 1>>> from cPyExtPatt import cPyRefs
 2>>> cPyRefs.leak_new_reference(1000, 10000)
 3loose_new_reference: value=1000 count=10000
 4Small block threshold = 512, in 32 size classes.
 5
 6class   size   num pools   blocks in use  avail blocks
 7-----   ----   ---------   -------------  ------------
 8    1     32           1              83           427
 9    2     48           1             146           194
10    3     64          48           12240             0
11    4     80          82           16690            38
12    5     96          74           12409           171
13    6    112          22            3099            91
14    7    128          10            1156           114
15    8    144          12            1346            10
16    9    160           4             389            19
17   10    176           4             366             2
18   11    192          32            2649            71
19   12    208           3             173            61
20   13    224           2             132            12
21   14    240          13             860            24
22   15    256           5             281            34
23   16    272           6             344            16
24   17    288           7             348            44
25   18    304           8             403            21
26   19    320           4             198             6
27   20    336           4             183             9
28   21    352           5             198            32
29   22    368           3             119            13
30   23    384           3             104            22
31   24    400           3             106            14
32   25    416           3              98            19
33   26    432           8             295             1
34   27    448           2              64             8
35   28    464           3              70            35
36   29    480           2              44            24
37   30    496           2              59             5
38   31    512           2              45            17
39
40# arenas allocated total           =                    6
41# arenas reclaimed                 =                    0
42# arenas highwater mark            =                    6
43# arenas allocated current         =                    6
446 arenas * 1048576 bytes/arena     =            6,291,456
45
46# bytes in allocated blocks        =            5,927,280
47# bytes in available blocks        =              224,832
480 unused pools * 16384 bytes       =                    0
49# bytes lost to pool headers       =               18,144
50# bytes lost to quantization       =               22,896
51# bytes lost to arena alignment    =               98,304
52Total                              =            6,291,456
53
54arena map counts
55# arena map mid nodes              =                    1
56# arena map bot nodes              =                    1
57
58# bytes lost to arena map root     =              262,144
59# bytes lost to arena map mid      =              262,144
60# bytes lost to arena map bot      =              131,072
61Total                              =              655,360
62loose_new_reference: DONE

20.2.5. Identifying the Python Build Configuration from the Runtime

The module sysconfig allows you access to the configuration of the Python runtime. At its simplest, and most verbose, this can be used thus:

 1$ ./python.exe -m sysconfig
 2Platform: "macosx-10.9-x86_64"
 3Python version: "3.4"
 4Current installation scheme: "posix_prefix"
 5
 6Paths:
 7    data = "/usr/local"
 8    ...
 9    stdlib = "/usr/local/lib/python3.4"
10
11Variables:
12    ABIFLAGS = "dm"
13    AC_APPLE_UNIVERSAL_BUILD = "0"
14    AIX_GENUINE_CPLUSPLUS = "0"
15    AR = "ar"
16    ...
17    py_version = "3.4.3"
18    py_version_nodot = "34"
19    py_version_short = "3.4"

Importing sysconfig into an interpreter session gives two useful functions are get_config_var(...) which gets the setting for a particular macro and get_config_vars() which gets a dict of {macro : value, ...}. For example:

>>> import sysconfig
>>> sysconfig.get_config_var('Py_DEBUG')
1

For advanced usage you can parse any pyconfig.h into a dict by opening that file and passing it to sysconfig.parse_config_h(f) as a file object. sysconfig.get_config_h_filename() will give you the configuration file for the runtime (assuming it still exists). So:

>>> with open(sysconfig.get_config_h_filename()) as f:
      cfg = sysconfig.parse_config_h(f)