call C code from Python via ctypes, use python object list

Question

I want to

write a function sum_list of sum a list in c, named sum_list.c
make the sum_list.c file to sum_list.so
sum_list = ctypes.CDLL('./sum_list.so')
result = sum_list.sum_list([1, 2, 3])

It raise error in step 4:

ctypes.ArgumentError: argument 1: : Don't know how to convert parameter 1

when I write a function in c which add two numbers, it works ok.

so, the point is i don't know how to pass a list (python object) to c.

sum_list.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

long sum_list(PyObject *list)
{
    Py_ssize_t i, n;
    long total = 0, value;
    PyObject *item;

    n = PyList_Size(list);
    if (n < 0)
        return -1; /* Not a list */
    for (i = 0; i < n; i++) {
        item = PyList_GetItem(list, i); /* Can't fail */
        if (!PyLong_Check(item)) continue; /* Skip non-integers */
        value = PyLong_AsLong(item);
        if (value == -1 && PyErr_Occurred())
            /* Integer too big to fit in a C long, bail out */
            return -1;
        total += value;
    }
    return total;
}

python code

from ctypes import CDLL

sum_list = CDLL('./sum_list.so')

l = [1, 2, 3]
sum_list.sum_list(l)

I expect the var result in my python code is 6.

PyList_GetItem(list, i); /* Can't fail */ is actually false. It can fail - especially if the list changes size for some reason during the iteration. It might be hard to deduce that it doesn't! — Antti Haapala
– Antti Haapala, Commented Jun 6, 2019 at 4:36
@AnttiHaapala I think i understand the reason... I copy the code from docs.python.org/3/c-api/intro.html#coding-standards can you show some brief demo of how the address passing can possible? — kun
– kun, Commented Jun 6, 2019 at 5:22
@kun Please, don't forget to come back and upvote/accept answers that were helpful to you. That's how we say "thank you" around here. — code_dredd
– code_dredd, Commented Jul 18, 2019 at 16:43

Antti Haapala · Accepted Answer · 2019-06-06 07:34:29Z

6

The ctypes library is meant for calling functions in pure C libraries. The example that you found is for a function that would go into a Python extension module, and be usable without ctypes. Ctypes actually has a type for PyObject *, namely ctypes.py_object. You need to wrap the list in this - be careful to ensure that it will stay alive!

In any case you must almost always provide the correct prototype of the function using the restype and argtypes.

Here's a working example:

import ctypes

sum_list = ctypes.PyDLL('./sum_list.so')
sum_list.restype = ctypes.c_long
sum_list.argtypes = [ ctypes.py_object ]

l = [1, 2, 3]
print(sum_list.sum_list(ctypes.py_object(l)))

Note, as noticed by Mark, you must use PyDLL here instead, to ensure that the GIL is not released since the function does not acquire it!

edited Jun 6, 2019 at 7:34

answered Jun 6, 2019 at 6:28

Antti Haapala

135k23 gold badges297 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mark Tolonen Over a year ago

CDLL releases the GIL. Make sure to use PyDLL if the function uses Python C APIs.

Antti Haapala Over a year ago

@MarkTolonen fixed. It shows that I've never done this ;)

Mark Tolonen Over a year ago

Esoteric stuff :^)

kun Over a year ago

thanks! use C in python are so different than I imagine.

code_dredd · Accepted Answer · 2019-06-06 06:59:05Z

2

I've up-voted @Antii's answer, but still wanted to add a short, complete, and self-contained example.

First, ctypes is to interact with C code that was written without Python interoperability in mind. Generally, this means code that does not #include <Python.h> and so on. For example:

/**
 * Sample function to call from Python 3
 * Build:
 *  $ gcc -shared -o libmath.so -fPIC math.c
 *
 * See https://stackoverflow.com/a/14884166/4594973
 *
 **/
int add(int x, int y) {
    return x + y;
}

While the code above is trivial, you need to make sure that more complicated pieces of code are compiled properly. For example, if you're using a C++ comnpiler and/or are using features specific to C++ (e.g. classes, method overloading, etc), then you must expose a C-compatible interface when building the library, as C++'s name mangling will make it next to impossible to find the function after the fact.

After building the C code above, use this Python script to call it:

#!/usr/bin/env python3

import ctypes

cmath = ctypes.CDLL('./libmath.so')
cmath.restype = ctypes.c_int
cmath.argtypes = [ctypes.c_int, ctypes.c_int]

x = 4
y = 5
r = cmath.add(x, y)

print(f'{x} + {y} = {r}')

Running the samples from my terminal:

$ gcc -shared -o libmath.so -fPIC math.c
$ python3 math.py
4 + 5 = 9

edited Jun 6, 2019 at 6:59

answered Jun 6, 2019 at 6:57

code_dredd

6,1351 gold badge27 silver badges57 bronze badges

2 Comments

code_dredd Over a year ago

BTW, the c in my sample cmath library here stands for cheap ._.

kun Over a year ago

I think I should consider this problem in a new way, in a C way maybe!

Mark Tolonen · Accepted Answer · 2019-06-06 07:43:48Z

As the other answer mentioned, you can use ctypes.py_object to represent an actual Python object, but you must also remember when calling Python C APIs you must not release the GIL (global interpreter lock). Use PyDLL instead of CDLL to accomplish this.

Here's a fully working example that works for sequences in general:

test.c

#include <Python.h>

#ifdef _WIN32
#   define API __declspec(dllexport)
#else
    define API
#endif

API long sum_list(PyObject* o)
{
    Py_ssize_t i, n;
    long total = 0, value;
    PyObject* list;
    PyObject* item;

    list = PySequence_Fast(o,"not a sequence");  // New reference!  Must DECREF eventually.
    if(list == NULL)
        return -1;
    n = PySequence_Fast_GET_SIZE(list);
    for (i = 0; i < PySequence_Fast_GET_SIZE(list); i++) {
        item = PySequence_Fast_GET_ITEM(list, i);
        if (!PyLong_Check(item))
        {
            PyErr_SetString(PyExc_TypeError,"sequence contains non-integer");
            Py_DECREF(list);
            return -1;
        }
        value = PyLong_AsLong(item);
        if (value == -1 && PyErr_Occurred())
        {
            Py_DECREF(list);
            return -1;
        }
        total += value;
    }
    Py_DECREF(list);
    return total;
}

test.py

from ctypes import *

dll = PyDLL('test')
dll.sum_list.restype = c_long
dll.sum_list.argtypes = py_object,

print(dll.sum_list((1,2,3,4,5)))
try:
    print(dll.sum_list(7))
except TypeError as e:
    print(e)
try:
    print(dll.sum_list((1,2,'a',4,5)))
except TypeError as e:
    print(e)
try:
    print(dll.sum_list((1,2,0x80000000,4,5)))
except OverflowError as e:
    print(e)
try:
    print(dll.sum_list((1,2,-0x80000001,4,5)))
except OverflowError as e:
    print(e)

Output:

15
not a sequence
sequence contains non-integer
Python int too large to convert to C long
Python int too large to convert to C long

Collectives™ on Stack Overflow

call C code from Python via ctypes, use python object list

3 Answers 3

4 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related