Inheritance in Python C++ extension

I have c++ library that need communicate with Python plugged in modules. Communication supposes implementing by Python some callback c++ interface.

I have read already about writing extensions, but no idea how to develop inheritance.

So something about: C++:

class Broadcast
{
   void set(Listener *){...
}

class Listener
{
    void notify(Broadcast* owner) = 0开发者_开发百科;
}

I need something like in Python:

class ListenerImpl(Listener):
    ...
    def notify(self, owner):
        ...

Note, I don't want use Boost.

Since I had to implement single inheritance as part of the Python C-API in a project of mine, I built a short example here. I marked the important statements in the code.

The trick is to inherit the base struct in the top of the subclass struct (leave out the PyObject_HEAD statement).

/* OBJECT */
typedef struct {
      MyPy_BaseClass super; // <----- PUTTING THIS FIRST INHERITS THE BASE PYTHON CLASS!!!
      // Own variables:
      // e.g int x = 0;
} MyPy_InheritanceClass;

Also dont forget to give the base type to the subclass type. There is a flag for it (see /* tp_base */).

  static PyTypeObject MyPy_InheritanceClass_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "MyPy_InheritanceClass",          /* tp_name */
    sizeof(MyPy_InheritanceClass),    /* tp_basicsize */
    0,                         /* tp_itemsize */
    (destructor)MyPy_InheritanceClass_dealloc, /* tp_dealloc */
    0,                         /* tp_print */
    0,                         /* tp_getattr */
    0,                         /* tp_setattr */
    0,                         /* tp_reserved */
    0,                         /* tp_repr */
    0,                         /* tp_as_number */
    0,                         /* tp_as_sequence */
    0,                         /* tp_as_mapping */
    0,                         /* tp_hash  */
    0,                         /* tp_call */
    0,                         /* tp_str */
    0,                         /* tp_getattro */
    0,                         /* tp_setattro */
    0,                         /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT |
    Py_TPFLAGS_BASETYPE,       /* tp_flags */ 
    "MyPy_InheritanceClass",   /* tp_doc */
    0,                         /* tp_traverse */
    0,                         /* tp_clear */
    0,                         /* tp_richcompare */
    0,                         /* tp_weaklistoffset */
    0,                         /* tp_iter */
    0,                         /* tp_iternext */
    MyPy_InheritanceClass_methods,    /* tp_methods */
    0,                         /* tp_members */
    0,                         /* tp_getset */
    &MyPy_BaseClass_Type,      /* tp_base */ // <------ GIVE THE BASE_CLASS TYPE
    0,                         /* tp_dict */
    0,                         /* tp_descr_get */
    0,                         /* tp_descr_set */
    0,                         /* tp_dictoffset */
    (initproc) MyPy_InheritanceClass_init, /* tp_init */
    0,                         /* tp_alloc */
    MyPy_InheritanceClass_new, /* tp_new */
};

Writing Python types in C that are inheritable is explained in PEP 253. It's not all that different from writing a normal builtin type as explained in the Extending/Embedding guide but you have to do certain things, like attribute access, through the Python API instead of accessing anything directly.

Exposing the Python subclasses back to C++ code is a little more tedious. The Python classes won't be C++ subclasses, so you need a C++ wrapper class (that does inherit from Listener) that contains a PyObject* for the Python subclass instance, and that has a notify method that translates the arguments to Python objects, calls the notify method of the PyObject* (using, e.g., PyObject_CallMethod), translates the result back to C++ types, and returns.

Minimal runnable single inheritance example with full C++ class interop

Defining Python classes with inheritance from C is "easy" because it is more or less well documented at: https://docs.python.org/3/extending/newtypes_tutorial.html#subclassing-other-types Here is an example that puts all those snippets into a single compilable file, and uses an explicitly defined base class instead of PyListObject: userland/libs/python_embed/pure.c.

What is harder and has no examples anywhere is a full C++ class object inheritance example, where you have to put data into C++ objects so that method calls will work normally.

The most annoying part is that there are two Python PyTypeObject objects, one for each class, but the C++ data must be allocated all at once for both. The only solution I could find to this was to use new on the derived class, which does not allocate if being initialized form the Derived class.

Maybe the following is correct. It passes my simple test, but is it leaking memory? Who knows!

Compile and run test:

g++ -std=c++11 -ggdb3 -O0 -pedantic-errors -Wall -Wextra -Wno-missing-field-initializers $(pkg-config --cflags python3) -o 'pure_cpp.out' 'pure_cpp.cpp' $(pkg-config --libs python3)
[ "$(PYTHONPATH="${PYTHONPATH:-}:." ./pure_cpp.out pure test_native_class)" = 13 ]

pure.py

import my_native_module

def test_native_class():
    # Positional arguments.
    my_native_object = my_native_module.MyNativeClass('ab', 'cd', 13)
    assert my_native_object.first == 'ab'
    assert my_native_object.last == 'cd'
    assert my_native_object.number == 13
    assert my_native_object.name() == 'ab cd 14'

    # Named arguments.
    my_native_object = my_native_module.MyNativeClass(first='ef', last='gh', number=13)
    assert my_native_object.name() == 'ef gh 14'

    # Default values and set property.
    my_native_object = my_native_module.MyNativeClass()
    my_native_object.first = 'ih'
    assert my_native_object.name() == 'ih  1'

    my_derived_native_object = my_native_module.MyDerivedNativeClass('ab', 'cd', 13, 'ef', 'gh', 21)
    assert my_derived_native_object.name() == 'ab cd 14'
    assert my_derived_native_object.name2() == 'ab cd 14 ef gh 23'

    return 13

pure_cpp.cpp

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"

#include <string>
#include <sstream>


/* Define a class in C to be accessible from the Python code.
 * and bind data and methods it to a C++ class!
 *
 * Data will be stored permanently only in the C++ class.
 * Python getters and setters copy data from that class
 * and create new objects.
 */

struct MyNativeClass {
    std::string first;
    std::string last;
    int number;

    MyNativeClass() : number(0) {};
    MyNativeClass(
        const std::string& first,
        const std::string& last,
        int number
    ) : first(first), last(last), number(number) {}

    std::string name() {
        std::stringstream ss;
        ss << first << " " << last << " " << number + 1;
        return ss.str();
    }
};

struct MyDerivedNativeClass : public MyNativeClass {
    std::string first2;
    std::string last2;
    int number2;

    MyDerivedNativeClass() {};
    MyDerivedNativeClass(
        const std::string& first,
        const std::string& last,
        int number,
        const std::string& first2,
        const std::string& last2,
        int number2
    ) : MyNativeClass(first, last, number),
      first2(first2), last2(last2), number2(number2) {}

    std::string name2() {
        std::stringstream ss;
        ss << first << " " << last << " " << number + 1 << " "
           << first2 << " " << last2 << " " << number2 + 2;
        return ss.str();
    }
};

typedef struct {
    PyObject_HEAD
    MyNativeClass *cpp_object;
} my_native_module_MyNativeClass;

static void
my_native_module_MyNativeClass_dealloc(my_native_module_MyNativeClass *self) {
    delete self->cpp_object;
    Py_TYPE(self)->tp_free((PyObject *) self);
}

static PyObject *
my_native_module_MyNativeClass_new_noalloc(my_native_module_MyNativeClass *self, PyObject *args, PyObject *kwds) {
    (void)args;
    (void)kwds;
    if (self != NULL) {
        if (self->cpp_object == NULL) {
            self->cpp_object = new MyNativeClass();
        }
    }
    return (PyObject *) self;
}

static PyObject *
my_native_module_MyNativeClass_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    my_native_module_MyNativeClass *self;
    self = (my_native_module_MyNativeClass *) type->tp_alloc(type, 0);
    return (PyObject *) my_native_module_MyNativeClass_new_noalloc(self, args, kwds);
}

static int
my_native_module_MyNativeClass_init(my_native_module_MyNativeClass *self, PyObject *args, PyObject *kwds)
{
    static const char *kwlist[] = {"first", "last", "number", NULL};
    PyObject *first = NULL, *last = NULL;
    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", (char**)kwlist,
            &first, &last, &self->cpp_object->number))
        return -1;
    if (first) {
        if (!PyUnicode_Check(first)) {
            PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string");
            return -1;
        }
        self->cpp_object->first = PyBytes_AsString(PyUnicode_AsASCIIString(first));
    }
    if (last) {
        if (!PyUnicode_Check(last)) {
            PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string");
            return -1;
        }
        self->cpp_object->last = PyBytes_AsString(PyUnicode_AsASCIIString(last));
    }
    return 0;
}

static PyObject *
my_native_module_MyNativeClass_getfirst(my_native_module_MyNativeClass *self, void *closure)
{
    (void)closure;
    return PyUnicode_FromString(self->cpp_object->first.c_str());
}

static int
my_native_module_MyNativeClass_setfirst(my_native_module_MyNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
    }
    if (!PyUnicode_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
    }
    self->cpp_object->first = PyBytes_AsString(PyUnicode_AsASCIIString(value));
    return 0;
}

static PyObject *
my_native_module_MyNativeClass_getlast(my_native_module_MyNativeClass *self, void *closure)
{
    (void)closure;
    return PyUnicode_FromString(self->cpp_object->last.c_str());
}

static int
my_native_module_MyNativeClass_setlast(my_native_module_MyNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
        return -1;
    }
    if (!PyUnicode_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The last attribute value must be a string");
        return -1;
    }
    self->cpp_object->last = PyBytes_AsString(PyUnicode_AsASCIIString(value));
    return 0;
}

static PyObject *
my_native_module_MyNativeClass_getnumber(my_native_module_MyNativeClass *self, void *closure)
{
    (void)closure;
    return PyLong_FromLong(self->cpp_object->number);
}

static int
my_native_module_MyNativeClass_setnumber(my_native_module_MyNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the number attribute");
        return -1;
    }
    if (!PyLong_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The number attribute value must be a string");
        return -1;
    }
    self->cpp_object->number = PyLong_AsLong(value);
    return 0;
}

static PyObject *
my_native_module_MyNativeClass_name(my_native_module_MyNativeClass *self, PyObject *Py_UNUSED(ignored))
{
    return PyUnicode_FromString(self->cpp_object->name().c_str());
}

static PyMethodDef my_native_module_MyNativeClass_methods[] = {
    {
        "name",
        (PyCFunction)my_native_module_MyNativeClass_name,
        METH_NOARGS,
        "Return the name, combining the first and last name"
    },
    {NULL}
};

static PyGetSetDef my_native_module_MyNativeClass_getsetters[] = {
    {(char *)"first", (getter) my_native_module_MyNativeClass_getfirst, (setter) my_native_module_MyNativeClass_setfirst,
     (char *)"first name", NULL},
    {(char *)"last", (getter) my_native_module_MyNativeClass_getlast, (setter) my_native_module_MyNativeClass_setlast,
     (char *)"last name", NULL},
    {(char *)"number", (getter) my_native_module_MyNativeClass_getnumber, (setter) my_native_module_MyNativeClass_setnumber,
     (char *)"number", NULL},
    {NULL}  /* Sentinel */
};

static PyTypeObject my_native_module_MyNativeClassType = {
    PyVarObject_HEAD_INIT(NULL, 0)
};

typedef struct {
    my_native_module_MyNativeClass base;
    MyDerivedNativeClass *cpp_object;
} my_native_module_MyDerivedNativeClass;

static void
my_native_module_MyDerivedNativeClass_dealloc(my_native_module_MyDerivedNativeClass *self) {
    Py_TYPE(self)->tp_base->tp_dealloc((PyObject *) self);
}

static PyObject *
my_native_module_MyDerivedNativeClass_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    (void)args;
    (void)kwds;
    my_native_module_MyDerivedNativeClass *self;
    self = (my_native_module_MyDerivedNativeClass *) type->tp_alloc(type, 0);
    self->cpp_object = new MyDerivedNativeClass();
    self->base.cpp_object = self->cpp_object;
    my_native_module_MyNativeClass_new_noalloc((my_native_module_MyNativeClass *) self, args, kwds);
    return (PyObject *) self;
}

static int
my_native_module_MyDerivedNativeClass_init(my_native_module_MyDerivedNativeClass *self, PyObject *args, PyObject *kwds)
{
    static const char *kwlist[] = {"first", "last", "number", "first2", "last2", "number2", NULL};
    PyObject *first = NULL, *last = NULL, *first2 = NULL, *last2 = NULL, *tmp;
    int ret;
    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOOOOi", (char**)kwlist,
            &first, &last, &tmp, &first2, &last2, &self->cpp_object->number2))
        return -1;
    /* args is a PyTuple, extract the first 3 arguments into a new
     * tuple to serve as arguments of the base class. */
    PyObject *base_args = PySequence_GetSlice(args, 0, 3);
    ret = my_native_module_MyNativeClassType.tp_init((PyObject *) self, base_args, kwds);
    Py_DECREF(base_args);
    if (ret < 0)
        return -1;
    if (first2) {
        if (!PyUnicode_Check(first2)) {
            PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string");
            return -1;
        }
        self->cpp_object->first2 = PyBytes_AsString(PyUnicode_AsASCIIString(first2));
    }
    if (last2) {
        if (!PyUnicode_Check(last2)) {
            PyErr_SetString(PyExc_TypeError, "The last2 attribute value must be a string");
            return -1;
        }
        self->cpp_object->last2 = PyBytes_AsString(PyUnicode_AsASCIIString(last2));
    }
    return 0;
}

static PyObject *
my_native_module_MyDerivedNativeClass_getfirst2(my_native_module_MyDerivedNativeClass *self, void *closure)
{
    (void)closure;
    return PyUnicode_FromString(self->base.cpp_object->first.c_str());
}

static int
my_native_module_MyDerivedNativeClass_setfirst2(my_native_module_MyDerivedNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first2 attribute");
        return -1;
    }
    if (!PyUnicode_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The first2 attribute value must be a string");
        return -1;
    }
    self->base.cpp_object->first = PyBytes_AsString(PyUnicode_AsASCIIString(value));
    return 0;
}

static PyObject *
my_native_module_MyDerivedNativeClass_getlast2(my_native_module_MyDerivedNativeClass *self, void *closure)
{
    (void)closure;
    return PyUnicode_FromString(self->base.cpp_object->last.c_str());
}

static int
my_native_module_MyDerivedNativeClass_setlast2(my_native_module_MyDerivedNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the last2 attribute");
        return -1;
    }
    if (!PyUnicode_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The last2 attribute value must be a string");
        return -1;
    }
    self->base.cpp_object->last = PyBytes_AsString(PyUnicode_AsASCIIString(value));
    return 0;
}

static PyObject *
my_native_module_MyDerivedNativeClass_getnumber2(my_native_module_MyDerivedNativeClass *self, void *closure)
{
    (void)closure;
    return PyLong_FromLong(self->base.cpp_object->number);
}

static int
my_native_module_MyDerivedNativeClass_setnumber2(my_native_module_MyDerivedNativeClass *self, PyObject *value, void *closure)
{
    (void)closure;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError, "Cannot delete the number2 attribute");
        return -1;
    }
    if (!PyLong_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The number2 attribute value must be a string");
        return -1;
    }
    self->base.cpp_object->number = PyLong_AsLong(value);
    return 0;
}

static PyObject *
my_native_module_MyDerivedNativeClass_name2(my_native_module_MyDerivedNativeClass *self, PyObject *Py_UNUSED(ignored))
{
    return PyUnicode_FromString(self->cpp_object->name2().c_str());
}

static PyGetSetDef my_native_module_MyDerivedNativeClass_getsetters[] = {
    {(char *)"first2", (getter) my_native_module_MyDerivedNativeClass_getfirst2, (setter) my_native_module_MyDerivedNativeClass_setfirst2,
     (char *)"first2 name", NULL},
    {(char *)"last2", (getter) my_native_module_MyDerivedNativeClass_getlast2, (setter) my_native_module_MyDerivedNativeClass_setlast2,
     (char *)"last2 name", NULL},
    {(char *)"number2", (getter) my_native_module_MyDerivedNativeClass_getnumber2, (setter) my_native_module_MyDerivedNativeClass_setnumber2,
     (char *)"number2", NULL},
    {NULL}  /* Sentinel */
};

static PyMethodDef my_native_module_MyDerivedNativeClass_methods[] = {
    {
        "name2",
        (PyCFunction)my_native_module_MyDerivedNativeClass_name2,
        METH_NOARGS,
        "Return the name2, combining the first2 and last2 name2"
    },
    {NULL}
};

static PyTypeObject my_native_module_MyDerivedNativeClassType = {
    PyVarObject_HEAD_INIT(NULL, 0)
};

static PyModuleDef my_native_module = {
    PyModuleDef_HEAD_INIT,
    "my_native_module",
    "My native module",
    -1,
};

PyMODINIT_FUNC
my_native_module_init_func(void) {
    PyObject *m;

    /* Create the module */
    m = PyModule_Create(&my_native_module);
    if (m == NULL)
        return NULL;

    /* Create MyNativeClass */
    if (PyType_Ready(&my_native_module_MyNativeClassType) < 0)
        return NULL;
    Py_INCREF(&my_native_module_MyNativeClassType);
    if (PyModule_AddObject(m, "MyNativeClass", (PyObject *) &my_native_module_MyNativeClassType) < 0) {
        Py_DECREF(&my_native_module_MyNativeClassType);
        Py_DECREF(m);
        return NULL;
    }

    /* Create MyDerivedNativeClass */
    my_native_module_MyDerivedNativeClassType.tp_base = &my_native_module_MyNativeClassType;
    if (PyType_Ready(&my_native_module_MyDerivedNativeClassType) < 0)
        return NULL;
    Py_INCREF(&my_native_module_MyDerivedNativeClassType);
    if (PyModule_AddObject(m, "MyDerivedNativeClass", (PyObject *) &my_native_module_MyDerivedNativeClassType) < 0) {
        Py_DECREF(&my_native_module_MyDerivedNativeClassType);
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    my_native_module_MyNativeClassType.tp_name = "my_native_module.MyNativeClass";
    my_native_module_MyNativeClassType.tp_doc = "My native class";
    my_native_module_MyNativeClassType.tp_basicsize = sizeof(my_native_module_MyNativeClass);
    my_native_module_MyNativeClassType.tp_itemsize = 0;
    my_native_module_MyNativeClassType.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC;
    my_native_module_MyNativeClassType.tp_new = my_native_module_MyNativeClass_new;
    my_native_module_MyNativeClassType.tp_init = (initproc) my_native_module_MyNativeClass_init;
    my_native_module_MyNativeClassType.tp_dealloc = (destructor) my_native_module_MyNativeClass_dealloc;
    my_native_module_MyNativeClassType.tp_methods = my_native_module_MyNativeClass_methods;
    my_native_module_MyNativeClassType.tp_getset = my_native_module_MyNativeClass_getsetters;

    my_native_module_MyDerivedNativeClassType.tp_name = "my_native_module.MyDerivedNativeClass";
    my_native_module_MyDerivedNativeClassType.tp_doc = "My native class";
    my_native_module_MyDerivedNativeClassType.tp_basicsize = sizeof(my_native_module_MyDerivedNativeClass);
    my_native_module_MyDerivedNativeClassType.tp_itemsize = 0;
    my_native_module_MyDerivedNativeClassType.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC;
    my_native_module_MyDerivedNativeClassType.tp_new = my_native_module_MyDerivedNativeClass_new;
    my_native_module_MyDerivedNativeClassType.tp_init = (initproc) my_native_module_MyDerivedNativeClass_init;
    my_native_module_MyDerivedNativeClassType.tp_dealloc = (destructor) my_native_module_MyDerivedNativeClass_dealloc;
    my_native_module_MyDerivedNativeClassType.tp_methods = my_native_module_MyDerivedNativeClass_methods;
    my_native_module_MyDerivedNativeClassType.tp_getset = my_native_module_MyDerivedNativeClass_getsetters;

    PyImport_AppendInittab("my_native_module", &my_native_module_init_func);
    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(argv[1]);
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("%ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            } else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        } else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    } else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    if (Py_FinalizeEx() < 0) {
        return 120;
    }
    return 0;
}

GitHub upstream.

You can reach similar results with about 10x less code (but possibly much longer compilation times) with pybind11 or other libraries, see e.g.: Calling C/C++ from Python?

Tested in Ubuntu 18.04, Python 3.6.8, GCC 7.4.0.