Cython and C pointers

Cython is a powerful tool that can speed up your Python code or can help you to quickly create an extension module for a library that has c bindings.

This post is not a cython tutorial but its about a problem I have encountered during the development of a wrapper to Wakaama LWM2M library; if you want to learn more about Cython please refer to its documentation.

During the development of said wrapper I have had a nasty strange bug affecting my code, at some point some object referenced by a pointer kept at the C library side seemed to change its value! For example think about this situation:

• you pass an object to a certain c function

• the object's pointer is stored internally by the c library

• you retrieve the object from the c library but its content its not the same...it's not even an instance of the same class!

To reproduce this behaviour please have a look at overwrite.py in my test project.

At first I've been thinking about memory corruption caused by the C library code; this wrong assumption costed me at least two days of messing with gdb (that saved my life in another memory related bug) that have not given any useful insight.

I don't remember how I've spotted this problem but probably I should have tryed everything until something pointed me to the right direction.

What seems to happen here is something like this:

1. create an object and assign it to a variable A

2. store its pointer in the C library

3. get back the pointer of the object; it will be the same object as referenced by A

4. assign a new object to A

5. create an object assign it to a variable B

6. get back the pointer of the object; it will NOT be the same object as referenced by A but it will be the one referenced by B

When the variable A is referencing the new object the old one is not referenced by anyone in the python world and its memory will became available to store the new object assigned to B and the pointer in the C world is now pointing to what is referenced by B.

It may seems obvious but when it happened to me it was inside a callback originating in the C code and being handled by a function defined in the cython context.