Other Topics

Wrapping Enums

SIP wraps C/C++ enums using a dedicated Python type and implements behaviour that mimics the C/C++ behaviour regarding the visibility of the enum’s members. In other words, an enum’s members have the same visibility as the enum itself. For example:

class MyClass
{
public:
    enum MyEnum
    {
        Member
    }
}

In Python the Member member is referenced as MyClass.Member. This behaviour makes it easier to translate C/C++ code to Python.

In more recent times C++11 has introduced scoped enums and Python has introduced the enum module. In both cases a member is only visible in the scope of the enum. In other words, the Member member is referenced as MyClass.MyEnum.Member.

SIP generates bindings for C++11 scoped enums and implements them as Python enum.Enum objects.

A disadvantage of the above is that the Python programmer needs to know the nature of the C/C++ enum in order to access its members. In order to avoid this, SIP makes the members of traditional C/C++ enums visible from the scope of the enum as well.

It is recommended that Python code should always specify the enum scope when referencing an enum member. A future version of SIP will remove support for the traditional behaviour.

Ownership of Objects

When a C++ instance is wrapped a corresponding Python object is created. The Python object behaves as you would expect in regard to garbage collection - it is garbage collected when its reference count reaches zero. What then happens to the corresponding C++ instance? The obvious answer might be that the instance’s destructor is called. However the library API may say that when the instance is passed to a particular function, the library takes ownership of the instance, i.e. responsibility for calling the instance’s destructor is transferred from the SIP generated module to the library.

Ownership of an instance may also be associated with another instance. The implication being that the owned instance will automatically be destroyed if the owning instance is destroyed. SIP keeps track of these relationships to ensure that Python’s cyclic garbage collector can detect and break any reference cycles between the owning and owned instances. The association is implemented as the owning instance taking a reference to the owned instance.

The TransferThis, Transfer and TransferBack argument annotations are used to specify where, and it what direction, transfers of ownership happen. It is very important that these are specified correctly to avoid crashes (where both Python and C++ call the destructor) and memory leaks (where neither Python and C++ call the destructor).

This applies equally to C structures where the structure is returned to the heap using the free() function.

See also sipTransferTo() and sipTransferBack().

Types and Meta-types

Every Python object (with the exception of the object object itself) has a meta-type and at least one super-type. By default an object’s meta-type is the meta-type of its first super-type.

SIP implements two super-types, sip.simplewrapper and sip.wrapper, and a meta-type, sip.wrappertype.

sip.simplewrapper is the super-type of sip.wrapper. The super-type of sip.simplewrapper is object.

sip.wrappertype is the meta-type of both sip.simplewrapper and sip.wrapper. The super-type of sip.wrappertype is type.

sip.wrapper supports the concept of object ownership described in Ownership of Objects and, by default, is the super-type of all the types that SIP generates.

sip.simplewrapper does not support the concept of object ownership but SIP generated types that are sub-classed from it have Python objects that take less memory.

SIP allows a class’s meta-type and super-type to be explicitly specified using the Metatype and Supertype class annotations.

SIP also allows the default meta-type and super-type to be changed for a module using the %DefaultMetatype and %DefaultSupertype directives. Unlike the default super-type, the default meta-type is inherited by importing modules.

If you want to use your own meta-type or super-type then they must be sub-classed from one of the SIP provided types. Your types must be registered using sipRegisterPyType(). This is normally done in code specified using the %InitialisationCode directive.

Note

It is not possible to define new super-types or meta-types if the limited Python API is enabled.

Lazy Type Attributes

Instead of populating a wrapped type’s dictionary with its attributes (or descriptors for those attributes) SIP only creates objects for those attributes when they are actually needed. This is done to reduce the memory footprint and start up time when used to wrap large libraries with hundreds of classes and tens of thousands of attributes.

SIP allows you to extend the handling of lazy attributes to your own attribute types by allowing you to register an attribute getter handler (using sipRegisterAttributeGetter()). This will be called just before a type’s dictionary is accessed for the first time.

Overflow Checking

By default SIP does not check for overflow when converting Python number objects to C/C++ types. Overflowed values are undefined - it cannot be assumed that upper bits are simply discarded.

SIP allows overflow checking to be enabled and disabled by the bindings author (using sipEnableOverflowChecking()) or by the application developer (using sip.enableoverflowchecking()).

It is recommended that bindings authors should always enable overflow checking by default.

Support for Python’s Buffer Interface

SIP supports Python’s buffer interface in that whenever C/C++ requires a char or char * type then any Python type that supports the buffer interface (including ordinary Python strings) can be used.

Support for Wide Characters

SIP supports the use of wide characters (i.e. the wchar_t type). Python’s C API includes support for converting between str objects and wide character strings and arrays. When converting from a str object to wide characters SIP creates the string or array on the heap (using memory allocated using sipMalloc()). This then raises the problem of how this memory is subsequently freed.

The following describes how SIP handles this memory in the different situations where this is an issue.

  • When a wide string or array is passed to a function or method then the memory is freed (using sipFree()) after that function or method returns.

  • When a wide string or array is returned from a virtual method then SIP does not free the memory until the next time the method is called.

  • When an assignment is made to a wide string or array instance variable then SIP does not first free the instance’s current string or array.

The Python Global Interpreter Lock

Python’s Global Interpretor Lock (GIL) must be acquired before calls can be made to the Python API. It should also be released when a potentially blocking call to C/C++ library is made in order to allow other Python threads to be executed. In addition, some C/C++ libraries may implement their own locking strategies that conflict with the GIL causing application deadlocks. SIP provides ways of specifying when the GIL is released and acquired to ensure that locking problems can be avoided.

SIP always ensures that the GIL is acquired before making calls to the Python API. By default SIP does not release the GIL when making calls to the C/C++ library being wrapped. The ReleaseGIL annotation can be used to override this behaviour when required.

If the release-gil key is set to true in the bindings-specific section of the pyproject.toml file then (for that set of bindings) then the default behaviour is changed and SIP releases the GIL every time is makes calls to the C/C++ library being wrapped. The HoldGIL annotation can be used to override this behaviour when required.

Writing %ConvertToSubClassCode

When SIP needs to wrap a C++ class instance it first checks to make sure it hasn’t already done so. If it has then it just returns a new reference to the corresponding Python object. Otherwise it creates a new Python object of the appropriate type. In C++ a function may be defined to return an instance of a certain class, but can often return a sub-class instead.

The %ConvertToSubClassCode directive is used to specify handwritten code that exploits any available real-time type information (RTTI) to see if there is a more specific Python type that can be used when wrapping the C++ instance. The RTTI may be provided by the compiler or by the C++ instance itself.

The directive is included in the specification of one of the classes that the handwritten code handles the type conversion for. It doesn’t matter which one, but a sensible choice would be the one at the root of that class hierarchy in the module.

Note

In a future version of SIP this use of the directive will be deprecated and it will instead be placed outside any class specification.

If a class hierarchy extends over a number of modules then this directive should be used in each of those modules to handle the part of the hierarchy defined in that module. SIP will ensure that the different pieces of code are called in the right order to determine the most specific Python type to use.

A class has at least one convertor if it or any super-class defines %ConvertToSubClassCode. A convertor has a base class. If a class that defines %ConvertToSubClassCode does not have a super-class that defines %ConvertToSubClassCode then that class is the base class. Otherwise the base class is that of the right-most super-class that has a convertor. In this case the %ConvertToSubClassCode extends all other convertors with the same base class.

Consider the following class hierarchy:

A
  \
   B*     C*
     \  /   \
      D      E
    /   \
  F       G*

The classes marked with an asterisk define %ConvertToSubClassCode.

Classes A to F are implemented in module X. Class G is implemented in module Y.

We can say the following:

A convertor is invoked when SIP needs to wrap a C++ instance and the type of that instance is a sub-class of the convertor’s base class. The convertor is passed a pointer to the instance cast to the base class. The convertor then, if possible, casts that pointer to an instance of a sub-class of its original class. It also returns a pointer to the corresponding generated type structure.

It is possible for a convertor to switch to another convertor. This can avoid duplication of convertor code where there is multiple inheritance.

When more than one convertor may be invoked they are done so in the order that reflects the module hierarchy. When the convertors are defined in the same module then the order is undefined. Convertors must be written with this mind.

Given the class hierarchy shown above, lets say that SIP needs to wrap an instance of known to be of class D but is actually of class F. We want the conversion mechanism to recognise that fact and return a Python object of type F. The following steps are taken:

  • G’s %ConvertToSubClassCode is invoked and passed the pointer to D cast to C. This convertor only recognises instances of class G and so returns a value that indicates it was unable to perform a conversion.

  • SIP will now invoke either B’s %ConvertToSubClassCode or C’s %ConvertToSubClassCode. As they are defined in the same module which is chosen is undefined. Let’s assume it is the C convertor that is invoked.

  • The convertor recognises that the instance is of class D (rather than C or E). It must also determine whether this really is D or whether it is actually F. Of course B’s %ConvertToSubClassCode must also make the same distinction. Rather than possibly duplicating the required code in both convertors the C convertor switches to the B convertor. It does this by casting the pointer it is trying to convert to B and returns B’s generated type structure.

Managing Incompatible APIs

Deprecated since version 5.0: This will be removed in v6.

Sometimes it is necessary to change the way something is wrapped in a way that introduces an incompatibility. For example a new feature of Python may suggest that something may be wrapped in a different way to exploit that feature.

SIP’s %Feature directive could be used to provide two different implementations. However this would mean that the choice between the two implementations would have to be made when building the generated module potentially causing all sorts of deployment problems. It may also require applications to work out which implementation was available and to change their behaviour accordingly.

Instead SIP provides limited support for providing multiple implementations (of classes, mapped types and functions) that can be selected by an application at run-time. It is then up to the application developer how they want to manage the migration from the old API to the new, incompatible API.

This support is implemented in three parts.

Firstly the %API directive is used to define the name of an API and its default version number. The default version number is the one used if an application doesn’t explicitly set the version number to use.

Secondly the API class, mapped type or function annotation is applied accordingly to specify the API and range of version numbers that a particular class, mapped type or function implementation should be enabled for.

Finally the application calls sip.setapi() to specify the version number of the API that should be enabled. This call must be made before any module that has multiple implementations is imported for the first time.

Note this mechanism is not intended as a way or providing equally valid alternative APIs. For example:

%API(name=MyAPI, version=1)

class Foo
{
public:
    void bar();
};

class Baz : Foo
{
public:
    void bar() /API=MyAPI:2-/;
};

If the following Python code is executed then an exception will be raised:

b = Baz()
b.bar()

This is because when version 1 of the MyAPI API (the default) is enabled there is no Baz.bar() implementation and Foo.bar() will not be called instead as might be expected.