C API: Making a context manager

[Chris Kaynor]

I am currently rewritting a class using the Python C API to improve
performance of it, however I have not been able to find any
documentation about how to make a context manager using the C API.

The code I am working to produce is the following (its a method of a class):

@contextlib.contextmanager
def connected(self, *args, **kwargs):
connection = self.connect(*args, **kwargs)
try:
yield
finally:
connection.disconnect()

For this, my first question is: is there any built-in method to make
this type of method in the C API? If not, is there a slot on the type
object I am missing for __enter__ and __exit__, or should just be
defined using the PyMethodDef struct on the class (presumably named
the same as the Python functions)?

Chris

[Brian Curtin]

Post by Chris Kaynor
I am currently rewritting a class using the Python C API to improve
performance of it, however I have not been able to find any
documentation about how to make a context manager using the C API.
@contextlib.contextmanager
       connection = self.connect(*args, **kwargs)
               yield
               connection.disconnect()
For this, my first question is: is there any built-in method to make
this type of method in the C API? If not, is there a slot on the type
object I am missing for __enter__ and __exit__, or should just be
defined using the PyMethodDef struct on the class (presumably named
the same as the Python functions)?

You'd just add "__enter__" and "__exit__" in the PyMethodDef. If you
have the CPython source, we do it in there in a few places. Off the
top of my head, PC\winreg.c contains at least one class that works as
a context manager (PyHKEY), although there are a few others scattered
around the source.

[Chris Kaynor]

Post by Brian Curtin
You'd just add "__enter__" and "__exit__" in the PyMethodDef. If you
have the CPython source, we do it in there in a few places. Off the
top of my head, PC\winreg.c contains at least one class that works as
a context manager (PyHKEY), although there are a few others scattered
around the source.

That is what I figured. I was just hoping there was some helper class
similar to the contextmanager decorator that would make it easier to
use, however at the same time it makes sense that there is not.

Thanks,
Chris

[Stefan Behnel]

Post by Chris Kaynor
I am currently rewritting a class using the Python C API to improve
performance of it, however I have not been able to find any
documentation about how to make a context manager using the C API.

You should take a look at Cython. It makes these things *so* much easier.

Post by Chris Kaynor
@contextlib.contextmanager
connection = self.connect(*args, **kwargs)
yield
connection.disconnect()

You can use the above in Cython code unmodified, and it's likely going to
be good enough (depending on what kind of connection we are talking about
here).

In case that's also performance critical, however, it's likely faster to
spell it out as a class, i.e.

...
def connected(self, *args, **kwargs):
return ConnectionManager(self, args, kwargs)


cdef class ConnectionManager:
cdef object args, kwargs, connection, connect

def __init__(self, connector, args, kwargs):
self.args, self.kwargs = args, kwargs
self.connect = connector.connect
def __enter__(self):
self.connection = self.connect(*self.args, **self.kwargs)
def __exit__(self, *exc):
self.connection.disconnect()
return True # or False? I always forget which means what

Not that much longer either.

Stefan

[Chris Kaynor]

Post by Stefan Behnel

Post by Chris Kaynor
I am currently rewritting a class using the Python C API to improve
performance of it, however I have not been able to find any
documentation about how to make a context manager using the C API.

You should take a look at Cython. It makes these things *so* much easier.

Unfortunately, all of the code has to be fully compatible with CPython
2.6 - it needs to function inside of Maya, which has CPython 2.6
embedded, and to which we do not have the source code.

While not all parts of our code base are used inside of Maya, most of
the performance-critical items are to some degree or another.

In this particular case, the connected context manager is not heavily
used (outside of unittests) and itself is not performance critical,
but the much of the rest of the package (and thus the class) it is
part of is.

Chris

[Stefan Behnel]

Post by Chris Kaynor

Post by Stefan Behnel

Post by Chris Kaynor
I am currently rewritting a class using the Python C API to improve
performance of it, however I have not been able to find any
documentation about how to make a context manager using the C API.

You should take a look at Cython. It makes these things *so* much easier.

Unfortunately, all of the code has to be fully compatible with CPython
2.6 - it needs to function inside of Maya, which has CPython 2.6
embedded, and to which we do not have the source code.

This sounds like you're misunderstanding what Cython is. Cython compiles
(and optimises) your Python code into fast C code that uses the C-API (and
that can happily call into external C code etc.). So you get basically the
same (and sometimes better) speed, but without all the C-level hassle and
maintenance issues. The C code that Cython generates is fully compatible
with CPython 2.4 up to the latest 3.3, and that includes 2.6.

Post by Chris Kaynor
While not all parts of our code base are used inside of Maya, most of
the performance-critical items are to some degree or another.
In this particular case, the connected context manager is not heavily
used (outside of unittests) and itself is not performance critical,
but the much of the rest of the package (and thus the class) it is
part of is.

In that case, I advise you to leave the context manager code as is, and
just compile the module that you are trying to speed up (and which, IIUC is
currently written in Python) with Cython, then optimise the parts of it
that need more speed by injecting static typing. Here's a quick howto:

http://docs.cython.org/src/quickstart/cythonize.html

Stefan