PyGILState_Ensure() deadlocks, why?

Discussion:

(too old to reply)

Tomas Ukkonen

2024-07-07 19:24:04 UTC

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
*
* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
* ./a.out
*
* uname:
* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
// Simulate a long-running task
std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
// Ensure this thread has the GIL
PyGILState_STATE gstate = PyGILState_Ensure();

// Execute some Python code
PyRun_SimpleString("print('Hello from std::thread!')");

// Release the GIL for long operation
Py_BEGIN_ALLOW_THREADS
perform_long_operation();
Py_END_ALLOW_THREADS

// Re-acquire the GIL and execute more Python code
gstate = PyGILState_Ensure();
PyRun_SimpleString("print('Thread operation completed!')");

// Release the GIL
PyGILState_Release(gstate);
}

int main() {
// Initialize the Python Interpreter
Py_Initialize();

// Create a vector of threads
std::vector<std::thread> threads;

// Launch threads
for (int i = 0; i < 5; ++i) {
threads.push_back(std::thread(thread_function));
}

// Join threads
for (auto& t : threads) {
t.join();
}

// Finalize the Python Interpreter
Py_Finalize();

return 0;
}

Tomas Ukkonen

Tomas Ukkonen

2024-07-07 19:40:56 UTC

Permalink

Hi

There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).

Updated code:

/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
*
* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
* ./a.out
*
* uname:
* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void thread_function() {
// Ensure this thread has the GIL
PyGILState_STATE gstate = PyGILState_Ensure();

// Execute some Python code
PyRun_SimpleString("print('Hello from std::thread!')");

// Release the GIL
PyGILState_Release(gstate);
}

int main() {
// Initialize the Python Interpreter
Py_Initialize();

// Create a vector of threads
std::vector<std::thread> threads;

// Launch threads
for (int i = 0; i < 5; ++i) {
threads.push_back(std::thread(thread_function));
}

// Join threads
for (auto& t : threads) {
t.join();
}

// Finalize the Python Interpreter
Py_Finalize();

return 0;
}

Post by Tomas Ukkonen
Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.
/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
*
* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
* ./a.out
*
* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
*/
#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>
void perform_long_operation() {
// Simulate a long-running task
std::this_thread::sleep_for(std::chrono::seconds(5));
}
void thread_function() {
// Ensure this thread has the GIL
PyGILState_STATE gstate = PyGILState_Ensure();
// Execute some Python code
PyRun_SimpleString("print('Hello from std::thread!')");
// Release the GIL for long operation
Py_BEGIN_ALLOW_THREADS
perform_long_operation();
Py_END_ALLOW_THREADS
// Re-acquire the GIL and execute more Python code
gstate = PyGILState_Ensure();
PyRun_SimpleString("print('Thread operation completed!')");
// Release the GIL
PyGILState_Release(gstate);
}
int main() {
// Initialize the Python Interpreter
Py_Initialize();
// Create a vector of threads
std::vector<std::thread> threads;
// Launch threads
for (int i = 0; i < 5; ++i) {
threads.push_back(std::thread(thread_function));
}
// Join threads
for (auto& t : threads) {
t.join();
}
// Finalize the Python Interpreter
Py_Finalize();
return 0;
}
Tomas Ukkonen

MRAB

2024-07-07 22:44:01 UTC

Permalink

Post by Tomas Ukkonen
Hi
There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).
/* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
*
* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
* ./a.out
*
* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
*/
#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>
void thread_function() {
// Ensure this thread has the GIL
PyGILState_STATE gstate = PyGILState_Ensure();
// Execute some Python code
PyRun_SimpleString("print('Hello from std::thread!')");
// Release the GIL
PyGILState_Release(gstate);
}
int main() {
// Initialize the Python Interpreter
Py_Initialize();
// Create a vector of threads
std::vector<std::thread> threads;
// Launch threads
for (int i = 0; i < 5; ++i) {
threads.push_back(std::thread(thread_function));
}
// Join threads
for (auto& t : threads) {
t.join();
}
// Finalize the Python Interpreter
Py_Finalize();
return 0;
}

At this point, there's only one thread (the main thread) and it owns the
GIL.

Post by Tomas Ukkonen

Post by Tomas Ukkonen
// Create a vector of threads
std::vector<std::thread> threads;
// Launch threads
for (int i = 0; i < 5; ++i) {
threads.push_back(std::thread(thread_function));
}

The threads will each try to acquire and release the GIL, but it's still
owned by the main thread.

Post by Tomas Ukkonen

Post by Tomas Ukkonen
// Join threads
for (auto& t : threads) {
t.join();
}

The main thread is waiting for the sub-threads to finish, and the
threads waiting for the GIL, but the main thread still owns the GIL, so
they'll be waiting forever. Deadlock.

Post by Tomas Ukkonen

Post by Tomas Ukkonen
// Finalize the Python Interpreter
Py_Finalize();
return 0;
}

Barry

2024-07-07 22:21:55 UTC

Permalink

Post by Tomas Ukkonen
Py_Initialize();

You also need to tell python to init threading.
I think you are missing more python setup code before you can use threads.
Also i think you need to tell python that your thread wants to call into python.
But I an not near my dev system to research this for you.

I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp
That saves the thread state and restores it.

But in my case python creates the threads and I release and acquire the GIL.

Barry

Barry Scott

2024-07-08 08:33:39 UTC

Permalink

Post by Barry

Post by Tomas Ukkonen
Py_Initialize();

You also need to tell python to init threading.

I'm in front of my dev machine now and checking up on threading.

There is no longer any extra init for threads required.

Post by Barry
I think you are missing more python setup code before you can use threads.
Also i think you need to tell python that your thread wants to call into python.
But I an not near my dev system to research this for you.

You are right to use PyGILState_Ensure()

But as MRAB says the main thread is holding the GIL.

Post by Barry
I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Source/pysvn.cpp
That saves the thread state and restores it.

You still might find the classes I wrong to manage GIL acquire and release interesting.
I have the C++ type system enforcing the rules of acquire and release.
As well as RAII ensuring never to leave a block with the GIL in the wrong state.

Barry