Discussion:
python C-api and thread
(too old to reply)
aotto1968
2024-08-05 21:19:14 UTC
Permalink
hi,

Is it possible to run two completely independent Python interpreters in one process, each using a thread?

By independent, I mean that no data is shared between the interpreters and thus the C API can be used without any other
"lock/GIL" etc.

mfg
Lawrence D'Oliveiro
2024-08-06 00:32:49 UTC
Permalink
Post by aotto1968
Is it possible to run two completely independent Python interpreters in
one process, each using a thread?
By independent, I mean that no data is shared between the interpreters
and thus the C API can be used without any other "lock/GIL" etc.
Seems like yes
<https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreterFromConfig>.
aotto1968
2024-08-06 06:01:41 UTC
Permalink
Post by Lawrence D'Oliveiro
Post by aotto1968
Is it possible to run two completely independent Python interpreters in
one process, each using a thread?
By independent, I mean that no data is shared between the interpreters
and thus the C API can be used without any other "lock/GIL" etc.
Seems like yes
<https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreterFromConfig>.
→ I think that could be a solution.
Grant Edwards
2024-08-06 02:34:05 UTC
Permalink
Post by aotto1968
Is it possible to run two completely independent Python interpreters
in one process, each using a thread?
By independent, I mean that no data is shared between the
interpreters and thus the C API can be used without any other
"lock/GIL" etc.
No, not using any OS I've ever seen. The usual definition of "threads"
is that they share data, and the definition of "processes" is that
processes don't share data.

How exactly does what you're trying to do differ from runnig two
Python interpreters in two processes?

--
Grant
aotto1968
2024-08-06 06:11:39 UTC
Permalink
Post by Grant Edwards
Post by aotto1968
Is it possible to run two completely independent Python interpreters
in one process, each using a thread?
By independent, I mean that no data is shared between the
interpreters and thus the C API can be used without any other
"lock/GIL" etc.
No, not using any OS I've ever seen. The usual definition of "threads"
is that they share data, and the definition of "processes" is that
processes don't share data.
How exactly does what you're trying to do differ from runnig two
Python interpreters in two processes?
--
Grant
I know but I use a thread like a process because the "conversation" between the threads is done by my
software. a Thread is usually faster to startup (thread-pool) this mean for high-load this is
significant faster even than fork.
Barry Scott
2024-08-06 20:12:46 UTC
Permalink
Post by aotto1968
I know but I use a thread like a process because the "conversation" between the threads is done by my
software. a Thread is usually faster to startup (thread-pool) this mean for high-load this is
significant faster even than fork.
using processes means you are more robust and can survive a process crash.
using async, assuming you do enough I/O, will out perform threads.

Barry
Lawrence D'Oliveiro
2024-08-06 23:41:45 UTC
Permalink
... a Thread is usually faster to startup (thread-pool) ...
Process pools exist, too. As I recall, Apache and Android use them.
Chris Angelico
2024-08-06 02:40:18 UTC
Permalink
On Tue, 6 Aug 2024 at 08:48, aotto1968 via Python-list
Post by aotto1968
hi,
Is it possible to run two completely independent Python interpreters in one process, each using a thread?
By independent, I mean that no data is shared between the interpreters and thus the C API can be used without any other
"lock/GIL" etc.
You're probably thinking of subinterpreters:

https://peps.python.org/pep-0734/

ChrisA
Loading...