Best (simplest) way to share data between processes

[Chris Green]

I have a Raspberry Pi in my boat that uses I2C to read a number of
voltages and currents (using ADS1115 A2D) so I can monitor the battery
condition etc.

At present various different scripts (i.e. processes) just read the
values using the I2C bus whenever they need to but I'm pretty sure
this (quite rarely) results in false readings because two processes
try to read at the same time.

Thus I'm looking for ways to prevent simultaneous access.

One fairly obvious way is to have single process/script which reads
the A2D values continuously and writes them to a file. All other
scripts then read from the file as needed, a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).

Is this the simplest approach? Are there better ways using
multiprocess? (They look more complicated though).

The I2C bus itself has a mutex but I don't think this guarantees that
(for example) an A2D reading is atomic because one reading takes more
than one I2C bus access.

Would a mutex of some sort around each I2C transaction (i.e. complete
A2D reading) be a better way to go?

[Stefan Ram]

Post by Chris Green
Would a mutex of some sort around each I2C transaction (i.e. complete
A2D reading) be a better way to go?

I not an expert here, but from what I heard, you could use
a named POSIX semaphore as a kind of system-wide mutex.

Or, the single process reading the A2D values could write
them to a shared memory segment the access to which is
controlled by a mutex.

Or, you could use a message queue (like POSIX mq) to
distribute readings.

But why overengineer? If you feel comfortable with the file
solution, go for it! The only drawback might be that it's a
bit slower than other approaches.

[Gordinator]

Post by Stefan Ram
But why overengineer? If you feel comfortable with the file
solution, go for it! The only drawback might be that it's a
bit slower than other approaches.

I absolutely agree. Overengineering is generally a bad idea because
you're using a complex solution to solve a simple problem, which leads
to unexpected breakage.

The file solution is perfectly fine, and file locks are used by really
important software (i.e package managers and such) because its
simplicity makes it almost never fail.

So yeah, go for it!

[Piergiorgio Sartor]

Post by Chris Green
I have a Raspberry Pi in my boat that uses I2C to read a number of
voltages and currents (using ADS1115 A2D) so I can monitor the battery
condition etc.
At present various different scripts (i.e. processes) just read the
values using the I2C bus whenever they need to but I'm pretty sure
this (quite rarely) results in false readings because two processes
try to read at the same time.
Thus I'm looking for ways to prevent simultaneous access.

Why using "different scripts"?
Is it there any particular reason?

Maybe it would be better, if possible, to have
a single script, which, sequentially, reads
whatever needs to be read (or written).
In a loop.

This is even simpler than using a file.

bye,

pg

Post by Chris Green
One fairly obvious way is to have single process/script which reads
the A2D values continuously and writes them to a file. All other
scripts then read from the file as needed, a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).
Is this the simplest approach? Are there better ways using
multiprocess? (They look more complicated though).
The I2C bus itself has a mutex but I don't think this guarantees that
(for example) an A2D reading is atomic because one reading takes more
than one I2C bus access.
Would a mutex of some sort around each I2C transaction (i.e. complete
A2D reading) be a better way to go?

[Chris Green]

Post by Piergiorgio Sartor

Post by Chris Green
I have a Raspberry Pi in my boat that uses I2C to read a number of
voltages and currents (using ADS1115 A2D) so I can monitor the battery
condition etc.
At present various different scripts (i.e. processes) just read the
values using the I2C bus whenever they need to but I'm pretty sure
this (quite rarely) results in false readings because two processes
try to read at the same time.
Thus I'm looking for ways to prevent simultaneous access.

Why using "different scripts"?
Is it there any particular reason?
Maybe it would be better, if possible, to have
a single script, which, sequentially, reads
whatever needs to be read (or written).
In a loop.
This is even simpler than using a file.

Yes, but it's conceptually (and programming wise) much simpler to have
separate scripts. Some of them are simple 'on demand' scripts that I
run from the command line when I want to know something. Others are
scripts that drive displays on control panels.

[Lawrence D'Oliveiro]

One fairly obvious way is to have single process/script which reads the
A2D values continuously and writes them to a file. All other scripts
then read from the file as needed, a simple file lock can then be used
to prevent simultaneous access (well, simultaneous access when the
writing process is writing).

The thing with a file is, it persists even when the collector process is
not running. Do you want data that persists when the collector process is
not running?

Is this a history of values, or just a snapshot of current values? A
history of values could be written to a database. Databases provide their
own transactions and interlocking to prevent readers from reading partial
updates.

If it’s a snapshot of current values, that does not persist when the
collector process is not running, then why not just keep the data in the
memory of the collector process, and have it concurrently listen on a
socket for connections from readers requesting a copy of the current data?

[Chris Green]

Post by Lawrence D'Oliveiro

One fairly obvious way is to have single process/script which reads the
A2D values continuously and writes them to a file. All other scripts
then read from the file as needed, a simple file lock can then be used
to prevent simultaneous access (well, simultaneous access when the
writing process is writing).

The thing with a file is, it persists even when the collector process is
not running. Do you want data that persists when the collector process is
not running?
Is this a history of values, or just a snapshot of current values? A
history of values could be written to a database. Databases provide their
own transactions and interlocking to prevent readers from reading partial
updates.

There's a separate (crontab driven) process that writes the history to
a sqlite3 database,

Post by Lawrence D'Oliveiro
If it’s a snapshot of current values, that does not persist when the
collector process is not running, then why not just keep the data in the
memory of the collector process, and have it concurrently listen on a
socket for connections from readers requesting a copy of the current data?

That's exactly the sort of solution I was wondering about. Is there a
ready made module/library for handling this sort of thing? Basically
it will just be a string of a few tens of characters that would be
kept up to date by one process and asked for by all the others.

[Stefan Ram]

Post by Chris Green
That's exactly the sort of solution I was wondering about. Is there a
ready made module/library for handling this sort of thing? Basically
it will just be a string of a few tens of characters that would be
kept up to date by one process and asked for by all the others.

I'm not an expert here, and just quickly tried to make it
run, so the code will still contain errors and not contain
something necessary, but might give you a starting point.

A process doing something (here: printing an incrementing value
named "info") and also serving requests from other processes
for this "info" value:

import asyncio
import socket

class Server:
def __init__( self ):
self.info = 0

async def handle_client( self, reader, writer ):
data = await reader.read( 100 )
message = data.decode()
addr = writer.get_extra_info( 'peername' )
print( f"Received {message!r} from {addr}" )

if message.strip() == "getinfo":
writer.write( str( self.info ).encode() )
await writer.drain()

writer.close()
await writer.wait_closed()

async def print_number( self ):
while True:
print( self.info )
self.info += 1
await asyncio.sleep( 1 )

async def serve( self ):
server = await asyncio.start_server( self.handle_client, '127.0.0.1', 8888 )
addr = server.sockets[0].getsockname()
print( f'Serving on {addr}' )

async with server:
await asyncio.gather( server.serve_forever(), self.print_number() )

asyncio.run( Server().serve() )

, and an example client:

import socket
import time

while True:
s = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
s.connect( ( '127.0.0.1', 8888 ))
s.send( "getinfo".encode() )
data = s.recv( 100 )
message = data.decode().strip()
print( f"Received: {message}" )
time.sleep( 3 )

.

[Chris Green]

Post by Stefan Ram

Post by Chris Green
That's exactly the sort of solution I was wondering about. Is there a
ready made module/library for handling this sort of thing? Basically
it will just be a string of a few tens of characters that would be
kept up to date by one process and asked for by all the others.

I'm not an expert here, and just quickly tried to make it
run, so the code will still contain errors and not contain
something necessary, but might give you a starting point.
A process doing something (here: printing an incrementing value
named "info") and also serving requests from other processes

[snip]

Thanks, that should get me started! :-)

[Lawrence D'Oliveiro]

Post by Chris Green

Post by Lawrence D'Oliveiro
If it’s a snapshot of current values, that does not persist when the
collector process is not running, then why not just keep the data in
the memory of the collector process, and have it concurrently listen on
a socket for connections from readers requesting a copy of the current
data?

That's exactly the sort of solution I was wondering about. Is there a
ready made module/library for handling this sort of thing? Basically it
will just be a string of a few tens of characters that would be kept up
to date by one process and asked for by all the others.

You probably don’t need to define any nontrivial kind of protocol at all.
Just accept a connection from a client, send it the current data in some
convenient format (e.g. JSON), and disconnect.

[Barry]

Post by Chris Green
a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).

There is a simple pattern to make this robust.

Write new values to a tmp file.
Close the tmp file.
Then use os.rename(tmpfile, productionfile).

This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents, never a mix of both.

Barry

[MRAB]

Post by Barry

Post by Chris Green
a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).

There is a simple pattern to make this robust.
Write new values to a tmp file.
Close the tmp file.
Then use os.rename(tmpfile, productionfile).
This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents, never a mix of both.

For clarity I'd recommend os.replace instead. This is because on Windows
os.rename it would complain if the target file already exists, but
os.replace has the same behaviour on both Linux and Windows.

[Barry Scott]

For clarity I'd recommend os.replace instead. This is because on Windows os.rename it would complain if the target file already exists, but os.replace has the same behaviour on both Linux and Windows.

Agreed.

In this case the OP is on an RPi.

Barry

[Left Right]

If resource usage isn't an issue, then the _easy_ thing to do, that
would also be easily correct is to have a server doing all the
h/w-related reading and clients talking to that server. Use for the
server the technology you feel most confident with. Eg. you may use
Python's http package. I believe that the server from this package
runs in a single thread, and thus processes all requests
synchronously. So, you'll get synchronization for free.

Then, the rest of the scripts that need to talk to h/w will instead be
talking to this server.

Again, this isn't an _efficient_ solution... but, sometimes you don't
need one. And this one is easy to make, easy to debug, easy to expand.
But, if instead you were looking for a more efficient solution, then,
the general idea that allows the http server to work in this case
would still apply: have a single synchronization program that takes
requests asynchronously, and orders them. So, a basic TCP server would
also work as well as a UNIX socket. Your idea with holding a lock on a
file would also work (in fact, plenty of Linux utilities work that
way, eg. apt-get or yum).

----

If you don't want to change the existing script, then instead of
running them directly, you could run them through batch:
https://man7.org/linux/man-pages/man1/batch.1p.html this is a very
simply queuing program that's available for Linux. It will take care
of synchronization by putting the scripts you want to run in a queue
and executing them one at a time.

On Sun, Jul 7, 2024 at 11:12 PM Chris Green via Python-list

Post by Chris Green
I have a Raspberry Pi in my boat that uses I2C to read a number of
voltages and currents (using ADS1115 A2D) so I can monitor the battery
condition etc.
At present various different scripts (i.e. processes) just read the
values using the I2C bus whenever they need to but I'm pretty sure
this (quite rarely) results in false readings because two processes
try to read at the same time.
Thus I'm looking for ways to prevent simultaneous access.
One fairly obvious way is to have single process/script which reads
the A2D values continuously and writes them to a file. All other
scripts then read from the file as needed, a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).
Is this the simplest approach? Are there better ways using
multiprocess? (They look more complicated though).
The I2C bus itself has a mutex but I don't think this guarantees that
(for example) an A2D reading is atomic because one reading takes more
than one I2C bus access.
Would a mutex of some sort around each I2C transaction (i.e. complete
A2D reading) be a better way to go?
--
Chris Green
·
--
https://mail.python.org/mailman/listinfo/python-list