Strategies for avoiding having to use --break-system-packages with pip

[Chris Green]

I have a (relatively) clean Debian 12 installation running on my two
workhorse systems, a desktop server at home and my laptop that travels
around with me.

I moved from Xubuntu to Debian on both these systems a few months ago.

I ran Xubuntu for many years and acquired a whole lot of python
packages installed with pip, as root. For the last couple of years I
had to use the --break-system-packages option to get things installed.

As far as I'm aware I never hit any dependency problems doing this.
It's probably because things I installed with pip were mostly quite
small, specialised, packages that I used in just one or two utility
programs that I had written myself. In quite a few cases these were
realated to image processing and such things.


So far I've managed to keep my Debian 12 installations 'pip free', I
haven't even got pip installed. However I may have just come across
something that would at least be very useful and it comes from PyPi.
(It's tkintertable if that's of any interest or relevance)


What are my options?

Just install it using pip as root and --break-system-packages,
what's likely to break?

Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.

Download tkintertable from git into my development environment and
use that. My PYTHONPATH will need to point to it but I can't see
any further issues with doing this.

Anything else? As far as I can see using pipx doesn't help me at
all (see recent thread here).

[Stefan Ram]

Post by Chris Green
What are my options?

You've got a few ways to skin this cat without turning your Debian
setup into a dumpster fire.

Virtual environment route:

This is your best bet for keeping things kosher. Here's the lowdown:

bash

python3 -m venv ~/myenv
source ~/myenv/bin/activate
pip install tkintertable

To make your program run smooth as butter, whip up a shell script
that fires up the virtual environment and kicks off your program:

bash

#!/bin/bash
source ~/myenv/bin/activate
python /path/to/your/program.py

Slap that bad boy in your PATH, make it executable, and you're
golden.

Source installation:

Pulling tkintertable from git and tweaking your PYTHONPATH is
solid. It's like growing your own organic produce - more work,
but you know what you're getting.

Pip with --break-system-packages:

It's like jaywalking - you might get away with it, but one day
you could end up in a world of hurt.

DIY Debian package:

For the overachievers out there. It's like building your
own surfboard - cool if you can pull it off, but not for
the faint of heart.

Bottom line:

The virtual environment play (option 1) is your ticket to ride.
It keeps your system clean as a whistle while letting you run
your program without breaking a sweat.

[Mats Wichmann]

Post by Chris Green
I have a (relatively) clean Debian 12 installation running on my two
workhorse systems, a desktop server at home and my laptop that travels
around with me.
I moved from Xubuntu to Debian on both these systems a few months ago.
I ran Xubuntu for many years and acquired a whole lot of python
packages installed with pip, as root. For the last couple of years I
had to use the --break-system-packages option to get things installed.
As far as I'm aware I never hit any dependency problems doing this.
It's probably because things I installed with pip were mostly quite
small, specialised, packages that I used in just one or two utility
programs that I had written myself. In quite a few cases these were
realated to image processing and such things.
So far I've managed to keep my Debian 12 installations 'pip free', I
haven't even got pip installed. However I may have just come across
something that would at least be very useful and it comes from PyPi.
(It's tkintertable if that's of any interest or relevance)
What are my options?
Just install it using pip as root and --break-system-packages,
what's likely to break?
Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.
Download tkintertable from git into my development environment and
use that. My PYTHONPATH will need to point to it but I can't see
any further issues with doing this.
Anything else? As far as I can see using pipx doesn't help me at
all (see recent thread here).

You might look at uv, which makes the managing of a virtualenv for your
program pretty transparent. You declare your dependencies, and then just:

uv run myscript.py

And of course if you don't want to type three words to launch, you can
put those in an executable shell script in your path, and invoke it with
a single command.

There was a nice article on this somewhere which I now can't find, but
the project docs cover the topics pretty well:

https://docs.astral.sh/uv/guides/scripts

[c***@posteo.jp]

Hello Chris,

I do have similar "problems" and still try to get used to the "new way".

Other might correct me. I am not sure yet.

To my current understanding the way to go is to install Python
applications via "pipx". That make the application available in your
system but also isolate it in its own virtual environment. Of course you
should prefer to install applications from your GNU/Linux distros
official repository if available.

If you install a Python package (library, not an application) you should
create your own Python environment via venv for example. Pipx is not
intended to install Python packages that are not applications.

Regrads,
Christian

[Thomas Passin]

Post by Chris Green
I have a (relatively) clean Debian 12 installation running on my two
workhorse systems, a desktop server at home and my laptop that travels
around with me.
I moved from Xubuntu to Debian on both these systems a few months ago.
I ran Xubuntu for many years and acquired a whole lot of python
packages installed with pip, as root. For the last couple of years I
had to use the --break-system-packages option to get things installed.
As far as I'm aware I never hit any dependency problems doing this.
It's probably because things I installed with pip were mostly quite
small, specialised, packages that I used in just one or two utility
programs that I had written myself. In quite a few cases these were
realated to image processing and such things.
So far I've managed to keep my Debian 12 installations 'pip free', I
haven't even got pip installed. However I may have just come across
something that would at least be very useful and it comes from PyPi.
(It's tkintertable if that's of any interest or relevance)
What are my options?
Just install it using pip as root and --break-system-packages,
what's likely to break?

You can also install with --user and --break-system-packages, but that
doesn't really solve the problem. Also, as just happened to me, if an
upgrade happens to change the system's python to a newer version (e.g.,
3.12.x to 3.13.y), you would have to install all your packages again
with the new Python install.

Post by Chris Green
Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.

You can write a shell script that activates the venv and then launches
your program. This works pretty well.

Post by Chris Green
Download tkintertable from git into my development environment and
use that. My PYTHONPATH will need to point to it but I can't see
any further issues with doing this.

This is an approach I use sometimes, mainly if I have cloned a project.
I run the project's program(s) using a script that sets the PYTHONPATH.

Post by Chris Green
Anything else? As far as I can see using pipx doesn't help me at
all (see recent thread here).

To completely avoid problems when the system's Python install gets
changed, you can install your own Python version outside of the package
manager; it doesn't have to be the same version as the system's. I've
done this when I wanted to run a later version of Python than the
system's. You would have to take care of updating it yourself since the
package manager won't know about it. On the system where I did this, I
ran the system's python version using "python3" and my own using
"python3.11" (I think the system was still on 3.8 or 3.9).

[Left Right]

I wouldn't trust pip to install anything into my system. It's not a
reliable program that I'd recommend anyone to use for things that they
might depend on.

My typical course of action is to create a virtual environment for the
package I need. Install the package into that virtual environment
using pip or w/e the package wants to be installed with. Investigate
and refine the dependencies I need (It's very common in the Python
world to incorrectly specify dependencies, to require a lot of
unnecessary dependencies, to depend on packages in the wrong way). And
after I've figured out what exactly I need to get the package to work,
copy the dependencies together with the package to the platlib
directory of Python I'm using for this task.

If platlib happens to be in the system directories or anywhere else: I
wouldn't care about that. In the process of installing the program, I
would've learned about the nature and the layout of dependencies, so
that I could make informed decisions about what goes where and whether
anything is needed or is in danger of being affected by system updates
or endangers system updates.

So far, this has worked every time. This worked for me personally,
for small companies that do internal deployments and for big companies
that distribute Python together with the entire Linux distribution in
this way. The key ingredient is to know what you are doing and to be
able to anticipate the bad things. Tools like pip take away from the
users the need to know what they are doing in hopes of reducing
complexity on the part of the users' knowledge necessary to accomplish
program installation. However, there's no free lunch, and tools like
pip bring an extra layer of complexity in trying to do what they
claim. This layer of complexity is usually revealed when these tools
fail to do what they claim, and users are forced to learn what they
actually need to know and, on top of that, how to troubleshoot
programs like pip.

From working in infra / automation, I get the knowledge about program

/ package installation "free" on the job, so, I don't see a point in
using a tool that automates that for me beyond what I already
described. I'm probably also biased because of that, but I still think
that learning to do the thing is more important than learning how to
use the tool that does the thing.

On Tue, Jan 14, 2025 at 4:42 PM Chris Green via Python-list

I have a (relatively) clean Debian 12 installation running on my two
workhorse systems, a desktop server at home and my laptop that travels
around with me.
I moved from Xubuntu to Debian on both these systems a few months ago.
I ran Xubuntu for many years and acquired a whole lot of python
packages installed with pip, as root. For the last couple of years I
had to use the --break-system-packages option to get things installed.
As far as I'm aware I never hit any dependency problems doing this.
It's probably because things I installed with pip were mostly quite
small, specialised, packages that I used in just one or two utility
programs that I had written myself. In quite a few cases these were
realated to image processing and such things.
So far I've managed to keep my Debian 12 installations 'pip free', I
haven't even got pip installed. However I may have just come across
something that would at least be very useful and it comes from PyPi.
(It's tkintertable if that's of any interest or relevance)
What are my options?
Just install it using pip as root and --break-system-packages,
what's likely to break?
Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.
Download tkintertable from git into my development environment and
use that. My PYTHONPATH will need to point to it but I can't see
any further issues with doing this.
Anything else? As far as I can see using pipx doesn't help me at
all (see recent thread here).
--
Chris Green
·
--
https://mail.python.org/mailman/listinfo/python-list

[Peter J. Holzer]

Post by Chris Green
Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.

Just use the python interpreter in the venv in the hashbang line.

For example, here's the first line of one my scripts:

#!/usr/local/share/wds/venv/bin/python3

As you can probably guess, the venv is in /usr/local/share/wds/venv.

There is no need for wrapper scripts which activate the venv. Python
does that all by itself.

I have a small script, install-python[1], to assist with setting the
hashbang, but if it's just a few scripts you can simply edit it manually.

hp

[1] https://git.hjp.at:3000/hjp/install-python/src/branch/master/install-python

[Chris Green]

[-- text/plain, encoding quoted-printable, charset: us-ascii, 32 lines --]

Post by Chris Green
Use a virtual environment, what do I have to do then to make using
my program (that uses tkintertable) 'transparent', i.e. I just
want to be able to run the program from the command prompt like
any other program.

Just use the python interpreter in the venv in the hashbang line.
#!/usr/local/share/wds/venv/bin/python3
As you can probably guess, the venv is in /usr/local/share/wds/venv.
There is no need for wrapper scripts which activate the venv. Python
does that all by itself.
I have a small script, install-python[1], to assist with setting the
hashbang, but if it's just a few scripts you can simply edit it manually.

OP here. Yes, thank you, I thought that the shebang line would do the
trick but it's nioce to have it confirmed.