Python Package Pain #

Python is a wonderful programming language.

But its package management can be an absolute pain.

Its so bad that xkcd thought it appropriate to dedicate one of its comic strips to the madness that can be the Python environment.

Fortunately, virtual environments – when used appropriately and consistently – do alleviate some of that pain.

What are Virtual Environments? #

Virtual environments create contained spaces for isolated projects. Within those isolated spaces, particular versions of Python and packages can be installed and run without affecting other environments.

Using virtual environments helps segregate the project environment from the global environment. So even if a particular version of a package is installed at the global level (i.e., machine or user level) for instance, something else can be installed at the local level.

This is especially handy when individual projects depend on particular package versions. It’s also a good way to safely play around with development versions of Python packages.

Pairing virtual environments with requirements.txt really makes the whole Python experience shine.

requirements.txt streamlines the process for spinning up a virtual environment with all the right packages and versions installed within that given environment.

venv #

venv is the default virtual environment that comes loaded with Python 3.

Below is some concise guidance on how to use venv.

Note: By default, venv virtual environments use the same Python version that was used to create the virtual environment in the first place. So if Python 3.9 is used to create the environment, then the environment’s working version of Python will also be 3.9.

Navigate to your project directory #

Ideally, you should have a single directory that contains all the working code for your project. Go there first.

For the examples here, we’ll just call that directory project.

Create a virtual environment #

Once you’re in your project directory, it’s time to create that virtual environment.

python3 -m venv <venv>

This creates a directory named whatever you put in <venv>. Within this path are a whole bunch of subdirectories and files that maintain the virtual environment.

By convention, the virtual environment is usually created within a project directory, and the virtual environment is usually named venv.

The structure then of a project directory might look something like this:

project
project/venv

The top-level view of a fresh venv looks like this:

$ ls <venv>
bin        
include    
lib        
pyvenv.cfg

venv should be considered a transient directory, in the sense that you should be free to delete it (i.e., rm -rf venv) and recreate it. It should therefore not contain any working files. It should only contain files generated as a part of maintaining the virtual environment.

Because venv is transient, it should be excluded from any git tracking by adding venv/* to the .gitignore file if there is one.

Create a virtual environment with inheritance #

It is possible to create venv so that it inherits all the packages already installed in the global environment.

However, any packages installed while the virtual environment is active would only be installed within the virtual environment itself.

python3 -m venv <venv> --system-site-packages

I do find this approach to be less than ideal as it muddles the segregation between the global and virtual environments.

Activate the virtual environment #

Activating a virtual environment puts you into a contained bubble. Any Python configuration will remain entirely localized to within the local environment.

Nothing done while the virtual environment is active will affect any other environment, including the global environment.

source <venv>/bin/activate

This will prepend (venv) to your command line to remind you that you’re in a virtual environment.

(venv) $

Install packages into your virtual environment #

You can install packages individually:

pip3 install <package name>

Alternatively, you can install packages from a requirements.txt:

pip3 install -r requirements.txt

To review the set of packages you’ve installed in the virtual environment, including their dependencies and versions, run:

pip3 list

If along the way you decide you’ve curated an ideal Python package environment for a given project, you can grab a snapshot of your package arrangement:

pip3 freeze > requirements.txt

If you happened to create a virtual environment with inheritance, you can generate a requirements.txt with only the locally-installed packages with:

pip3 freeze --local > requirements.txt

Deactive the virtual environment #

Once you’re done with working on your project and you’re ready to leave the virtual environment bubble, you have to explicitly deactive the bubble.

(directory) $ deactivate

Once deactivated, any Python configurations will affect the global Python environment.

Resources #

Here’s a great video by Corey Schafer on venv.