How a Python Virtual Environment Works

Virtual environments, they’re what we use in Python to allow us to install project dependencies into neat isolated folders instead of directly into our global system Python. Once activated, dependencies go into our virtual environment folder instead of our system Python and we can use them without interfering with our global system. This makes them really lightweight environments that we can dispose of and recreate very easily.

So how do they work? We’ve got a lot of options to create them, venv module, pipenv and poetry to name a few, but whatever you use, the venv that’s created has a structure that’s mostly the same and mimics what we see in our system Python.

In this article I’m going to explore their structure and how the core of their behaviour is defined by a single PEP. If you’d prefer, you can watch the following video I created for YouTube.

Structure of a Virtual Environment

Let’s create a virtual environment now with the venv module and explore what’s created. We’re calling the venv module and passing the name argument ‘venv’.

python -m venv venv

Within the folder that is created we have a bin folder containing symlinked versions of Python, a site packages directory and we also have a pyvenv.cfg file.

PEP 405 specifies how these files work, which introduced the concept of this pyvenv.cfg file. Python is going to look for this when deciding on its path configuration and if that file exists, it looks for key value pairs within it. Here’s what pyvenv.cfg looks like.

home = /Users/ian/pyenv/versions/3.11.2/bin
include-system-site-packages = false
version = 3.11.2
executable = /Users/ian/pyenv/versions/3.11.2/bin/python3.11
command = /Users/ian/pyenv/versions/3.11.2/bin/python -m venv /Users/ian/dev/python/tutorials/how-venvs-work/venv

If it finds a “home” value, as seen here - it’s going to use the version of Python specified there as the basis for the virtual environment’s Python. So Python knows that we’re now talking about a virtual environment due to pyvenv.cfg and it’s going to be including site packages from this relative directory to this bin folder.

We’ve also created a whole bunch of activate shell scripts, within which we have the setup for virtual environment variables and also modifying our system path to include Python from our bin folder. We have a deactivate method that will become available in our shell once we execute this activate script and then we can deactivate all these environment variables with it.

There are many versions of this script for whatever shell you happen to use eg. bash, fish, PowerShell.

Python Paths for a Virtual Environment

Let’s take a look at how the PATH environment variable is configured prior to running activation for the environment.

echo $PATH

Theres a whole load of paths delimited by colons which are all things that we’re going to be able to call from the command line and Python will exist in one of these directories. Here’s mine for reference.

/Users/ian/dev/google-cloud-sdk/bin:/Users/ian/go/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/opt/homebrew/opt/openss1@1.1/bin:/opt/homebrew/opt/openjdk/bin:/Users/ian/pyenv/shims:/Users/ian/-pyenv/bin:/Users/ian/.nvm/versions/node/v19.5.0/bin:/opt/homebrew/bin:/op t/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/us/local/go/bin:/Users/ian/dev/google-clo ud-sdk/bin:/Users/ian/go/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/opt/homebrew/opt/openssla1.1/bin:/opt/homebrew/opt/openjdk/bin:/Users/ian/pyenv/bin:/Users/ian/.nvm/versions/node/v19.5.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/ian/.local/bin:/Users/ian/local/bin:/Users/1an/.local/bin

Let’s find out where Python is currently being called from too:

> which python
/Users/ian/.pyenv/shims/python

You can see on my system that I’m using pyenv to manage my version of Python and that it appears a few paths into my $PATH env var.

Importantly, things that are earlier on in this PATH string are going to get executed earlier. If you put another folder with another python in it as the first in that string, that’s going to get precedence any other Python that appears later.

We can also see with the python that we’re using which site packages directory it thinks it’s going to be using by running a short script in the command line.

> python -c 'import site; print(site.getsitepackages()[0])'
/Users/ian/.pyenv/versions/3.11.2/lib/python3.11/site-packages

I can see that I’m using site-packages within pyenv from the PATH. You may be using a global Python version on your system, so that’s where our packages will go if we use something like pip to install somethingt.

Lets call the activate script which activates it and shows its name (venv) as part of my prompt. This way I know I’m working within the venv.

> . ./venv/bin/activate
(venv) >

If we echo out the PATH again, we can see it’s now changed and the venvs bin folder at the beginning.

/Users/ian/dev/python/tutorials/how-venvs-work/venv/bin:/Users/ian/dev/google-cloud-sdk/bin:/Users/ian/go/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/opt/homebrew/opt/openss1@1.1/bin:/opt/homebrew/opt/openjdk/bin:/Users/ian/pyenv/shims:/Users/ian/-pyenv/bin:/Users/ian/.nvm/versions/node/v19.5.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/us/local/go/bin:/Users/ian/dev/google-cloud-sdk/bin:/Users/ian/go/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/opt/homebrew/opt/openssla1.1/bin:/opt/homebrew/opt/openjdk/bin:/Users/ian/pyenv/bin:/Users/ian/.nvm/versions/node/v19.5.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/ian/.local/bin:/Users/ian/local/bin:/Users/1an/.local/bin

If we also check what our site packages directory is, we can see that it’s now the one within our venv. So it’s correctly configured this so that we are using the standard library from our symlinked python but we are using site packages from the venv. This is all down to the pyvenv.cfg file.

> python -c 'import site; print(site.getsitepackages()[0])'
/Users/ian/dev/python/tutorials/how-venvs-work/ven/lib/python3.11/site-packages

Installing Packages with pip

Having a venv limited to a folder makes them easily disposable so that we can just delete it if we wanted to and we’re not going to mess with our global version of python.

Lets actually go ahead and install requests:

> (venv) pip install requests

We can see when we go into this that requests now appears in this folder and if we uninstall it, it’ll get removed from there as well.

We can also deactivate this because we now have that deactivate command and we come back out of our environment.

> (venv) deactivate
>

Our path and location for site packages will once again be set back to how they were originally as well. Notice that the prompt has updated too.

Installing Packages with Poetry

This structure is the same for any package manager - if you were to go ahead and say install with Poetry you get the same results.

First of all I’m going to set Poetry my in project variable so that we venvs installed within a directory within the folder I’m in.

poetry config virtualenvs.in-project true

If we init a project accepting defaults and install requests:

poetry init
poetry add requests

Poetry automatically creates a venv for me that looks very similar to what I created with the venv module.

We’ve can see the pyvenv.cfg file but we don’t have the include folder and now we’ve got this wheel because poetry also makes it very easy to create wheels to publish. Then we’ve got the site packages and our requests installed in that.

Final Thoughts

As you can see, most of the features that make virtual environments work are controlled with the pyvenv.cfg file I’ve shown. Many of the available package managers add features over and above the PEP405 specfication with the structures and PATH’s we’ve discussed underpinning them. I hope this article helped demystify how they work for you.

NB. I’m in the process of writing an ebook on virtual environments, subscribe to my newsletter if you’re interested in hearing about it when it launches