How to Create a Python Virtual Environment By Hand

A virtual environment is a simple thing. No, really - since PEP405 made them much simpler, it’s possible to create a lightweight environment by hand without any python tooling involved.

All that’s needed is a symlink to a python binary, a pyvenv.cfg file and a site-packages directory. Everything else is icing on the cake.

I thought this would be a fun exercise for those who are interested and it serves as a great way of understanding what’s going on under the hood of your favourite Python virtual environment manager.

Creating Our Environment

First we need to know where our current version of Python exists. For most, it’s likely which python will be enough to return this. I’m using pyenv which will mean a shim is returned. Pyenv stores binaries under ~/.pyenv/versions so I can find a version there (there’s also pyenv which python but this would break my Python tooling rule).

Having done this I know my default Python version is at /Users/ian/.pyenv/versions/3.11.2/bin/python.

Next, let’s create some directories.

mkdir -p venv/bin
mkdir -p venv/lib/python3.11/site-packages 

In the bin directory, we need to symlink to our system python like so:

ln -s /Users/ian/.pyenv/versions/3.11.2/bin/python venv/bin/python

We also need a pyvenv.cfg file, which we’ll put in ./venv. It’s contents should look something like below, updated with your own Python version.

home = /Users/ian/.pyenv/versions/3.11.2/bin
include-system-site-packages = false
version = 3.11.2

Strictly the only necessary key here is home, which indicates where we’re going to load our Pythons standard library from. By default Python assumes you want to use the global system site packages too, unless you specify otherwise as I have done here.

The folder structure we’ve created mimics a standard Python installation. When Python 3.3+ encounters the pyvenv.cfg file in a directory above either a python binary or a symlink to one it knows what it’s working with is a virtual environment. It then knows that the site-packages folder created should be where installed packages go.

How Do I Install Packages?

Congrats, we’ve created a virtual environment as defined by the spec. But, we probably also want something to enable us to install packages. We can do that by taking a copy of our system pip and putting it directly into ./venv/bin. We’ll also need to copy it’s package folder to our environments site-packages for it to work.

cp /Users/ian/.pyenv/versions/3.11.2/bin/pip3.11 ./venv/bin/pip
cp -R /Users/ian/.pyenv/versions/3.11.2/lib/python3.11/site-packages/pip ./venv/lib/python3.11/site-packages

The first line of our pip file in ./venv/bin also needs to be updated to point to our symlinked Python. So in my case:

#!/Users/ian/dev/tutorials/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
...

We also need to tell our shell to default to using our env versions of Python and pip. We can do that by making sure our bin directory is the first thing on our PATH environment variable.

PATH=/Users/ian/dev/tutorials/venv/bin:$PATH

Having done that my python executable and site-packages should now point to my virtualenv. Let’s check to be sure.

python -c 'import sys; print(sys.executable)'
...
/Users/ian/dev/tutorials/venv/bin/python

python -c 'import site; print(site.getsitepackages())'
...
['/Users/ian/dev/tutorials/venv/lib/python3.11/site-packages']

So having executed all of the above when calling pip we’ll now install into our local site packages directory. Let’s try that out.

pip install requests

If you’ve followed all the above commands correctly, you should find that requests ends up in our virtual environments site-packages folder instead of your global python.

Final Thoughts

This definitely isn’t a replacement for any existing environment creation tools. We don’t have any shell scripts for instance that will modify paths and update with the environment name when we move between each environment. We also didn’t copy any build or setuptools so we won’t be able to publish our own packages to the Python Package Index (PyPI). The fact is that all of the above can be achieved with a simple call to the venv module.

I always thought there was a lot of complexity to how virtual environments worked and maybe that was true when you needed to use external packages to create them but it’s not any more. Hopefully this article demystified how they work a little for you too.

NB. I’m in the process of writing an ebook on virtual environments, subscribe to my newsletter if you’re interested in hearing about it when it launches