Python virtual environments

Maude Le Jeune Computer science crash course, November 2021

Outline

  1. Why using a python virtual enviroment

  2. First steps with venv

  3. Overview of existing tools

  4. Changing python version with pyenv

  5. Advanced usage and references

1. Why using a python virtual environment

Some use cases

Python virtual environment like conda, pyenv or virtualenv, are made to ease the following common issues:

  • I need this new fancy package, but it requires numpy > 1.14 and I’ve 1.12. Let’s update numpy. But wait, is this going to break the other tools which were working with 1.12 ?

  • This supercomputer that I’m using now provides a more recent python version, how can I switch to it without breaking everything.

  • I need to test this old package which is only available with python2… How to work with both Python 2 and 3 while upgrading this tool ?

Why are those situations an issue

  • Python is installed by default with any Linux/Free BSD OS.

$ which python
/usr/bin/python
$ python --version
Python 3.9.1
  • Python third-party packages have a default, common location defined by the system

import sys
print(sys.path)
['', '/usr/lib/python39.zip', '/usr/lib/python3.9', 
 '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', 
 '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
  • You can use the Python version installed with your system and complete those locations with your favorites packages.

  • Issues arrive when it comes to move from one version to another or use different versions at a time.

Tip

The main idea behind python virtual environment is to define a different location for each python/package installation you may need to use, and easily switch from one to another.

2. First steps with venv

Create a virtual environment

We are going to create a virtual environment with venv, and name it after the project (software or software suite) that will use it.

The environement is linked to a specific python installation, we’re going to use the one already installed with your system.

$ cd my-project-dir
$ python -m venv venv
$ ls venv
bin  include  lib  lib64  pyvenv.cfg  share

To use the environment, one needs to activate it. This can be done automatically by adding the following line to in your .bashrc file.

$ source my-project-dir/venv/bin/activate
(venv)$ which python
my-project-dir/venv/bin/python

Let’s have a look at the new PYTHONPATH.

import sys
print(sys.path)
['', '/usr/lib/python3/dist-packages', '/usr/lib/python3', '/usr/lib/python3.9/dist-packages', '/usr/lib/python3.9', '/usr/lib/python39.zip', '/usr/lib/python3.9/lib-dynload', '/home/me/my-project-dir/venv/lib/python3.9/site-packages']

Install some new packages within the environement

We use pip to install new packages.

(venv>$ python -m ensurepip
(venv)$ python -m pip install astropy
Collecting astropy
  Downloading astropy-4.3.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (10.8 MB)
     |████████████████████████████████| 10.8 MB 1.7 MB/s 
Collecting pyerfa>=1.7.3
  Using cached pyerfa-2.0.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (742 kB)
Requirement already satisfied: numpy>=1.17 in /usr/lib/python3/dist-packages (from astropy) (1.19.4)
Installing collected packages: pyerfa, astropy
Successfully installed astropy-4.3.1 pyerfa-2.0.0.1

Let’s check where it has been installed:

(venv)$ ls my-project-dir/venv/lib/python3.9/site-packages
astropy                  erfa                  pkg_resources                  pyerfa-2.0.0.1.dist-info
astropy-4.3.1.dist-info  pip                   pkg_resources-0.0.0.dist-info  setuptools
easy_install.py          pip-20.1.1.dist-info  __pycache__                    setuptools-44.0.0.dist-info

Or from python:

import astropy
print(astropy.__file__)
'/home/me/my-project-dir/venv/lib/python3.9/site-packages/astropy/__init__.py'

Give a try to a new numpy version

First, we’re going to save the current environement in a requirements.txt file.

(venv)$ pip freeze > requirements.txt 
(venv)$ more requirements.txt
astropy==4.3.1
numpy==1.19.4
...

Then, we deactivate the current virtual environement and create new one:

(venv)$ deactivate
$ python -m venv venv-newnp
$ source venv-newnp/bin/activate
(venv-newnp)$ pip install -r requirements.txt 
Collecting astropy==4.3.1
  Using cached astropy-4.3.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (10.8 MB)
Installing collected packages: astropy
Successfully installed astropy-4.3.1 

Now, we’re goig to upgrade the numpy package.

(venv-newnp)$ python -m pip install numpy
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.19.4)
(venv-newnp)$ python -m pip install numpy==1.21
Collecting numpy==1.21
  Downloading numpy-1.21.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.4
    Not uninstalling numpy at /usr/lib/python3/dist-packages, outside environment /home/me/my-project-dir/venv
    Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed numpy-1.21.0

Let’s check where it has been installed:

(venv-newnp)$ ls my-project-dir/venv-newnp/lib/python3.9/site-packages
astropy                  erfa                    numpy.libs            pkg_resources                  pyerfa-2.0.0.1.dist-info
astropy-4.3.1.dist-info  numpy                   pip                   pkg_resources-0.0.0.dist-info  setuptools
easy_install.py          numpy-1.21.4.dist-info  pip-20.1.1.dist-info  __pycache__                    setuptools-44.0.0.dist-info
import numpy
print(numpy.__version)
'1.21.0'

After a while, if this new version fits your needs, just replace the two venv folders.

$ mv venv-newnp venv

3. Overview of existing tools

virtualenv/venv

pyenv

conda

switch across package versions

switch across Python versions

manage Python and its dependencies + packages

If not already installed, one needs to get pip to install new Python packages, except for conda which comes with it’s own package management system and repository.

Since Python 3.3: virtualenv is built-in as venv, but contains only a subset of it. In particular it doesn’t offer full support for Python 2. It installs pip by default.

One can combine pyenv with virtualenv/venv to easily switch from different python versions and package versions.

Conda requires ~3Gb, but one can use miniconda (400Mb) instead.

4. Changing python version with pyenv

If you need to try a new python version, which is not installed on your system, then pyenv is the right tool to use.

Install pyenv

curl https://pyenv.run | bash
pyenv install --list

Available versions:
  2.1.3
  2.2.3
  2.3.7
  ...

Choose one version and install it.

$ pyenv install -v 3.7.2
Installed Python-3.7.2 to /home/me/.pyenv/versions/3.7.2
$ pyenv versions

 * system (set by /home/me/.pyenv/version)
  3.7.2

Set this new version as default

$ pyenv global 3.7.2
$ pyenv which python3
/home/me/.pyenv/versions/3.7.2/bin/python3

Create a new virtualenv

$ python -m venv venv372
$ source venv372/bin/activate

To get back to default python version, type:

(venv372)$ deactivate
$ pyenv global system