Python virtual environments#
Maude Le Jeune Computer science crash course, November 2021
Outline#
Why using a python virtual environment
First steps with venv
Overview of existing tools
Changing python version with pyenv
Advanced usage and references
1. Why using a python virtual environment#
Some use cases#
Python virtual environment like conda, pyenv or virtualenv, are made to ease the following common issues:
I need this new fancy package, but it requires numpy > 1.14 and I’ve 1.12. Let’s update numpy. But wait, is this going to break the other tools which were working with 1.12 ?
This supercomputer that I’m using now provides a more recent python version, how can I switch to it without breaking everything.
I need to test this old package which is only available with python2… How to work with both Python 2 and 3 while upgrading this tool ?
Why are those situations an issue#
Python is installed by default with any Linux/Free BSD OS.
$ which python
/usr/bin/python
$ python --version
Python 3.9.1
Python third-party packages have a default, common location defined by the system
import sys
print(sys.path)
['', '/usr/lib/python39.zip', '/usr/lib/python3.9',
'/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages',
'/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']
You can use the Python version installed with your system and complete those locations with your favorites packages.
Issues arrive when it comes to move from one version to another or use different versions at a time.
Tip
The main idea behind python virtual environment is to define a different location for each python/package installation you may need to use, and easily switch from one to another.
2. First steps with venv#
Create a virtual environment#
We are going to create a virtual environment with venv, and name it
after the project (software or software suite) that will use it.
The environment is linked to a specific python installation, we’re going to use the one already installed with your system.
$ cd my-project-dir
$ python -m venv venv
$ ls venv
bin include lib lib64 pyvenv.cfg share
To use the environment, one needs to activate it. This can be done
automatically by adding the following line to in your .bashrc file.
$ source my-project-dir/venv/bin/activate
(venv)$ which python
my-project-dir/venv/bin/python
Let’s have a look at the new PYTHONPATH.
import sys
print(sys.path)
['', '/usr/lib/python3/dist-packages', '/usr/lib/python3', '/usr/lib/python3.9/dist-packages', '/usr/lib/python3.9', '/usr/lib/python39.zip', '/usr/lib/python3.9/lib-dynload', '/home/me/my-project-dir/venv/lib/python3.9/site-packages']
Install some new packages within the environment#
We use pip to install new packages.
(venv)$ python -m ensurepip
(venv)$ python -m pip install astropy
Collecting astropy
Downloading astropy-4.3.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (10.8 MB)
|████████████████████████████████| 10.8 MB 1.7 MB/s
Collecting pyerfa>=1.7.3
Using cached pyerfa-2.0.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (742 kB)
Requirement already satisfied: numpy>=1.17 in /usr/lib/python3/dist-packages (from astropy) (1.19.4)
Installing collected packages: pyerfa, astropy
Successfully installed astropy-4.3.1 pyerfa-2.0.0.1
Let’s check where it has been installed:
(venv)$ ls my-project-dir/venv/lib/python3.9/site-packages
astropy erfa pkg_resources pyerfa-2.0.0.1.dist-info
astropy-4.3.1.dist-info pip pkg_resources-0.0.0.dist-info setuptools
easy_install.py pip-20.1.1.dist-info __pycache__ setuptools-44.0.0.dist-info
Or from python:
import astropy
print(astropy.__file__)
'/home/me/my-project-dir/venv/lib/python3.9/site-packages/astropy/__init__.py'
Give a try to a new numpy version#
First, we’re going to save the current environment in a requirements.txt file.
(venv)$ pip freeze > requirements.txt
(venv)$ more requirements.txt
astropy==4.3.1
numpy==1.19.4
...
Then, we deactivate the current virtual environment and create new one:
(venv)$ deactivate
$ python -m venv venv-newnp
$ source venv-newnp/bin/activate
(venv-newnp)$ pip install -r requirements.txt
Collecting astropy==4.3.1
Using cached astropy-4.3.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (10.8 MB)
Installing collected packages: astropy
Successfully installed astropy-4.3.1
Now, we’re going to upgrade the numpy package.
(venv-newnp)$ python -m pip install numpy
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.19.4)
(venv-newnp)$ python -m pip install numpy==1.21
Collecting numpy==1.21
Downloading numpy-1.21.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.19.4
Not uninstalling numpy at /usr/lib/python3/dist-packages, outside environment /home/me/my-project-dir/venv
Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed numpy-1.21.0
Let’s check where it has been installed:
(venv-newnp)$ ls my-project-dir/venv-newnp/lib/python3.9/site-packages
astropy erfa numpy.libs pkg_resources pyerfa-2.0.0.1.dist-info
astropy-4.3.1.dist-info numpy pip pkg_resources-0.0.0.dist-info setuptools
easy_install.py numpy-1.21.4.dist-info pip-20.1.1.dist-info __pycache__ setuptools-44.0.0.dist-info
import numpy
print(numpy.__version)
'1.21.0'
After a while, if this new version fits your needs, just replace the
two venv folders.
$ mv venv-newnp venv
3. Overview of existing tools#
virtualenv/venv |
pyenv |
conda/mamba |
|---|---|---|
switch across package versions |
switch across Python versions |
manage Python and its dependencies + packages |
If not already installed, one needs to get pip to install new Python packages, except for conda which comes with it’s own package management system and repository.
Since Python 3.3: virtualenv is built-in as venv, but contains only a subset of it. In particular it doesn’t offer full support for Python 2. It installs pip by default.
One can combine pyenv with virtualenv/venv to easily switch
from different python versions and package versions (see below).
Conda requires ~3Gb, but one can use miniconda (400Mb) instead.
4. Changing python version with pyenv#
If you need to try a new python version, which is not installed on
your system, then pyenv is the right tool to use.
Install pyenv#
curl https://pyenv.run | bash
pyenv install --list
Available versions:
2.1.3
2.2.3
2.3.7
...
Choose one version and install it.
$ pyenv install -v 3.7.2
Installed Python-3.7.2 to /home/me/.pyenv/versions/3.7.2
$ pyenv versions
* system (set by /home/me/.pyenv/version)
3.7.2
Set this new version as default
$ pyenv global 3.7.2
$ pyenv which python3
/home/me/.pyenv/versions/3.7.2/bin/python3
Combine with virtualenv using the pyenv-virtualenv plugin#
The pyenv-virtualenv
plugin allows you to manage virtual environments attached to a
specific python version.
It also comes with the nice feature which allows you to automatically activate an environment when working in a specific directory.
To install this plugin:
$ git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
$ eval "$(pyenv virtualenv-init -)"
Add the virtualenv-init step to your .bashrc to automatically load it
$ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
Now building a new virtualenv with a specific python version installed
with pyenv:
$ pyenv virtualenv 3.7.2 my-virtual-env-3.7.2
You can switch from one to another with the usual command lines:
$ pyenv virtualenvs
$ pyenv activate my-virtual-env-3.7.2
$ pyenv deactivate
Or doing it automatically when entering/leaving a directory by setting
a .python-version file, which contains a single string matching an
existing virtual environment. To do, just go into a specific
directory and type:
$ pyenv local my-virtual-env-3.7.2