virtualenvs.rst

Package Installation and Project Isolation

This tutorial walks you through installing and using Python packages.

It will show you how to install and use the necessary tools and make strong recommendations on best practices. Keep in mind that Python is used for a great many different purposes, and precisely how you want to manage your dependencies may change based on how you decide to publish your software. The guidance presented here is most directly applicable to the development and deployment of network services (including web applications), but is also very well suited to managing development and testing environments for any kind of project.

Note

This guide is written for Python 3, however, these instructions should work fine on Python 2.7—if you are still using it, for some reason.

Make sure you've got Python & pip

Before you go any further, make sure you have Python and that it's available from your command line. You can check this by simply running:

$ python --version

You should get some output like 3.6.2. If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of this guide.

Note

If you're newcomer and you get an error like this:

>>> python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'python' is not defined

It's because this command is intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system's shell and interacting with Python.

Additionally, you'll need to make sure you have pip available. You can check this by running:

$ pip --version

If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you're on Linux and installed using your OS package manager, you may have to install pip separately.

Pipenv

Installing Pipenv

Pipenv is a dependency manager for Python projects. If you're familiar with Node.js' npm or Ruby's bundler, it is similar in spirit to those tools. While pip can install Python packages, Pipenv is recommended as it's a higher-level tool that simplifies dependency management for common use cases.

Use pip to install Pipenv:

$ pip install --user pipenv

Note

This does a user installation to prevent breaking any system-wide packages. If pipenv isn't available in your shell after installation, you'll need to add the user base's binary directory to your PATH.

On Linux and macOS you can find the user base binary directory by running python -m site --user-base and adding bin to the end. For example, this will typically print ~/.local (with ~ expanded to the absolute path to your home directory) so you'll need to add ~/.local/bin to your PATH. You can set your PATH permanently by modifying ~/.profile.

On Windows you can find the user base binary directory by running py -m site --user-site and replacing site-packages with Scripts. For example, this could return C:\Users\Username\AppData\Roaming\Python36\site-packages so you would need to set your PATH to include C:\Users\Username\AppData\Roaming\Python36\Scripts. You can set your user PATH permanently in the Control Panel. You may need to log out for the PATH changes to take effect.

Installing packages for your project

Pipenv manages dependencies on a per-project basis. To install packages, change into your project's directory (or just an empty directory for this tutorial) and run:

$ cd project_folder
$ pipenv install requests

Pipenv will install the excellent Requests library and create a Pipfile for you in your project's directory. The Pipfile is used to track which dependencies your project needs in case you need to re-install them, such as when you share your project with others. You should get output similar to this (although the exact paths shown will vary):

Creating a Pipfile for this project...
Creating a virtualenv for this project...
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in ~/.local/share/virtualenvs/tmp-agwWamBd/bin/python3.6
Also creating executable in ~/.local/share/virtualenvs/tmp-agwWamBd/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: ~/.local/share/virtualenvs/tmp-agwWamBd
Installing requests...
Collecting requests
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Installing collected packages: idna, urllib3, chardet, certifi, requests
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

Adding requests to Pipfile's [packages]...
P.S. You have excellent taste! ✨ 🍰 ✨

Using installed packages

Now that Requests is installed you can create a simple main.py file to use it:

import requests

response = requests.get('https://httpbin.org/ip')

print('Your IP is {0}'.format(response.json()['origin']))

Then you can run this script using pipenv run:

$ pipenv run python main.py

You should get output similar to this:

Your IP is 8.8.8.8

Using $ pipenv run ensures that your installed packages are available to your script. It's also possible to spawn a new shell that ensures all commands have access to your installed packages with $ pipenv shell.

Next steps

Congratulations, you now know how to install and use Python packages! ✨ 🍰 ✨

Project Isolation with Virtual Environments

If you choose not to use Pipenv or it does not fit your needs, you can use the venv tool directly to create isolated Python environments. The venv module is part of Python's standard library, and was introduced in Python 3.3. It creates a folder which contains all the necessary executables to use the packages that a Python project would need.

It can be used standalone, in place of Pipenv.

Basic Usage

1. Create a virtual environment for a project:

$ cd project_folder
$ python -m venv venv

On Windows use this command:

$ py -m venv venv

python -m venv venv will create a folder in the current directory which will contain the Python executable files, and a copy of the pip application which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything.

Note

'venv' is the general convention used globally. As it is readily available in ignore files (eg: .gitignore')

This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named :file:`venv`.

You can also use the Python interpreter of your choice, like python3.8. Sometimes python will still point to a Python 2 interpreter, so you can do this instead to be sure you are using the right Python version.

$ python3.8 -m venv venv

On Windows use this command:

$ py -3.8 -m venv venv

2. To begin using the virtual environment, you can either invoke the virtual environment's executables directly, or activate it.

To use the virtual environment's Python executable directly, run venv/bin/python; to use its pip executable, venv/bin/pip.

Alternatively, you can "activate" the environment so you can just type python or pip and it will automatically use the executables in the virtual environment (in this case, at venv/bin).

$ source venv/bin/activate

Now, the name of the current virtual environment will appear on the left of the prompt (e.g. (venv)Your-Computer:project_folder UserName$) to let you know that it's active. From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

For Windows, the same command mentioned in step 1 can be used to create a virtual environment. However, activating the environment requires a slightly different command.

Assuming that you are in your project directory:

C:\Users\SomeUser\project_folder> venv\Scripts\activate

Install packages using the pip command:

$ pip install requests

3. If you are done working in the virtual environment for the moment, you can deactivate it:

$ deactivate

This puts you back to the system's default Python interpreter with all its installed libraries. This is not necessary if you invoked the executables directly.

To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf venv.)

After a while, though, you might end up with a lot of virtual environments littered across your system. It's possible you'll forget their names or where they were placed, so try to follow a convention across your projects.

Note

The venv module is part of Python's standard library in Python3.3+. Older versions of Python can use the 3rd party package virtualenv.

Other Notes

Running python -m venv with the option --system-site-packages will include the packages that are installed globally. Usually you do not want to do this so the package list remains clean in case it needs to be accessed later.

In order to keep your environment consistent, it's a good idea to "freeze" the current state of the environment packages. To do this, run:

$ pip freeze > requirements.txt

This will create a :file:`requirements.txt` file, which contains a simple list of all the packages in the current environment, and their respective versions. You can see the list of installed packages without the requirements format using pip list. Later it will be easier for a different developer (or you, if you need to re-create the environment) to install the same packages using the same versions:

$ pip install -r requirements.txt

This can help ensure consistency across installations, across deployments, and across developers.

Lastly, remember to exclude the virtual environment folder from source control by adding it to the ignore list (see :ref:`Version Control Ignores<version_control_ignores>`).

Other Tools

There are many tools to complement usage of pip and virtual environments. Here are some useful ones we like.

direnv

When you cd into a directory containing a :file:`.env`, direnv automagically activates the environment.

Install it on Mac OS X using brew:

$ brew install direnv

On Linux follow the instructions at direnv.net

pip-tools

pip-tools is a suite of two tools that complement pip and virtual environments. It has similar functionaly to pipenv, and in fact pipenv uses pip-tools in its implementation. However, compared to pipenv, pip-tools lets you have a little more control over how and when operations are performed.

It does two things: 1.) Generate a complete dependency list (lockfile, or requirements.txt file) from abstract dependencies. It does this with a full dependency resolver, which pip does not currently have, and can optionally generate the lockfile with hashes. 2.) Synchronize a virtual environment to exactly match a requirements lockfile.

pipx

pipx is a tool to install system-wide command line tools, each to their own individual environment. Unlike pip which installs all packages to the same environment, pipx isolates tools in their own virtual environment, and exposes the command-line tools to your shell. pipx is used for installing command-line tools, similar to brew or apt, but for Python applications. It's not used to install libraries.

tox and nox

tox and nox are widely used command-line tools that automate environment setup and task execution. tox and nox are often used to run tests across multiple Python versions, among other things. They do this by reading a configuration file, either tox.ini for tox, or noxfile.py for nox.

For example, if you have unit tests that you want to run with Python 3.6, 3.7, and 3.8, you can use one of these tools. It will create three virtual environments, one for each Python version, then install necessary dependencies, and finally run tests in each environment. You can also run specific tasks like running a lint check, or publishing a new version of your package.

The main difference between the two tools are tox uses a custom file format for configuration, while nox uses a standard Python file for configuration.