Pip constraints files

In Python, it is common practice to write all the application dependencies that are installed via pip into a separate text file. The name of the dependency file is arbitrary, but the requirements.txt file is often used.

Pip is still widely used but you may consider using more powerful build tools like Poetry. It can handle private repositories, optional dependencies, dev dependencies, etc.

It is good practice to fully specify the versions of the packages in the requirements file in order to achieve repeatable installations via:

pip install -r requirements.txt

The requirements file contains every package with a pinned version — both the direct dependencies of our application and the indirect dependencies — dependencies of dependencies (without packages that pip itself depends on). But sometimes, especially in a long-lived project, it is hard to recognize what are the original dependencies of the application. There are many reasons for doing so — we need to keep them up to date and not depend on packages that are obsolete or for some reason no longer needed.

For example, which of the following dependencies are the original ones?

# requirements.txt
numpy==1.17.4
pandas==0.24.2
python-dateutil==2.8.1
pytz==2019.3
six==1.13.0

Yes, it’s just pandas.

In pip since version 7.1, the --constraint flag appeared in the pip install command and can be used to solve this problem.

Constraint files are requirement files that only control which version of a requirement is installed, not whether it is installed or not. They essentially help solve CSP problems with respect to Python dependencies. Their syntax and content is a subset of requirements files:

# constraints.txt
numpy==1.17.4
python-dateutil==2.8.1
pytz==2019.3
six==1.13.0

Some types of syntax are not allowed: constraints must have a name, they cannot be editable, and they cannot specify complements.

Semantically, there is one key difference: the file will not be used to determine which packages to install, but will be used to lock versions for all packages that are already installed. This means that we can put our base requirements (i.e. direct dependencies) in the requirements file, and then keep version locks for all environments in a separate constraints.txt file.

First, we want to make sure that we never forget to add --constraint constraint.txt by adding it to the beginning of our requirements.txt file (and any other requirements file).

--constraint constraints.txt
pandas==0.24.2

Then create a constraint file using the pip freeze > constraint.txt command. Now we can modify all your requirements by removing or loosening version constraints and removing nested dependencies.

Now we can keep everything clean — just list direct dependencies in requirements.txt or setup.py, without the exact versions, and keep the exact versions for all packages in the constraint file. When pip installs a package (or a dependency on a package), it checks in the constraints file which version to install. If the required dependency was a top-level requirement for the project, that particular line in the requirements file can simply be replaced. If it is a sub-dependency of another file, the above command will be added as a new line.

We can have organization-wide constraints file to specify which versions of possible dependencies should be installed for one reason or another - security, consistency, or dependency management. For example, the API has changed, but the project using it hasn't been updated yet, and projects may be mixed, so you need them all to use compatible versions. Fix libraries and install them from custom locations without any problems.

Constraint files are also very useful when you provide production environments with optimized package compilations for production infrastructure. For example, docker containers with optimized builds of numpy, scipy, tensorflow, opencv, ... for specific needs and environments. This way, teams get optimized configurations automatically when they build the container, while still being able to use other versions for development.

Don't limit yourself to this - perhaps there is a use case that will fit the use of this feature.

Additional materials

• Learning Python by Mark Lutz
• Advanced Python Programming by Quan Nguyen
• Python Distilled by David Beazley