Managing Python Environments and Dependencies

Posted by Chris Dinh on June 06, 2021 · 8 mins read

The great thing about Python is that whatever problems (including IoT problems) we are trying to solve, there is a very high chance there’s is a Python library out there that’s going to make our life a lot easier. The troublesome thing about Python is that, even at this point in time, there is still two Python universe and it makes life very hard for those that wants to move forward.

Let’s consider the following scenario:

  • We have a solution that depends on a set of libraries and at specific versions.
  • Those libraries also depends on other libraries and also at specific versions.
  • Runs on Python 2.7
  • Libraries owners may not have the time to migrate to Python 3.

If we take the above scenario and try to migrate the solution to Python 3, it can be a daunting task.

Managing Python environments in Windows

The purpose of this post is not solving our migration problems but more about keeping things nice and tidy, manageable, and us developers happy. To do that I will discuss some concepts that every Python developers should be familiar with:

  • Python Launcher
  • Virtual Environments
  • Pip

Python Launcher

Python Launcher is a Windows utility used to identify which Python versions as well as where they are installed. It also provide a mechanism to launch Python programs with a specified Python interpreter.

For my development environment, I know that I need to work with both Python 2 and Python 3 so let’s head over to python.org download page download the latest version for each (as of this writing they were Python 3.9.5 and Python 2.7.18). Also since Python Launcher was introduced in Python 3.3, it will be included in Python 3.9.5.

Installing Python 2.7.18 Installing Python 2.7.18

Installing Python 3.9.5 Installing Python 3.9.5

After installation has been completed, open a terminal window and execute the command “py –help” which will print out the help for “py” (Python Launcher).

Python launcher command demo Python launcher command demo

Virtual Environments

Python virtual environment allows developers to create self-contained environments when working in Python. My preferred workflow is to create a virtual environment per project I work with so that Python version and external libraries are only relevant to the environment they were created in. Below are examples of create a Python 2 and Python 3 virtual environment, as well as activating and deactivating them.

    PS D:\Workspace> mkdir python2-project-demo
        Directory: D:\Workspace
    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----           5/06/2021 08:45 pm                python2-project-demo
    PS D:\Workspace> cd .\python2-project-demo\
    PS D:\Workspace\python2-project-demo> py -2 -m virtualenv venv
    created virtual environment CPython2.7.18.final.0-64 in 8526ms
    creator CPython2Windows(dest=D:\Workspace\python2-project-demo\venv, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=C:\Users\Chris\AppData\Local\pypa\virtualenv)
        added seed packages: pip==20.3.4, setuptools==44.1.1, wheel==0.36.2
    activators PythonActivator,FishActivator,BatchActivator,BashActivator,PowerShellActivator
    PS D:\Workspace\python2-project-demo> ls
        Directory: D:\Workspace\python2-project-demo
    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----           5/06/2021 08:46 pm                venv
    PS D:\Workspace\python2-project-demo> .\venv\Scripts\activate
    (venv) PS D:\Workspace\python2-project-demo> python -V
    Python 2.7.18
    (venv) PS D:\Workspace\python2-project-demo> deactivate
    PS D:\Workspace\python2-project-demo>
    PS D:\Workspace> mkdir python3-project-demo
        Directory: D:\Workspace
    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----           6/06/2021 08:50 pm                python3-project-demo
    PS D:\Workspace> cd .\python3-project-demo\
    PS D:\Workspace\python3-project-demo> py -3 -m venv venv
    PS D:\Workspace\python3-project-demo> ls
        Directory: D:\Workspace\python3-project-demo
    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----           6/06/2021 08:51 pm                venv
    PS D:\Workspace\python3-project-demo> .\venv\Scripts\activate
    (venv) PS D:\Workspace\python3-project-demo> python -V
    Python 3.9.5
    (venv) PS D:\Workspace\python3-project-demo> deactivate
    PS D:\Workspace\python3-project-demo>

Pip

Pip is a Python package installer. It is used for installing and managing external libraries for a given Python environment.

One reason I use Python is when working with CoAP. CoAP is a client/server transport protocol ideally used for communicating with constraint resource devices. In Python 2.7, CoAPthon is a good library to use. While in Python 3, Aiocoap is a nice option.

Using the Python 3 virtual environment created earlier, let’s take a look at some of the useful pip commands.

For installing external packages, we can use pip install <package-to-install>.

    (venv) PS D:\Workspace\python3-program> pip install aiocoap
    Collecting aiocoap
    Using cached aiocoap-0.4.1.tar.gz (201 kB)
    Building wheels for collected packages: aiocoap
    Building wheel for aiocoap (setup.py) ... done
    Created wheel for aiocoap: filename=aiocoap-0.4.1-py3-none-any.whl size=181210 sha256=09fafe85ee179a31e52bb57b088af1fcdf3e4114b664c6af7615af5715feb54c
    Stored in directory: c:\users\chris\appdata\local\pip\cache\wheels\2c\a6\f8\63466bc1a04e8fb6e88707a6005a65529e451f02221e88f542
    Successfully built aiocoap
    Installing collected packages: aiocoap
    Successfully installed aiocoap-0.4.1
    (venv) PS D:\Workspace\python3-program>

We can use pip list to list all installed packages.

    (venv) PS D:\Workspace\python3-program> pip list
    Package    Version
    ---------- -------
    aiocoap    0.4.1
    pip        21.1.2
    setuptools 56.0.0
    wheel      0.36.2
    (venv) PS D:\Workspace\python3-program>

We can use pip show <package-to-show> to show the specified package’s details.

    (venv) PS D:\Workspace\python3-program> pip show aiocoap
    Name: aiocoap
    Version: 0.4.1
    Summary: Python CoAP library
    Home-page: https://github.com/chrysn/aiocoap
    Author: Maciej Wasilak, Christian Amsüss
    Author-email: c.amsuess@energyharvesting.at
    License: MIT
    Location: d:\workspace\python3-program\venv\lib\site-packages
    Requires:
    Required-by:
    (venv) PS D:\Workspace\python3-program>

We can use “pip list –format=freeze > requirements.txt” to export the list to a file which could then be used in another virtual environment.

    (venv) PS D:\Workspace\python3-program> pip list --format=freeze > requirements.txt
    (venv) PS D:\Workspace\python3-program> Get-Content .\requirements.txt
    aiocoap==0.4.1
    pip==21.1.2
    setuptools==56.0.0
    wheel==0.36.2
    (venv) PS D:\Workspace\python3-program> deactivate
    PS D:\Workspace\python3-program> cd ..
    PS D:\Workspace> mkdir python3-program-v2

        Directory: D:\Workspace

    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----           7/06/2021 8:46 pm                python3-program-v2

    PS D:\Workspace> cd .\python3-program-v2\
    PS D:\Workspace\python3-program-v2> py -3 -m venv venv
    PS D:\Workspace\python3-program-v2> .\venv\Scripts\activate
    (venv) PS D:\Workspace\python3-program-v2> pip list
    Package    Version
    ---------- -------
    pip        21.1.1
    setuptools 56.0.0
    WARNING: We are using pip version 21.1.1; however, version 21.1.2 is available.
    We should consider upgrading via the 'd:\workspace\python3-program-v2\venv\scripts\python.exe -m pip install --upgrade pip' command.
    (venv) PS D:\Workspace\python3-program-v2> python.exe -m pip install --upgrade pip
    Requirement already satisfied: pip in d:\workspace\python3-program-v2\venv\lib\site-packages (21.1.1)
    Collecting pip
    Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
    Installing collected packages: pip
    Attempting uninstall: pip
        Found existing installation: pip 21.1.1
        Uninstalling pip-21.1.1:
        Successfully uninstalled pip-21.1.1
    Successfully installed pip-21.1.2
    (venv) PS D:\Workspace\python3-program-v2> cp ..\python3-program\requirements.txt .
    (venv) PS D:\Workspace\python3-program-v2> pip install -r .\requirements.txt
    Collecting aiocoap==0.4.1
    Using cached aiocoap-0.4.1-py3-none-any.whl
    Requirement already satisfied: pip==21.1.2 in d:\workspace\python3-program-v2\venv\lib\site-packages (from -r .\requirements.txt (line 2)) (21.1.2)
    Requirement already satisfied: setuptools==56.0.0 in d:\workspace\python3-program-v2\venv\lib\site-packages (from -r .\requirements.txt (line 3)) (56.0.0)
    Collecting wheel==0.36.2
    Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
    Installing collected packages: wheel, aiocoap
    Successfully installed aiocoap-0.4.1 wheel-0.36.2
    (venv) PS D:\Workspace\python3-program-v2> pip list
    Package    Version
    ---------- -------
    aiocoap    0.4.1
    pip        21.1.2
    setuptools 56.0.0
    wheel      0.36.2
    (venv) PS D:\Workspace\python3-program-v2>

Installing the latest Python interpreter on Linux (Raspberry Pi OS)

As of writing, Python 3.9 was the latest version, however this was not installed by default.

Check installed Python demo Check installed Python demo

We can install Python 3.9 from source. Python 3.9 install on Raspberry PI OS is a good post showing how we can go about doing that.

Install from source demo Install from source demo