The tools to dynamically install and use different versions of Python, packages, all within Virtualenv
- What is the magic of pyenv?
- What are installed?
- System Python2
- Ways to Install Python3
- Windows install
- Ubuntu install
- Brew Install Pyenv on Mac
- How is Pyenv enabled?
- What Python Releases are available?
- Order of Path override
- Virtual Environments
- References for pyenv
- References for pipenv
- More about Python
- More on OSX
This tutorial describes the different options to install, uninstall, configure, and use various versions of Python with its various packages, all running in a virtual enviornment also managed by pyenv.
The value of pyenv is:
- Install Python in your user space (without need for sudo)
- Install multiple parallel versions of Python
- Dynamically specify the exact Python version you want
- Switch between installed versions without resetting your bash session
In this article I take a carefully crafted narrated tour. Here is a hands-on “deep dive” tutorial so you better grasp the complexities in a shorter time. Why? Because I haven’t seen one on the internet.
What is the magic of pyenv?
pyenv uses a technique called “rehashing” so it can switch among multiple versions of Python2 or Python3.
Commands for the operating system to execute “python3” are intercepted by a shim executable which passes commands along to the actual Python installation of the desired version.
That’s achieved by setting the shim programs to be the first folder in your PATH because the operating system searches for executables in folders in the PATH from left to right.
What are installed?
PROTIP: Before installing things, first see what is already installed.
Get a list of the various locations where Python is installed (by various installers):
type -a python
A new macOS version would show only:
python is /usr/bin/python
PROTIP: The /usr/bin/ folder is owned by the operating system, so elevated sudo priviledges are required to modify files in it (such as “python”). So Homebrew and other installers install to /usr/local/ which does NOT require sudo to access.
Different installers install Python in different paths (but instead of “wilson_mar”, you’ll see your own user name):
python is /Users/wilson_mar/.pyenv/shims/python python is /Users/wilson_mar/anaconda3/bin/python python3 is /firstname.lastname@example.org/bin/python3 python is /usr/local/anaconda3/bin/python python is /usr/bin/python
NOTE: The Azure CLI installs the version of Python it uses at:
PROTIP: There may be several Python executables installed in different folders. But the Python program actually executed is the first one that the operating system finds among folders defined in your PATH system variable defined in ~/.bash_profile or ~/.zshenv. The operating system searches for executables in each folder in the PATH, from left to right.
You are good to go with Pipenv if you see in the first response to the type -a python command (but instead of “wilson_mar”, you’ll see your own user name):
python3 is /Users/wilson_mar/.pyenv/shims/python3
Get the path of a program:
The response is always
because the file “python” in actually a symbolic link to the actual executable.
Although macOS comes with a version of Python, it is Python2, which is now obsolete in favor of python3.
Python 2 reached end of life in January 2020.
List symbolic link “/usr/bin/python” for the default Python that comes with macOS Catalina:
ls -l /usr/bin/python
lrwxr-xr-x 1 root wheel 75 Apr 15 03:55 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
The “../../” above means that it’s above your HOME folder, in the root of macOS.
So let’s see what files are there:
Listed are executables “python”, “python2”, and “python2.7”, plus others.
View the contents of the $PATH variable to see if it includes “/usr/bin/python”:
If that’s too much, view it in a file by a text editor:
echo "$PATH" >path.txt edit path.txt rm path.txt
If it’s listed, invoke the program to get the version number:
On a MacOS Catalina version operating system, you would see:
The version is also returned from:
The Python3 interpreter is the default Python now, but it needs to be installed.
By the end of this tutorial, you would set “python” to Python 3, the current version when you do:
The response would be:
Ways to Install Python3
Before pyenv, people define what version of “python” is executed by editing the PATH variable – putting the path to Python3 executables before Python2 in $PATH. To have Python3 execute instead of Python2:
(Colon characters separate folders in the PATH)
Alternately, Azure recommends this line at the bottom of ~/.bash_profile files:
The line above overrides the system’s type command described above.
Don’t alias with pyenv
Some add aliases within the ~/.bash_profile or ~/.zshenv file:
alias python=python3 alias pip=pip3
But don’t do that when you’re using pyenv, which makes use of shims.
NOTE: pyenv is for macOS.
On Windows, consider using @kirankotari’s pyenv-win fork at:
Brew Install Pyenv on Mac
After installing the Homebrew package manager:
brew install pyenv
The response at time of this writing:
==> Installing dependencies for pyenv: email@example.com and pkg-config ==> Installing pyenv dependency: firstname.lastname@example.org ==> Downloading https://email@example.com ==> Downloading from https://akamai.bintray.com/19/1926679569c6af5337de812d86f4d ######################################################################## 100.0% ==> Pouring firstname.lastname@example.org ==> Caveats A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /email@example.com/certs and run /firstname.lastname@example.org/bin/c_rehash email@example.com is keg-only, which means it was not symlinked into /usr/local, because macOS provides LibreSSL. If you need to have firstname.lastname@example.org first in your PATH run: echo 'export PATH="/email@example.com/bin:$PATH"' >> ~/.bash_profile For compilers to find firstname.lastname@example.org you may need to set: export LDFLAGS="-Lemail@example.com/lib" export CPPFLAGS="-Ifirstname.lastname@example.org/include" For pkg-config to find email@example.com you may need to set: export PKG_CONFIG_PATH="/firstname.lastname@example.org/lib/pkgconfig" ==> Summary 🍺 /usr/local/Cellaremail@example.com/1.1.1g: 8,059 files, 18MB ==> Installing pyenv dependency: pkg-config ==> Downloading https://homebrew.bintray.com/bottles/pkg-config-0.29.2_3.catalin ==> Downloading from https://akamai.bintray.com/80/80f141e695f73bd058fd82e9f539d ######################################################################## 100.0% ==> Pouring pkg-config-0.29.2_3.catalina.bottle.tar.gz Error: The `brew link` step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink bin/pkg-config Target /usr/local/bin/pkg-config already exists. You may want to remove it: rm '/usr/local/bin/pkg-config' To force the link and overwrite all conflicting files: brew link --overwrite pkg-config To list all files that would be deleted: brew link --overwrite --dry-run pkg-config Possible conflicting files are: /usr/local/bin/pkg-config /usr/local/share/aclocal/pkg.m4 /usr/local/share/doc/pkg-config/pkg-config-guide.html /usr/local/share/man/man1/pkg-config.1 ==> Summary 🍺 /usr/local/Cellar/pkg-config/0.29.2_3: 11 files, 623.7KB ==> Installing pyenv ==> Downloading https://homebrew.bintray.com/bottles/pyenv-1.2.18.catalina.bottl ==> Downloading from https://akamai.bintray.com/bd/bd9f719f153e9574dcc65dc7fea28 ######################################################################## 100.0% ==> Pouring pyenv-1.2.18.catalina.bottle.tar.gz 🍺 /usr/local/Cellar/pyenv/1.2.18: 695 files, 2.5MB ==> Caveats ==> firstname.lastname@example.org A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /email@example.com/certs and run /firstname.lastname@example.org/bin/c_rehash email@example.com is keg-only, which means it was not symlinked into /usr/local, because macOS provides LibreSSL. If you need to have firstname.lastname@example.org first in your PATH run: echo 'export PATH="/email@example.com/bin:$PATH"' >> ~/.bash_profile For compilers to find firstname.lastname@example.org you may need to set: export LDFLAGS="-Lemail@example.com/lib" export CPPFLAGS="-Ifirstname.lastname@example.org/include" For pkg-config to find email@example.com you may need to set: export PKG_CONFIG_PATH="/firstname.lastname@example.org/lib/pkgconfig"
Notice that brew takes care of installing dependencies needed. That’s a huge benefit of using the brew package manager. Otherwise, you would have to install dependencies yourself before installing pyenv, which can be error-prone. See
- https://realpython.com/intro-to-pyenv/ about Pyenv Build Dependencies.
Can the program execute?
At time of writing, the response was the version.release.path semantic version:
Let’s ask where the operating system thinks pyenv is installed:
/usr/local/bin/ is NOT owned by the operating system, so sudo is NOT required to modify files in it. That’s why installers such as Brew install programs there.
Note that file /usr/local/bin/pyenv is a symbolic link which points to the actual executable (at bash script) in another folder:
head -3 /usr/local/bin/pyenv
head -3lists the first 3 lines of the file.
The response #!/usr/bin/env bash is the “shebang” line which specifies that the text file be treated as a bash shell script.
What is the latest version of pyenv?
brew info pyenv
The response, at time of writing, several months after the initial install:
pyenv: stable 1.2.20 (bottled), HEAD Python version management https://github.com/pyenv/pyenv /usr/local/Cellar/pyenv/1.2.20 (708 files, 2.5MB) * Poured from bottle on 2020-07-28 at 16:54:33 From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pyenv.rb License: MIT ==> Dependencies Required: autoconf ✔, email@example.com ✔, pkg-config ✔, readline ✔ ==> Options --HEAD Install HEAD version ==> Analytics install: 68,833 (30 days), 178,833 (90 days), 650,651 (365 days) install-on-request: 65,514 (30 days), 169,985 (90 days), 606,662 (365 days) build-error: 0 (30 days)
Previously in 1.2.19:
install: 61,768 (30 days), 168,980 (90 days), 637,536 (365 days) install-on-request: 58,851 (30 days), 160,831 (90 days), 594,089 (365 days) build-error: 0 (30 days)
To find the actual executable file which brew installed:
ls -al /usr/local/bin/pyenv
The response shows a link to where Homebrew stores its executable:
lrwxr-xr-x 1 wilson_mar admin 32 Apr 24 21:55 /usr/local/bin/pyenv -> ../Cellar/pyenv/1.2.18/bin/pyenv
PROTIP: The “..” means “/usr/local/” should be added to make the full path:
ls -al /usr/local/Cellar/pyenv/1.2.18/bin/pyenv
lrwxr-xr-x 1 wilson_mar admin 32 Jul 28 16:54 /usr/local/bin/pyenv -> ../Cellar/pyenv/1.2.20/bin/pyenv
List sub-commands recognized by pyenv:
At time of writing:
pyenv 1.2.20 Usage: pyenv <command> [<args>] Some useful pyenv commands are: --version Display the version of pyenv activate Activate virtual environment commands List all available pyenv commands deactivate Deactivate virtual environment exec Run an executable with the selected Python version global Set or show the global Python version(s) help Display help for a command hooks List hook scripts for a given pyenv command init Configure the shell environment for pyenv install Install a Python version using python-build local Set or show the local application-specific Python version(s) prefix Display prefix for a Python version sed: RE error: illegal byte sequence rehash Rehash pyenv shims (run this after installing executables) root Display the root directory where versions and shims are kept shell Set or show the shell-specific Python version shims List existing pyenv shims uninstall Uninstall a specific Python version version Show the current Python version(s) and its origin version-file Detect the file that sets the current pyenv version version-name Show the current Python version version-origin Explain how the current Python version is set versions List all Python versions available to pyenv virtualenv Create a Python virtualenv using the pyenv-virtualenv plugin virtualenv-delete Uninstall a specific Python virtualenv virtualenv-init Configure the shell environment for pyenv-virtualenv virtualenv-prefix Display real_prefix for a Python virtualenv version virtualenvs List all Python virtualenvs found in `$PYENV_ROOT/versions/*'. whence List all Python versions that contain the given executable which Display the full path to an executable See `pyenv help
' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readme
For a full list of sub-commands recognized by pyenv:
At time of writing:
--version commands completions exec global help hooks init install local prefix realpath.dylib rehash root shell shims uninstall version version-file version-file-read version-file-write version-name version-origin versions whence which
Where are pyenv shims installed?
PROTIP: When pyenv is installed, a folder ~/.pyenv is created under your user $HOME folder.
Reveal folders under the pyenv root containing the shims folder:
ls $( pyenv root)
The response shows that the installer created a folder at my user $HOME folder (but your user name instead of my “wilson_mar”):
plugins shims version versions
See the various versions of python (and associated utilities) we want to put at the top of the $PATH:
ls $(pyenv root)/shims
2to3 pbr python3-config 2to3-3.7 pip python3.7 2to3-3.8 pip3 python3.7-config bandit pip3.7 python3.7-gdb.py bandit-baseline pip3.8 python3.7m bandit-config-generator pipenv python3.7m-config chardetect pipenv-resolver python3.8 easy_install pydoc python3.8-config easy_install-3.7 pydoc3 python3.8-gdb.py easy_install-3.8 pydoc3.7 pyvenv estimator_ckpt_converter pydoc3.8 pyvenv-3.7 f2py pyrsa-decrypt saved_model_cli f2py3 pyrsa-encrypt tensorboard f2py3.7 pyrsa-keygen tf_upgrade_v2 google-oauthlib-tool pyrsa-priv2pub tflite_convert idle pyrsa-sign toco idle3 pyrsa-verify toco_from_protos idle3.7 python virtualenv idle3.8 python-config virtualenv-clone markdown_py python3 wheel
Within the shims folder are every Python command in every installed version of Python—python, pip, etc.
Recursively count files (-type f) within the shims folder:
* system (set by /Users/wilson_mar/.pyenv/version)
find ~/.pyenv/shims -type f | wc -l
How is Pyenv enabled?
Edit your ~/.bash_profile and/or ~/.zshenv file so that the PATH comes before “/usr/local/bin” and “/usr/bin” where others install programs:
# Set it so ~/.pyenv provides Python before others of the same name: export PYENV_ROOT=$(pyenv root) export PATH="$PYENV_ROOT/bin:$PATH"
Put the above at the bottom of the file.
Alternately, if you don’t want to mess with a text editor, use this command:
echo 'export PATH="$(pyenv root)/bin:$PATH"' >> ~/.bash_profile
Instead of .bash_profile, specify ~/.zshrc or ~/.bashrc (for Ubuntu/Fedora).
(Colon characters separate folders in the PATH)
Whether you edited your shell script or not, restart your shell terminal so the path changes take effect:
exec "$SHELL" source ~/.bash_profile
echo "$PATH" >path.txt edit path.txt rm path.txt
Confirm whether pyenv is now active:
Response should be:
(“wilson_mar” would be replaced with your user name)
What Python Releases are available?
Pyenv can install several releases of Python.
PROTIP: Each version of Python is installed within pyenv’s versions folder:
ls $( pyenv root)/versions
List folders within the versions folder where pyenv installs versions of Python (for example):
tree -d -L 1 $( pyenv root )/versions
├── 3.7.7 └── 3.8.2
None would be listed until versions are added.
Limit the long list of Python Releases to 3.7 or 3.8 or 3.9:
pyenv install --list | grep " 3\."
At time of writing:
3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 3.7.4 3.7.5 3.7.6 3.7.7 3.8.0 3.8-dev 3.8.1 3.8.2 3.9.0a5 3.9-dev
PROTIP: Select the last patch number of a release, such as “3.7.7” or “3.8.2” in the example above.
-dev releases are the ones under active development (changes).
## Install Python using pyenv
Navigate to the folder containing your Python .py file.
To install a specific version of Python (selected from the response above):
pyenv install 3.7.7
python-build: use firstname.lastname@example.org from homebrew python-build: use readline from homebrew Downloading Python-3.7.7.tar.xz... -> https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tar.xz Installing Python-3.7.7... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.7.7 to /Users/wilson_mar/.pyenv/versions/3.7.7
Note that pyenv installs under a versions folder.
pyenv install 3.8.2
python-build: use email@example.com from homebrew python-build: use readline from homebrew Downloading Python-3.8.2.tar.xz... -> https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz Installing Python-3.8.2... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.8.2 to /Users/wilson_mar/.pyenv/versions/3.8.2
List the versions of Python installed and managed by pyenv:
In the response such as:
* 3.7.7 (set by /Users/wilson_mar/.pyenv/version) 3.8.2
*indicates the current system Python release.
Set Version of Python using Pyenv
To set a specific version as the default:
pyenv global 3.8.2
No response is returned if successful.
To select a specific version:
pyenv local 3.7.7
No response is returned if successful.
PROTIP: The pyenv local command above creates in the folder a hidden file named .python-version containing the Python version specified in the command.
Verify what version.release.patch:
Order of Path override
When a shim (such as “python”) is executed, it calls pyenv which determines which Python release to use by reading sources in this order:
- $PYENV_VERSION environment variable
- pyenv local x.y.z
- pyenv global x.y.z
The PYENV_VERSION environment variable overrides all other specifications.
Set this environment variable in your current shell session using the pyenv shell command.
If the PYENV_VERSION system variable is not specified, the application-specific .python-version file in the current directory.
Modify the current directory’s .python-version file with the command:
pyenv local 2.7.15
If there is no .python-version file locally, parent directories are searched upward until reaching the root of your filesystem.
If no .python-version is specified in any folder, the global $(pyenv root)/version file.
Modify the global file using the command:
pyenv global 3.7.7
If the global version file is not present, pyenv assumes you want to use the “system” Python. (In other words, whatever version would run if pyenv weren’t in your PATH.)
Both Pipenv and Pipenv automatically creates and manages a virtualenv for your projects. But only Pipenv adds/removes packages from your Pipfile as you install/uninstall packages. Pipenv generates the Pipfile.lock file used to produce deterministic builds.
Pipenv enables always use the latest versions of dependencies, to minimize security risks arising from outdated components.
pyenv copies an entire Python installation every time a new pyenv version is created.
By contrast, virtualenv makes use of symbolic links, which decreases the size of each virtualenv.
There are two utilities to add virtualenv functionality to pyenv: pyenv-virtualenvwrapper (described by https://alysivji.github.io/setting-up-pyenv-virtualenvwrapper.html) is less convenient and has less stars than the pyenv-virtualenv plugin. To install it:
brew install pyenv-virtualenv
git clone https://github.com/pyenv/pyenv-virtualenv.git \ $(pyenv root)/plugins/pyenv-virtualenv source ~/.bashrc
Cloning into '/Users/wilson_mar/.pyenv/plugins/pyenv-virtualenv'... remote: Enumerating objects: 2064, done. remote: Total 2064 (delta 0), reused 0 (delta 0), pack-reused 2064 Receiving objects: 100% (2064/2064), 580.31 KiB | 30.00 KiB/s, done. Resolving deltas: 100% (1413/1413), done.
This is needed because Virtualenv and Anaconda also activate scripts by mutating $PATH variable of user’s interactive shell, which intercepts pyenv’s shim style command execution hooks.
To automatically activate/deactivate virtualenvs on entering/leaving directories which contain a .python-version file that contains the name of a valid virtual environment as shown in the output of pyenv virtualenvs
echo ‘eval “$(pyenv virtualenv-init -)”’ » ~/.bash_profile or ~/.zshrc
List what virtualenv has been defined
List what virtualenv have been defined:
Setup Virtual enviornments
There are several ways to enter and setup a virtual environment:
Navigate to where you .py files are. This is important.
Set up a virtual environment in the present working directory, using a common conventional name:
python -m venv venv
No response is returned if it’s all good.
The above creates a virtual folder named after the project.
To deactivate, type:
Alternately, to activate virtualenv, run:
pipenv shell</pip> Alternatively, run a command inside the virtualenv
Activate the virtual environment in the present working directory:
This would turn the command prompt as a constant reminder:
The Pipfile.lock file contains a hash of each package installed, which provides security. The file pins semantic versions of all dependencies and sub-dependencies, which provides replicable environments.
References for pyenv
References for pipenv
https://github.com/VaultVulp/pipenv-alpine/blob/master/Dockerfile is the Python Alpine image with pre-installed pipenv
More about Python
This is one of a series about Python:
- Python install on MacOS using Pyenv
- Python install on MacOS
- Test Python using Pytest BDD Selenium framework
- Python certifications
- Python tutorials
- Python coding notes
- Pulumi controls cloud using Python, etc.
- Microsoft Azure Machine Learning makes use of Python
- Python REST API programming using the Flask library
- Python coding for AWS Lambda Serverless programming
- Streamlit visualization framework powered by Python
- Web scraping using Scrapy, powered by Python
- Neo4j graph databases accessed from Python
More on OSX
This is one of a series on Mac OSX:
- MacOS Hardware and accessories
- MacOS Boot-up
- MacOS Terminal Tips and Tricks
- MacOS Find (files and text in files)
- MacOS Keyboard tricks
- MacOS Setup automation
- MacOS Homebrew installers
- Printing from macOS or Linux
- Manage Disk Space on MacOS
- Data Backups on MacOS
- Ports open
- Applications on MacOS
- Windows on Apple MacOS
- Packer create Vagrant Windows image
- Python on MacOS
- Maven on MacOS
- Ruby on MacOS
- Java on MacOS
- Node on MacOS installation
- PHP on MacOS
- Scala ecosystem