The tools to dynamically install and use different versions of Python, packages, all within Virtualenv
- Why pipenv?
- What is the magic of pyenv?
- What are installed?
- System Python2
- Ways to Install Python3
- Windows install
- Ubuntu install
- Brew Install Pyenv on Mac
- How is Pyenv enabled?
- What Python Releases are available?
- Install Python using pyenv
- Order of Path override
- Virtual Environments
- Upgrade Pyenv
- References for pyenv
- References for pipenv
- More about Python
- More on OSX
This tutorial describes the different options to install, uninstall, configure, and use various versions of Python with its various packages, all running in a virtual environment also managed by pyenv.
NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.
pipenv brings “the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world.”
The value of pyenv is:
- Install Python in your user space (without need for sudo)
- Install multiple parallel versions of Python
- Dynamically specify the exact Python version you want
- Switch between installed versions without resetting your bash session
What is the magic of pyenv?
In this article I take a carefully crafted narrated tour. Here is a hands-on “deep dive” tutorial so you better grasp the complexities in a shorter time. Why? Because I haven’t seen one on the internet.
pyenv uses a technique called “rehashing” so it can switch among multiple versions of Python2 or Python3.
Commands for the operating system to execute “python3” are intercepted by a shim executable which passes commands along to the actual Python installation of the desired version.
That’s achieved by a line at the bottom of your ~/.bash_profile which makes the operating system search in the shims folder for programs:
This technique is possible because the operating system searches for executables in folders in the PATH from left to right.
Thus, which aws would return:
The alternative to Pyenv is Conda and MiniConda.
What are installed?
PROTIP: Before installing things, first see what is already installed.
Get a list of the various locations where Python is installed (by various installers):
type -a python
A new macOS version would show:
python is /usr/bin/python
PROTIP: The /usr/bin/ folder is owned by the operating system, so elevated sudo priviledges are required to modify files in it (such as “python”). So Homebrew and other installers install to the User-owned /usr/local/ which does NOT require sudo to access.
Different installers install Python in different paths (but instead of “wilson_mar”, you’ll see your own user name):
python is /Users/wilson_mar/.pyenv/shims/python python is /Users/wilson_mar/anaconda3/bin/python python3 is /firstname.lastname@example.org/bin/python3 python is /usr/local/anaconda3/bin/python python is /usr/bin/python
You are good to go with Pipenv if you see in the first response to the type -a python command (but instead of “wilson_mar”, you’ll see your own user name):
python3 is /Users/wilson_mar/.pyenv/shims/python3
PROTIP: There may be several Python executables installed in different folders. But the Python program actually executed is the first one that the operating system finds among folders defined in your PATH system variable defined in ~/.bash_profile or ~/.zshenv. The operating system searches for executables in each folder in the PATH, from left to right.
NOTE: The Azure CLI installs the version of Python it uses at:
Get the path of a program:
The response is
because the file “python” in actually a symbolic link to the actual executable.
Although earlier macOS comes with a version of Python, it is Python2, which is now obsolete in favor of python3.
Python 2 reached end of life in January 2020.
List symbolic link “/usr/bin/python” for the default Python that comes with macOS Catalina:
ls -l /usr/bin/python
lrwxr-xr-x 1 root wheel 75 Apr 15 03:55 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
The “../../” above means that it’s above your HOME folder, in the root of macOS.
So let’s see what files are there:
Listed are executables “python”, “python2”, and “python2.7”, plus others.
View the contents of the $PATH variable to see if it includes “/usr/bin/python”:
If that’s too much, view it in a file by a text editor:
echo "$PATH" >path.txt edit path.txt rm path.txt
If it’s listed, invoke the program to get the version number:
On a MacOS Catalina version operating system, you would see:
The version is also returned from:
The Python3 interpreter is the default Python now, but it needs to be installed.
By the end of this tutorial, you would set “python” to Python 3, the current version when you do:
The response would be:
Ways to Install Python3
Before pyenv, people define what version of “python” is executed by editing the PATH variable – putting the path to Python3 executables before Python2 in $PATH. To have Python3 execute instead of Python2:
(Colon characters separate folders in the PATH)
Alternately, Azure recommends this line at the bottom of ~/.bash_profile files:
The line above overrides the system’s type command described above.
Don’t alias with pyenv
Some add aliases within the ~/.bash_profile or ~/.zshenv file:
alias python=python3 alias pip=pip3
But don’t do that when you’re using pyenv, which makes use of shims.
NOTE: pyenv is for macOS.
On Windows, consider using @kirankotari’s pyenv-win fork at:
Brew Install Pyenv on Mac
After installing the Homebrew package manager:
brew install pyenv
The response at time of this writing:
==> Installing dependencies for pyenv: email@example.com and pkg-config ==> Installing pyenv dependency: firstname.lastname@example.org ==> Downloading https://email@example.com ==> Downloading from https://akamai.bintray.com/19/1926679569c6af5337de812d86f4d ==> Pouring firstname.lastname@example.org ==> Caveats A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /email@example.com/certs and run /firstname.lastname@example.org/bin/c_rehash email@example.com is keg-only, which means it was not symlinked into /usr/local, because macOS provides LibreSSL. If you need to have firstname.lastname@example.org first in your PATH run: echo 'export PATH="/email@example.com/bin:$PATH"' >> ~/.bash_profile For compilers to find firstname.lastname@example.org you may need to set: export LDFLAGS="-Lemail@example.com/lib" export CPPFLAGS="-Ifirstname.lastname@example.org/include" For pkg-config to find email@example.com you may need to set: export PKG_CONFIG_PATH="/firstname.lastname@example.org/lib/pkgconfig" ==> Summary 🍺 /usr/local/Cellaremail@example.com/1.1.1g: 8,059 files, 18MB ==> Installing pyenv dependency: pkg-config ==> Downloading https://homebrew.bintray.com/bottles/pkg-config-0.29.2_3.catalin ==> Downloading from https://akamai.bintray.com/80/80f141e695f73bd058fd82e9f539d ==> Pouring pkg-config-0.29.2_3.catalina.bottle.tar.gz Error: The `brew link` step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink bin/pkg-config Target /usr/local/bin/pkg-config already exists. You may want to remove it: rm '/usr/local/bin/pkg-config' To force the link and overwrite all conflicting files: brew link --overwrite pkg-config To list all files that would be deleted: brew link --overwrite --dry-run pkg-config Possible conflicting files are: /usr/local/bin/pkg-config /usr/local/share/aclocal/pkg.m4 /usr/local/share/doc/pkg-config/pkg-config-guide.html /usr/local/share/man/man1/pkg-config.1 ==> Summary 🍺 /usr/local/Cellar/pkg-config/0.29.2_3: 11 files, 623.7KB ==> Installing pyenv ==> Downloading https://homebrew.bintray.com/bottles/pyenv-1.2.18.catalina.bottl ==> Downloading from https://akamai.bintray.com/bd/bd9f719f153e9574dcc65dc7fea28 ==> Pouring pyenv-1.2.18.catalina.bottle.tar.gz 🍺 /usr/local/Cellar/pyenv/1.2.18: 695 files, 2.5MB ==> Caveats ==> firstname.lastname@example.org A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /email@example.com/certs and run /firstname.lastname@example.org/bin/c_rehash email@example.com is keg-only, which means it was not symlinked into /usr/local, because macOS provides LibreSSL. If you need to have firstname.lastname@example.org first in your PATH run: echo 'export PATH="/email@example.com/bin:$PATH"' >> ~/.bash_profile For compilers to find firstname.lastname@example.org you may need to set: export LDFLAGS="-Lemail@example.com/lib" export CPPFLAGS="-Ifirstname.lastname@example.org/include" For pkg-config to find email@example.com you may need to set: export PKG_CONFIG_PATH="/firstname.lastname@example.org/lib/pkgconfig"
Notice that brew takes care of installing dependencies needed. That’s a huge benefit of using the brew package manager. Otherwise, you would have to install dependencies yourself before installing pyenv, which can be error-prone. See
- https://realpython.com/intro-to-pyenv/ about Pyenv Build Dependencies.
Can the program execute?
At time of writing, the response was the version.release.path semantic version:
TODO: Create variable containing “1.2.26”.
Let’s ask where the operating system thinks pyenv is installed:
/usr/local/bin/ is NOT owned by the operating system, so sudo is NOT required to modify files in it. That’s why installers such as Brew install programs there.
Note that file /usr/local/bin/pyenv is a symbolic link which points to the actual executable (at bash script) in another folder:
head -3 /usr/local/bin/pyenv
head -3lists the first 3 lines of the file.
The response #!/usr/bin/env bash is the “shebang” line which specifies that the text file be treated as a bash shell script.
What is the latest version of the pyenv installer on Homebrew?
brew info pyenv
The response, at time of writing, several months after the initial install:
pyenv: stable 1.2.26 (bottled), HEAD Python version management https://github.com/pyenv/pyenv /usr/local/Cellar/pyenv/1.2.26 (747 files, 2.6MB) * Poured from bottle on 2021-04-06 at 10:17:17 From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pyenv.rb License: MIT ==> Dependencies Required: autoconf ✔, email@example.com ✔, pkg-config ✔, readline ✔ ==> Options --HEAD Install HEAD version
Previously in 1.2.19:
install: 61,768 (30 days), 168,980 (90 days), 637,536 (365 days) install-on-request: 58,851 (30 days), 160,831 (90 days), 594,089 (365 days) build-error: 0 (30 days)
To find the actual executable file which brew installed:
ls -al /usr/local/bin/pyenv
The response shows a link to where Homebrew stores its executable:
lrwxr-xr-x 1 wilson_mar admin 32 Apr 6 10:17 /usr/local/bin/pyenv -> ../Cellar/pyenv/1.2.26/bin/pyenv
PROTIP: The “..” means “/usr/local/” should be added to make the full path:
ls -al /usr/local/Cellar/pyenv/1.2.26/bin/pyenv
lrwxr-xr-x 1 wilson_mar staff 16 Apr 5 05:27 /usr/local/Cellar/pyenv/1.2.26/bin/pyenv -> ../libexec/pyenv
List sub-commands recognized by pyenv:
At time of writing:
pyenv 1.2.26 Usage: pyenv <command> [<args>] Some useful pyenv commands are: --version Display the version of pyenv activate Activate virtual environment commands List all available pyenv commands deactivate Deactivate virtual environment exec Run an executable with the selected Python version global Set or show the global Python version(s) help Display help for a command hooks List hook scripts for a given pyenv command init Configure the shell environment for pyenv install Install a Python version using python-build local Set or show the local application-specific Python version(s) prefix Display prefix for a Python version rehash Rehash pyenv shims (run this after installing executables) root Display the root directory where versions and shims are kept shell Set or show the shell-specific Python version shims List existing pyenv shims uninstall Uninstall a specific Python version version Show the current Python version(s) and its origin version-file Detect the file that sets the current pyenv version version-name Show the current Python version version-origin Explain how the current Python version is set versions List all Python versions available to pyenv virtualenv Create a Python virtualenv using the pyenv-virtualenv plugin virtualenv-delete Uninstall a specific Python virtualenv virtualenv-init Configure the shell environment for pyenv-virtualenv virtualenv-prefix Display real_prefix for a Python virtualenv version virtualenvs List all Python virtualenvs found in `$PYENV_ROOT/versions/*'. whence List all Python versions that contain the given executable which Display the full path to an executable See `pyenv help
' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readme
For a full list of sub-commands recognized by pyenv:
At time of writing:
--version commands completions exec global help hooks init install local prefix realpath.dylib rehash root shell shims uninstall version version-file version-file-read version-file-write version-name version-origin versions whence which
Where are pyenv shims installed?
PROTIP: When pyenv is installed, a folder ~/.pyenv is created under your user $HOME folder.
Reveal folders under the pyenv root containing the shims folder:
ls $( pyenv root )
The response shows that the installer created a folder at my user $HOME folder (but your user name instead of my “wilson_mar”):
plugins shims version versions
See the various versions of python (and associated utilities) we want to put at the top of the $PATH:
ls $(pyenv root)/shims
2to3 nltk python 2to3-3.7 pathy python-config 2to3-3.8 pbr python3 __pycache__ pdf2txt.py python3-config bandit pdfplumber python3.7 bandit-baseline pip python3.7-config bandit-config-generator pip3 python3.7-gdb.py chardetect pip3.7 python3.7m distro pip3.8 python3.7m-config dumppdf.py pipenv python3.8 easy_install pipenv-resolver python3.8-config easy_install-3.7 pkginfo python3.8-gdb.py easy_install-3.8 pydoc pyuic6 estimator_ckpt_converter pydoc3 pyvenv f2py pydoc3.7 pyvenv-3.7 f2py3 pydoc3.8 saved_model_cli f2py3.7 pyfiglet spacy get_objgraph pyi-archive_viewer spark google-oauthlib-tool pyi-bindepend tabulate idle pyi-grab_version tensorboard idle3 pyi-makespec tf_upgrade_v2 idle3.7 pyi-set_version tflite_convert idle3.8 pyinstaller toco isort pylupdate6 toco_from_protos jake pyrsa-decrypt tqdm latin2ascii.py pyrsa-encrypt undill macho_dump pyrsa-keygen virtualenv macho_find pyrsa-priv2pub virtualenv-clone macho_standalone pyrsa-sign wheel markdown_py pyrsa-verify
Within the shims folder are every Python command in every installed version of Python—python, pip, etc.
Count files within the shims folder:
find ~/.pyenv/shims -type f | wc -l
How is Pyenv enabled?
Edit your ~/.bash_profile and/or ~/.zshenv file so that the PATH comes before “/usr/local/bin” and “/usr/bin” where others install programs:
# Set it so ~/.pyenv provides Python before others of the same name: export PYENV_ROOT=$(pyenv root) export PATH="$PYENV_ROOT/shims:$PATH"
Put the above at the bottom of the file.
Alternately, if you don’t want to mess with a text editor, use this command:
echo 'export PATH="$(pyenv root)/bin:$PATH"' >> ~/.bash_profile
Instead of .bash_profile, specify ~/.zshrc or ~/.bashrc (for Ubuntu/Fedora).
(Colon characters separate folders in the PATH)
Whether you edited your shell script or not, restart your shell terminal so the path changes take effect:
exec "$SHELL" source ~/.bash_profile
echo "$PATH" >path.txt edit path.txt rm path.txt
Confirm whether pyenv is now active:
Response should be:
(“wilson_mar” would be replaced with your user name)
What Python Releases are available?
Pyenv can install several releases of Python.
PROTIP: Each version of Python is installed within pyenv’s versions folder:
ls $( pyenv root)/versions
If you haven’t already, install the tree utility:
brew install tree
List folders within the versions folder where pyenv installs versions of Python (for example):
tree -d -L 1 $( pyenv root )/versions
/Users/wilson_mar/.pyenv/versions ├── 3.7.7 ├── 3.7.9 ├── 3.8.2 └── 3.8.5
None would be listed until versions are added.
Limit the long list of Python Releases to 3.7 or 3.8 or 3.9:
pyenv install --list | grep " 3\."
At time of writing:
3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 3.7.4 3.7.5 3.7.6 3.7.7 3.7.8 3.7.9 3.7.10 3.8.0 3.8-dev 3.8.1 3.8.2 3.8.3 3.8.4 3.8.5 3.8.6 3.8.7 3.8.8 3.8.9 3.9.0 3.9-dev 3.9.1 3.9.2 3.9.3 3.9.4
PROTIP: Select the last patch number of a release, such as “3.7.7” or “3.8.2” in the example above.
-dev releases are the ones under active development (changes).
Install Python using pyenv
Navigate to the folder containing your Python .py file.
To install a specific version of Python (selected from the response above):
pyenv install 3.7.7
python-build: use firstname.lastname@example.org from homebrew python-build: use readline from homebrew Downloading Python-3.7.7.tar.xz... -> https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tar.xz Installing Python-3.7.7... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.7.7 to /Users/wilson_mar/.pyenv/versions/3.7.7
Note that pyenv installs under a versions folder.
pyenv install 3.8.2
python-build: use email@example.com from homebrew python-build: use readline from homebrew Downloading Python-3.8.2.tar.xz... -> https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz Installing Python-3.8.2... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.8.2 to /Users/wilson_mar/.pyenv/versions/3.8.2
List the versions of Python installed and managed by pyenv:
In the response such as:
* 3.7.7 (set by /Users/wilson_mar/.python-version) 3.7.9 3.8.2 3.8.5
*indicates the current system Python release.
Set Version of Python using Pyenv
To set a specific version as the default:
pyenv global 3.7.9
No response is returned if successful.
To select a specific version:
pyenv local 3.7.7
No response is returned if successful.
PROTIP: The pyenv local command above creates in the folder a hidden file named .python-version containing the Python version specified in the command.
Verify what version.release.patch:
Order of Path override
When a shim (such as “python”) is executed, it calls pyenv which determines which Python release to use by reading sources in this order:
- $PYENV_VERSION environment variable
- pyenv local x.y.z
- pyenv global x.y.z
This means the PYENV_VERSION environment variable overrides all other specifications.
Set this environment variable in your current shell session using the pyenv shell command.
If the PYENV_VERSION system variable is not specified, the application-specific .python-version file in the current directory.
Modify the current directory’s .python-version file with the command:
pyenv local 2.7.15
If there is no .python-version file locally, parent directories are searched upward until reaching the root of your filesystem.
If no .python-version is specified in any folder, the global $(pyenv root)/version file.
Modify the global file using the command:
pyenv global 3.7.7
If the global version file is not present, pyenv assumes you want to use the “system” Python. (In other words, whatever version would run if pyenv weren’t in your PATH.)
Both Pipenv and Pipenv automatically creates and manages a virtualenv for your projects. But only Pipenv adds/removes packages from your Pipfile as you install/uninstall packages. Pipenv generates the Pipfile.lock file used to produce deterministic builds.
Pipenv enables always use the latest versions of dependencies, to minimize security risks arising from outdated components.
pyenv copies an entire Python installation every time a new pyenv version is created.
By contrast, virtualenv makes use of symbolic links, which decreases the size of each virtualenv.
There are two utilities to add virtualenv functionality to pyenv: pyenv-virtualenvwrapper (described by https://alysivji.github.io/setting-up-pyenv-virtualenvwrapper.html) is less convenient and has less stars than the pyenv-virtualenv plugin. To install it:
brew install pyenv-virtualenv
git clone https://github.com/pyenv/pyenv-virtualenv.git \ $(pyenv root)/plugins/pyenv-virtualenv source ~/.bashrc
Cloning into '/Users/wilson_mar/.pyenv/plugins/pyenv-virtualenv'... remote: Enumerating objects: 2064, done. remote: Total 2064 (delta 0), reused 0 (delta 0), pack-reused 2064 Receiving objects: 100% (2064/2064), 580.31 KiB | 30.00 KiB/s, done. Resolving deltas: 100% (1413/1413), done.
This is needed because Virtualenv and Anaconda also activate scripts by mutating $PATH variable of user’s interactive shell, which intercepts pyenv’s shim style command execution hooks.
To automatically activate/deactivate virtualenvs on entering/leaving directories which contain a .python-version file that contains the name of a valid virtual environment as shown in the output of pyenv virtualenvs
echo ‘eval “$(pyenv virtualenv-init -)”’ » ~/.bash_profile or ~/.zshrc
List what virtualenv has been defined
List what virtualenv have been defined:
Setup Virtual environments
There are several ways to enter and setup a virtual environment:
Navigate to where you .py files are. This is important.
Set up a virtual environment in the present working directory, using a common conventional name:
python -m venv venv
No response is returned if it’s all good.
The above creates a virtual folder named after the project.
To deactivate, type:
Alternately, to activate virtualenv, run:
Alternatively, run a command inside the virtualenv
Activate the virtual environment in the present working directory:
This would turn the command prompt as a constant reminder:
The Pipfile.lock file contains a hash of each package installed, which provides security. The file pins semantic versions of all dependencies and sub-dependencies, which provides replicable environments.
If you see this error:
/Users/.../.pyenv/shims/python3: line 21: /usr/local/Cellar/pyenv/1.2.20/libexec/pyenv: No such file or directoryRun*
brew upgrade pyenv
References for pyenv
References for pipenv
https://github.com/VaultVulp/pipenv-alpine/blob/master/Dockerfile is the Python Alpine image with pre-installed pipenv
More about Python
This is one of a series about Python:
- Python install on MacOS
- Python install on MacOS using Pyenv
- Python tutorials
- Python Examples
- Python coding notes
- Pulumi controls cloud using Python, etc.
- Test Python using Pytest BDD Selenium framework
- Test Python using Robot testing framework
- Python REST API programming using the Flask library
- Python coding for AWS Lambda Serverless programming
- Streamlit visualization framework powered by Python
- Web scraping using Scrapy, powered by Python
- Neo4j graph databases accessed from Python
More on OSX
This is one of a series on Mac OSX:
- MacOS Setup step-by-step, with automation
- MacOS Hardware and accessories
- MacOS dotfiles for System Preferences setup automation
- MacOS Boot-up
- MacOS Keyboard tricks
- MacOS Terminal Tips and Tricks
- Text editors and IDEs on MacOS
- MacOS Xcode.app and CommandTools (gcc)
- MacOS Command-line utilities
- Applications on MacOS
- 1password on MacOS
- Manage Disk Space on MacOS
- Screen capture on MacOS
- Windows on Apple MacOS
- Packer create Vagrant Windows image
- Python on MacOS
- Maven on MacOS
- Ruby on MacOS
- Node on MacOS installation
- Java on MacOS
- Scala ecosystem