Python Install (Conda Anaconda Miniconda Pip) on MacOS

Confusion and errors from too many alternatives and options

This tutorial describes the different options to install and uninstall Python and Python packages within various package managers.

Here I’m taking a “deep dive” approach because I haven’t seen one on the internet.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

I’ve pulled out the various incantations suggested by others on StackOverlow and put them here in context.

TL;DR Summary - It’s a mess

The version of Python that comes with Apple MacOS is obsolete and needs to be updated along with Apple XCode CLI for the MacOS version you’re using.

There are two separate versions of Python: 2 and 3. some Python functions in one version do not work with commands in another version.

This has given rise to several versions of Python frameworks being maintained in parallel. For example, both Django 1.3 and Django 1.0 the web application development framework for Python need to be maintained at the same time.

PROTIP: Various methods of installing Python are incompatible with each other. This has given rise to the need for package managers such as pip (Python Installation Packager) that enable one to switch among different versions of Python installed.

This complexity necessitates the packaging of whole virtual environments to isolate within a folder (directory) everything (all dependencies) that each Python project (application) needs to run. This means duplicated files for each Python application, which consume more disk space.

An additional complication is that there are several alternative virtual environment packagers such as easy_install, virtualenv, and pipenv.

• MacOS does not come installed with a package manager for Python until pip.

• pip (Python Installation Packager) is a package manager built on top of setuptools which is what downloads and installs Python packages from the PyPI (Python Package Index) library online at https://pypi.org.

• Setuptools itself is installed using easy_install.

• easy_install is an environment manager.

• Virtualenv (venv) is an environment manager for Python.

• Conda does both, and is language agnostic (not just for Python).

• Miniconda is a lightweight distribution of Conda, and uses conda commands.
• Anaconda is installed on top of miniconda to provide a curated collection of over 720 “common” packages for scientific Python users. It is no longer recommended due to the security vulnerabilities in that many packages. So it’s best to install the base Miniconda and add only the packages you actually use.

Python

Despite all this hassle around versioning, Python is the preferred language of Artificial Intelligence and Machine Learning at the forefront of computer science innovation today. The heavy use of math in AI and ML by TensorFlow means it’s best to install Anaconda and use conda commands (instead of Miniconda or pip with virtualenv).

IPython (Jupyter) Notebook enables a “notebook” interface to re-run commands. See http://sjbyrnes.com/python

PROTIP: PIP install is troublesome, often because they are more recent than those in Conda.

Python Install Options

There is what can be a confusing conflict of choice here for installing Python and its package manager.

• Even though that’s the method described in various (obsolete) websites and books, NOT recommended is the manual approach of download Python installer from python.org: https://www.python.org/downloads/mac-osx
  
  

easy_install is an environment manager. One writes: “Avoid easy_install or pip to install a Python package that needs a library used by non Python programs, such as Qt bindings (PySide)” or Django*.

PROTIP: If you must use easy_install, delete it after.

Alternatively, use a package manager. CAUTION: MacPorts, Fink, and Homebrew do not coexist well on the same machine.

• Install using Homebrew, then add homebrew science for scientific work (according to this).

• MacPorts is an alternative to Homebrew some claim is more compatible with other Linux. However, not all packages are available in MacPorts.
  
  

Use Docker

1. From Dockerhub account “python” get a Docker image on your machine containing Python3 running within Alpine Linux OS:

   docker pull python:3-alpine

2. Build a container containing a small python environment:

   docker container run -d --memory=20m --memory-swap=20m -name myPython python:3-alpine sleep 3600

3. Execute the container:

   docker exec -it myPython /bin/sh

4. Use the container

5. Stop the Docker instance:

   docker stop python:3-alpine

Obsolete Python comes with MacOS

Ever since the Mavericks version of macOS, Python 2 comes installed on MacOS machines.

(Use the index at the right if you want to jump ahead)

1. Open a Terminal shell window and issue command:

   python --version

   The response, for example:

   Python 2.7.19
   

   CAUTION: The sub-version of Python that comes installed with MacOS may be obsolete and needs to be upgraded. But keep to version 2, not version 3 of Python.

2. Find where Python2 is installed:

   type -a python
   

   The response:

   python is /usr/bin/python
   

   You may also see the following if you’ve installed a shim to enable switching of Python versions:

   python is /Users/wilsonmar/.pyenv/shims/python
   python is /usr/bin/python
   

Folders Python on CLI

1. List your present working directory:

   pwd

   We next use a Python interactive command to obtain this again.

2. Open a Python command-line prompt:

   python

   The response shows that version 2 comes with MacOS:

   Python 2.7.12 (default, Oct 16 2016, 19:01:27) 
   [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   

   The response on a new El Capitan machine:

   Python 2.7.6 (default, Sep  9 2014, 15:04:36)
   [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> 
   

   >>>   is the Python interpreter prompt.

   Don’t enter or copy >>> when typing or copying Python commands.

   Current Working Directory

3. Display Python’s current working directory:

   >>> import os
   >>> os.getcwd()

   In the response, the “mac” user name is substituted with your user name:

   '/Users/mac/Downloads'
   

   The above should be the same as the path obtained from pwd before entering Python.

4. Futhermore…

   >>> import os, sys
   >>> os.path.dirname(os.path.abspath(sys.argv[0]))

   The response is your home folder (substitue “mac” with your user name):

   '/Users/mac'
   

   Exit Python

5. Exit the Python interpreter by typing the exit function (with the parentheses symbols):

   >>> exit()
   

   The exit() commmand works the same for both Python 2 and Python 3.

   Using Python 3

6. After installing Python3, obtain the Python 3 command line with:

   python3

   The response I got:

   Python 3.7.5 (default, Nov  1 2019, 02:16:32) 
   [Clang 11.0.0 (clang-1100.0.33.8)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> 
   

Python 3 vs. 2

Since 2018, many say “all new Python code should be written for version 3. There are so many new features in Python 3 that it doesn’t make much sense to stick with Python 2 unless you’re working with old code.”

Most new features introduced with Python 3 are not backwards compatible with version 2.

Where are Python executables?

1. Python3 is installed in a different folder than Python 2.

   type -a python3
   

   The response:

   python3 is /Users/wilsonmar/.pyenv/shims/python3
   python3 is /usr/local/bin/python3
   

2. Get the location where Python is installed:

   which python
   which python3

   • Python v2 is installed in /usr/bin/python
   • Python v3 is installed in /usr/local/bin/python3
     
     

   If you get /Users/user/.pyenv/shims/python3

3. Let’s see what’s there:

   ls -al /usr/local/bin/python3 

   The result shows a link to where Homebrew stores:

   lrwxr-xr-x  1 wilson_mar  admin  34 Nov 30 16:27 /usr/local/bin/python3 -> ../Cellar/python/3.7.5/bin/python3
   

   Python command for Python3

   Some prefer to If you want If you tried to commit suicide like the above, the work-around is an alias, which the operating system resolves before going down PATH.

4. To use Python3 as the default version for the python command, set in you Mac’s ~/.bash_profile:

   alias python=python3
   alias pip=pip3

Print is different in Python3

For the most part, Python 2 code works with Python 3.

Where Python 2 code fails most often is the print statement. Printing in Python 2 is done like so:

print "Hello", "world!"

The response:

Hello world!

If you input the above in Python 3, the response is:

SyntaxError: Missing parentheses in call to ‘print’

This is because Python 3 uses a function:

print("Hello", "world!")

So in Python 2.6+, use the future module to back-port:

from __future__ import print_function
   print("Hello", "world!")
   

Try this:

import sys
   print('Python: {}'.format(sys.version))
   

The response:

Python: 3.5.2 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:52:12) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]
   

Floating point

In Python 2, type in the REPL:

1/2

The response is:

0

In Python3, type:

1/2

The response is:

0.5

Python programming code

1. Download or view a Python program:

   By convention, Python programming source files have a file suffix of .py regardless of the version. This is because, unlike on Windows, Linux looks at the first line of script files to identify what program is used to run the file.

   The “shebang” first line in Python 2 programs start with:

   #!/usr/bin/env python
   

   This is an absolute file system path because the executor doesn’t look on the $PATH for the program defined in ~/.bash_profile.

   The “shebang” first line in Python 3 programs start with:

   #!/usr/bin/env python3
   

   env is an executable file from virtualenv.

Start a HTTP Server Using Python

A simple HTTP server service can be started with this Python 2 command:

python -m SimpleHTTPServer

For Python3:

python3 -m http.server

The response:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...   
   

CAUTION: Hitting Ctrl-C on a Mac, does not shutdown the server gracefully, and the binded address will still be in use.

TODO: Add port designation in command line.

File handling using Pathlib

1. To create a new folder using Python3: TODO:

2. To create a new file using Python3, import the pathlib module’s Path object:

   from pathlib import Path
   Path('empty_file').touch()
   

Don’t Uninstall Default Python on macs

The version of Python that comes with macOS should not be removed because some Apple system software have hard-coded references to it.

In this bad advice to harm yourself:

1. Elevated privilages (sudo) are necessary to remove Python from your Mac

   sudo rm -rf ~/Library/Frameworks/Python.framework/Versions/2.7
   sudo rm -rf "/Applications/Python 2.7"
   

2. Remove symbolic links pointing to the python version:

   cd /usr/local/bin/
   ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework/Versions/2.7' | awk '{print $9}' | tr -d @ | xargs rm
   

3. Remove references to deleted paths in PATH environment variable within shell profile files.
   Depending on which shell you use, any of the following files may have been modified:

   ~/.bash_profile
   ~/.bash_login
   ~/.cshrc
   ~/.profile
   ~/.tcshrc
   ~/.zprofile
   

4. List symbolic links pointing to the python version:

   ls -l /usr/bin/python
   

   On El Capitan, this should display: a sym link such as:

   -rwxr-xr-x  2 root  wheel 58416 Jul 14  2015 /usr/bin/python
   

   Or if instead you followed some bad advice and see something like this:

   lrwxr-xr-x  1 root  wheel  18 Feb  7 20:54 /usr/bin/python -> /usr/bin/python2.7
   

5. List symbolic links pointing to the python version:

   cd ~
   ls -l /usr/bin/python/python2.7
   

   Response:

   lrwxr-xr-x  1 root  wheel  75 Oct  8 10:46 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
   

   The “../../” means that it’s above your HOME folder, in the root of the Mac OS.

6. So let’s go there:

   cd /System/Library/Frameworks/Python.framework/Versions/2.7/bin/
   

   There are the executables “python” and “python2.7” plus others.

7. Run:

   python2.7

   The response:

   Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
   [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> 
   

8. Exit

9. Run the generic python generically:

   python
   

   The response is a newer Python:

   Python 2.7.12 (default, Oct 16 2016, 19:01:27) 
   [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> exit()
   

10. List symbolic links pointing to the Python version:

    ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework/Versions/2.7'
    

    According to this:

Upgrade Python

1. Install XCode CLI.

Download Python installer

If you must do it the hard way, bareback, etc:

1. In an internet browser at
   https://www.python.org/downloads/mac-osx

2. Click the Latest link at the top or a
   specific “Mac OS X 64-bit installer” for macOS 10,9+.

   File	Date	Download
   python-2.7.15-macosx10.9.pkg	2018-05-01	22.7 MB
   python-2.7.12-macosx10.6.pkg	2016-06-25	21.3 MB
   python-2.7.11-macosx10.6.pkg	2015-12-05	21.1 MB
   python-2.7.10-macosx10.6.pkg	2015-05-23	21.1 MB
   python-2.7.09-macosx10.6.pkg	2014-12-10	21.0 MB
   python-2.7.08-macosx10.6.pkg	2014-07-02	19.4 MB
   python-2.7.07-macosx10.6.pkg	2014-05-31	19.4 MB
   python-2.7.06-macosx10.6.pkg	2013-11-10	19.2 MB

   This table shows the growth in download size over time, an analysis unique to this page.

Upgrade pip and setuptools

Many disagree with https://techworm.net/programming/install-pip-python-mac-windows-linux/ which recommends use of easy_install

https://packaging.python.org/discussions/pip-vs-easy-install/ which is part of setuptools.

https://en.wikipedia.org/wiki/Setuptools#EasyInstall easy_install in not a full fledged package installer like pip is.

kinda like a disgraced parent.

Install pip homebrew without setuptools

Instead of following instructions such as this with:

pip install -U pip setuptools

BTW, on Windows it’s:

python -m pip install -U pip setuptools

On Ubuntu 17.04+ Linux:

sudo apt update;
   

On previous Ubuntu versions, see https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa

sudo add-apt-repository ppa:deadsnakes/ppa
   sudo apt-get-update; sudo apt install python3.x
   

1. On a Mac use the Mac system package manager Homebrew to install pip (as recommended by this site and https://pip.readthedocs.io/en/stable/installing):

   brew install pip
   

   Conda installs outside the standard structure, so

2. Run:

   brew doctor
   

   this warning appears (which can be safely ignored):

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
    /Users/mac/miniconda3/bin/python-config
    /Users/mac/miniconda3/bin/python2-config
    /Users/mac/miniconda3/bin/python2.7-config

QUESTION: Is there a way to suppress these messages?

Requirements.txt

In a GitHub repo cloned locally, if you see a file Requirements.txt, it is likely a list of Python packages needed by the application:

# pip install awscli mkdocs
awscli==1.7.26
bcdoc==0.14.0
botocore==0.107.0
certifi==2015.4.28
colorama==0.3.3
docutils==0.12
ghp-import==0.4.1
Jinja2==2.7.3
jmespath==0.7.1
livereload==2.3.2
Markdown==2.6.2
MarkupSafe==0.23
mkdocs==0.14.0
pyasn1==0.1.7
python-dateutil==2.4.2
PyYAML==3.11
rsa==3.1.4
six==1.9.0
tornado==4.1
   

Notice each specific version is specified.

1. In a Terminal shell window, change the directory to where app resides.

2. Just for laughs:

   pip install

   The response:

   You must give at least one requirement to install (see "pip help install")
   

   The use of the word “requirement” in this message is partly why lists of Python package dependencies are in a requirements.txt file.

Di

1. Get a Python 2.7 installed. For example, at:

   ../Cellar/python/2.7.12_2/bin/python2.7

2. Verify the sym link:

   ls -l /usr/local/bin
   

   The response:

   lrwxr-xr-x  1 mac  staff  39 Mar  5 00:41 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.12_2/bin/python2.7
   

3. Edit ~/.bash_shell to add a shell alias:

   alias python=/usr/local/bin/python2.7
   

   alias pip=pip3
   

4. Close and open another Terminal.

5. Verify the version.

   cd ~
   python --version
   

   The response should be the newer sub-version:

   2.7.12 

Virtual Environments

Examples of instructions for installing a requirements.txt file are typically preceded by a source bin/activate command which executes the activate script in the project’s bin folder.

An activate script is placed in each virtual environment established to store different sets of dependencies required by different projects in separate isolated locaations.

This solves the "Project X depends on version 1.x but, Project Y needs 4.x" dilemma, and keeps your global site-packages directory clean and manageable.

• Python Guide

Before the pip install command: Automatically download the packages listed (after you manually change the /path/to)

source bin/activate
   pip install -r /path/to/requirements.txt  
   

This downloads dependencies from PyPI (the Python Package Index), a public repository of software for the Python programming language at https://pypi.python.org/pypi.

However, adding the “--no-index” option would not use it.

NOTE: pip compiles everything from source if a “wheel” is not available for it. Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist), especially when a project contains compiled (C language) extensions.

pip install --use-wheel package

pip scikit-learn

1. In a Terminal on any folder, globally install dependencies libraries:

   pip install -U scikit-learn

2. Edit the Python script to add at the top:

   from sklearn import tree
   

   pip iPython Jupyter

3. iPython is the kernel of Jupyter.

   pip install ipython

Virtualenv and Docker

Hynek Schlawack recommends

• PROTIP: Don’t pip install anything beyond virtualenv into its global site-packages outside of virtualenv.

• Install both virtualenv and system isolation (they are not mutually exclusive):

  • Isolate your application server’s OS from its host using Docker/lxc/jails/zones/kvm/VMware/… to one container/vm per application.

  • inside of them also do isolate your Python environment using virtualenv from unexpected surprises in the system site-packages.

Remove hassles from managing per-project virtualenvs by using one of these, depending on the shell and operating system used:

• virtualenvwrapper
• virtualenvwrapper-win on MS Windows
• virtualfish

PIP (Python Installation Packager)

As of Python 2.7.9 and Python 3.4.x, python.org installers for OS X install pip as well from Activestate.com and download ActivePython. It’s a simple install that gives you both Python and pip.

According to https://www.python.org/download/mac/tcltk/, download from http://www.activestate.com/activetcl/downloads file ActiveTcl8.6.3.1.298624-macosx10.5-i386-x86_64-threaded.dmg

After install, the ActiveTcl User Guide is popped up.

Jesse Noller notes in So you want to use python on the Mac:

“Now, some people may recommend you install Macports or Fink: these are both “sorta” package managers for OS/X, and while I do have Macports installed, I do not use it for Python work. I prefer compilation and self management.”

Easy_install

Others use easy_install (with setuptools) to install packages from the web.

sudo easy_install pip

The response:

Password:
Searching for pip
Best match: pip 8.1.2
Adding pip 8.1.2 to easy-install.pth file
Installing pip script to /Users/mac/miniconda3/bin
Installing pip3.5 script to /Users/mac/miniconda3/bin
Installing pip3 script to /Users/mac/miniconda3/bin
 
Using /Users/mac/miniconda3/lib/python2.7/site-packages
Processing dependencies for pip
Finished processing dependencies for pip
   

Virtual pip environments

The best way to have painless and reproducible deployments is to package whole virtual environments of the application you want to deploy including all dependencies but without configuration.

In the world of Python, an environment is a folder (directory) containing everything that a Python project (application) needs to run in an organised, isolated fashion.

When it is initiated, it automatically comes with its own Python interpreter

• a copy of the one used to create it - alongside its very own pip.

The ability to work with either version 3 or 2.7 on the same machine is needed because, as this MacWorld article points out, Mac Mavericks and Yosemite are installed with Python 2.7, cannot run python3 scripts.

You can work on a Python project which requires Django 1.3
while also maintaining a project which requires Django 1.0.

It’s done by creating isolated Python environments using virtualenv (Virtual python environment builder).

sudo pip install virtualenv

As the reponse requests, activate:

source /usr/local/opt/autoenv/activate.sh

This does not issue a response.

1. Instead of "venv", substitute your project name to to create:

   cd my_project_folder
   virtualenv venv

   Exclude the virtual environment folder from source control by adding it to the git ignore list.

   New python executable in /Users/mac/gits/wilsonmar/shippable/one/bin/python
   Installing setuptools, pip, wheel...
     Complete output from command /Users/mac/gits/wils...pable/one/bin/python - setuptools pip wheel:
     Traceback (most recent call last):
     File "<stdin>", line 4, in <module>
     File "/Users/mac/miniconda3/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
     File "/Users/mac/miniconda3/lib/python2.7/io.py", line 51, in <module>
    import _io
   ImportError: dlopen(/Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
     Referenced from: /Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so
     Expected in: flat namespace
    in /Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so
   ----------------------------------------
   ...Installing setuptools, pip, wheel...done.
   Traceback (most recent call last):
     File "/Users/mac/miniconda3/bin/virtualenv", line 11, in <module>
    sys.exit(main())
     File "/Users/mac/miniconda3/lib/python2.7/site-packages/virtualenv.py", line 711, in main
    symlink=options.symlink)
     File "/Users/mac/miniconda3/lib/python2.7/site-packages/virtualenv.py", line 944, in create_environment
    download=download,
     File "/Users/mac/miniconda3/lib/python2.7/site-packages/virtualenv.py", line 900, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
     File "/Users/mac/miniconda3/lib/python2.7/site-packages/virtualenv.py", line 795, in call_subprocess
    % (cmd_desc, proc.returncode))
   OSError: Command /Users/mac/gits/wils...pable/one/bin/python - setuptools pip wheel failed with error code 1
   

   QUESTION: How to fix this? This occurs when virtualenv was installed with easy_install (or “python setup.py install”)

2. List all virtual environments:

   lsvirtualenv

3. To use a particular Python interpreter:

   virtualenv -p /usr/bin/python2.7 venv

4. Activate your project:

   source venv/bin/activate

   The name of the current virtual environment should now appear on the left of the prompt (e.g. (venv)Your-Computer:your_project UserName$).

   From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

   autoenv

5. To automatically activate an environment when you cd into it:

   brew install autoenv

   The response:

   ==> Downloading https://github.com/kennethreitz/autoenv/archive/v0.1.0.tar.gz
   ==> Downloading from https://codeload.github.com/kennethreitz/autoenv/tar.gz/v0.
   ######################################################################## 100.0%
   ==> Caveats
   To finish the installation, source activate.sh in your shell:
     source /usr/local/opt/autoenv/activate.sh
   ==> Summary
   🍺  /usr/local/Cellar/autoenv/0.1.0: 4 files, 5K, built in 2 seconds
   

6. Install packages as usual, for example:

   pip install request

7. When you are done working in the virtual environment for the moment:

   deactivate

   The above puts you back to the system’s default Python interpreter with all its installed libraries.

8. To delete a virtual environment, delete its folder. In this case, it would be:

   rm -rf venv

9. To keep your environment consistent,

   freeze

   the current state of the environment packages:

   pip freeze > requirements.txt

   This creates (or overwrites) a requirements.txt file containing a simple list of all the packages in the current environment and their respective versions. This file would make it easier to re-create the environment and to install the same packages using the same versions:

   pip install -r requirements.txt

   This ensures consistency across installations, deployments, and developers.

As noted in http://docs.python-guide.org/en/latest/dev/virtualenvs/ this create a folder in the current directory containing the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything; omitting the name will place the files in the current directory instead.

1. Open an internet browser to https://www.python.org/downloads/ and download file python-3.4.2-macosx10.6.pkg.

   See http://docs.python-guide.org/en/latest/dev/virtualenvs/ https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

2. To list what packages have been installed:

   pip list

3. Look for packages by keyword:

   pip search django

Use pip to install Tensorflow

As Siraj shows in his video, on a Mac Terminal, define an environment variable that points to the download URL:

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.10.0rc0-py3-none-any.whl
   

(This is for Python 3 on a Mac.)

Install it using PIP and the variable:

sudo pip3 install --upgrade $TF_BINARY_URL

Vagrant

This explanation forks another.

Setup a provider VM solution to store the image (like VMware, AWS, Hyper-V). https://www.virtualbox.org/wiki/Downloads click “x86/amd64” next to VirtualBox 4.3.20 for OS X hosts to download VirtualBox-4.3.20.96.dmg (109 MB) Read the 368 page User Manual.

Install https://github.com/dotless-de/vagrant-vbguest to keep VirtualBox guest additions up to date. http://schof.org/2014/03/31/working-around-virtualbox-bug-12879/

In the Finder, in the Applications folder, drag and drop it onto your Dock for quicker use later. Double-click on VirtualBox for the VirtualBox Manager.

Download the 224.3 MB vagrant_1.7.1.dmg

The binary gets installed in the Applications folder with a link to the /usr/bin so it is added to the shell path.

List commands:

vagrant
vagrant list-commands

Change directory to where you want to store the Vagrant project and run

vagrant init

The response:

A Vagrantfile has been placed in this directory. You are now ready to vagrant up your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on vagrantup.com for more information on using Vagrant.

ls

This shows Vagrantfile (with capital V).

https://www.vagrantup.com/downloads.html

vagrant up --provider=PROVIDER
   

https://vagrantcloud.com/boxes/search lists boxes created by the community.

I am pulling the box from ATT M2X https://m2x.att.com/developer/sample-code This Repo provides a Vagrant virtual machine that contains several demo applications (Ruby and Python) that report data to AT&T M2X. https://github.com/attm2x/m2x-demo-vagrant

git clone https://github.com/attm2x/m2x-demo-vagrant.git

vagrant box add chef/centos-6.5

bootstrap.bash

User data for Vagrant is filed in the directory from which vagrant was used and is stored in an invisible directory .vagrant.d

Python 2 executables

This advice from 2010.

Path to executables

1. To see what MacOS

   which python

   If you have MiniConda installed:

   /Users/mac/miniconda3/bin/python
   

   If you have Anaconda installed and your user is “mac”:

   /Users/mac/anaconda/bin/python
   

   If you are inside a conda activated environment:

   /Users/mac/anaconda/envs/py2/bin/python
   

2. For a list of what Python executes:

   ls ~/miniconda3/bin

   The response begins with this:

   2to3        openssl        python2-config
   activate    pip         python2.7
   aws         pip2        python2.7-config
   

   Type python

3. Find where you are picking up Python from?

   type python

   If Python was installed:

   python is hashed (/usr/bin/python)

   Alternately, if Conda was installed:

   python is hashed (/Users/mac/miniconda3/bin/python)

4. Open a Terminal shell window and issue command:

   python --version
   

   The response is its version. My Mac Yosemite default of Python shows this:

   Python 2.7.6 (default, Sep  9 2014, 15:04:36)
   [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>>
   

   With Miniconda installed on El Capitan, it’s this instead:

   Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:43:17) 
   [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   Anaconda is brought to you by Continuum Analytics.
   Please check out: http://continuum.io/thanks and https://anaconda.org
   >>>
   

5. Get the folder:

   python -m site --user-site
   

The response I got is this:

/Users/mac/Library/Python/2.7/lib/python/site-packages

Alternately:

/Users/mac/.local/lib/python2.7/site-packages

Anaconda Install

PROTIP: Skip to <a href=#Miniconda-install”>install Miniconda</a> instead because there is so much in Anaconda that there is inevitably vulnerabilities lurking there.

• This video by Corey Schafer explains it well.
• https://docs.continuum.io/anaconda/install

1. Go to web page:

   https://www.continuum.io/downloads

   QUESTION: Is there a brew anaconda?

2. Click on the operating system icon (Mac, Windows, Linux) or scroll down and press the tab.

3. Click to download the “command-line installer”.

   | Version    | File                          | Installer | Installed |
   | :--------- | :---------------------------- | -------: | ----: |
   | Python 3.6 | Anaconda3-4.3.0-MacOSX-x86_64 | 363 MB | - |
   | Python 2.7 | Anaconda2-4.3.0-MacOSX-x86_64 | 358 MB | 1.41 GB |
   

4. In a (bash) Terminal:

   cd Downloads
   chmod 555 ./Anaconda3-4.3.0-MacOSX-x86_64.sh 
   ./Anaconda3-4.3.0-MacOSX-x86_64.sh 
   

   The response:

   Welcome to Anaconda3 4.3.0 (by Continuum Analytics, Inc.)
    
   In order to continue the installation process, please review the license
   agreement.
   Please, press ENTER to continue
   

5. Press Enter (as if you cared).
6. Press Tab until you’re exhausted.

7. Type yes and press Enter.

   Anaconda3 will now be installed into this location: /Users/mac/anaconda3

8. Press ENTER to confirm the location
9. Press CTRL-C to abort the installation

10. Or specify a different location below

11. Press Enter.

    PREFIX=/Users/mac/anaconda3

12. Wait for it to come back to you.

    installing: _license-1.1-py36_1 ...
    installing: alabaster-0.7.9-py36_0 ...
    installing: anaconda-client-1.6.0-py36_0 ...
    installing: anaconda-navigator-1.4.3-py36_0 ...
    installing: appnope-0.1.0-py36_0 ...
    installing: appscript-1.0.1-py36_0 ...
    installing: astroid-1.4.9-py36_0 ...
    installing: astropy-1.3-np111py36_0 ...
    installing: babel-2.3.4-py36_0 ...
    ...
    installation finished.
    Do you wish the installer to prepend the Anaconda3 install location
    to PATH in your /Users/mac/.bash_profile ? [yes|no]
    [yes] >>> yes
    

13. Type yes.

    You may wish to edit your .bashrc or prepend the Anaconda3 install location:
     
    export PATH=/Users/mac/anaconda3/bin:$PATH
     
    Thank you for installing Anaconda3!
     
    Share your notebooks and packages on Anaconda Cloud!
    Sign up for free: https://anaconda.org
    

Miniconda install

1. Automated install from https://github.com/wilsonmar/mac-setup/blob/master/mac-setup.zsh

   ./mac-setup.zsh

2. Setup run-time Environment variables: https://github.com/wilsonmar/mac-setup/blob/master/.zshrc

Below is a more “hands-on” description than what pydata.org and Kyle Purdon offers.

1. Use an internet browser to visit the Miniconda Download page at http://conda.pydata.org/miniconda.html

   Alternately, download:

   cd ~/Downloads
   wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh

2. Select version to download. For various versions of Python:

   Version	File	Size
   Python 3.10	Miniconda3-latest-MacOSX-x86_64.sh	53.9 MB
   Python 3.9	Miniconda3-latest-MacOSX-x86_64.sh	67.8 MB
   Python 3.5	Miniconda3-latest-MacOSX-x86_64.sh	23.4 MB
   Python 2.7	miniconda3-latest-MacOSX-x86_64.sh	20.3 MB

   NOTE: Python3 is not backward compatible with Version 2.

   Notice the “.sh” means these are shell scripts.

3. Open a Terminal shell window to navigate to your Downloads folder and run the Python 2.7 script:

   cd ~/Downloads
   bash miniconda3-latest-MacOSX-x86_64.sh -b

   PROTIP: The “-b” option specifies unattended with defaults.

   The response:

   Welcome to miniconda3 4.0.5 (by Continuum Analytics, Inc.)
    
   In order to continue the installation process, please review the license
   agreement.
   Please, press ENTER to continue
   >>> 
   

4. Press Enter to accept the license.
5. Press “q” to the “:” prompt.

6. Type yes.

   “ERROR: File or directory already exists:” appears if miniconda was already installed.

7. Confirm conda version.

Miniconda Uninstall

1. Install a package to uninstall itself:

   conda install anaconda-clean
   anaconda-clean --yes
   

   find / -name anaconda3 2>/dev/null

2. Delete the anaconda install folder, which is usually under your home dir:

   rm -rf /usr/local/anaconda2
   rm -rf /usr/local/anaconda3
   

Conda Uninstall

1. Install a package to uninstall itself:

   conda install anaconda-clean
   anaconda-clean --yes
   

   find / -name anaconda3 2>/dev/null

2. Delete the anaconda install folder, which is usually under your home dir:

   rm -rf /usr/local/anaconda2
   rm -rf /usr/local/anaconda3
   

Conda version

See https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/

1. For the version of conda installed, specify the upper-case V:

   conda -V

   Alternately:

   conda --version

   The response is like:

   conda 4.11.0
   

Update conda

1. To update miniconda’s version, use the conda command line installed above:

   conda update conda

   The response is a list of packages to be updated if you agree:

   Fetching package metadata: ....
   .Solving package specifications: ..........
    
   Package plan for installation in environment /Users/mac/miniconda3:
    
   The following packages will be downloaded:
    
    package                    |            build
    ---------------------------|-----------------
    sqlite-3.13.0              |                0         1.4 MB
    python-2.7.12              |                1         9.5 MB
    conda-env-2.5.1            |           py27_0          26 KB
    ruamel_yaml-0.11.7         |           py27_0         174 KB
    conda-4.1.8                |           py27_0         201 KB
    ------------------------------------------------------------
                                           Total:        11.2 MB
    
   The following NEW packages will be INSTALLED:
    
    ruamel_yaml: 0.11.7-py27_0
    
   The following packages will be UPDATED:
    
    conda:       4.0.4-py27_0 --> 4.1.8-py27_0 
    conda-env:   2.4.5-py27_0 --> 2.5.1-py27_0 
    python:      2.7.11-0     --> 2.7.12-1     
    sqlite:      3.9.2-0      --> 3.13.0-0     
    
   Proceed ([y]/n)? 
   

2. Press “y” to proceed. A sample response:

   Fetching packages ...
   sqlite-3.13.0- 100% |################################| Time: 0:00:02 483.10 kB/s
   python-2.7.12- 100% |################################| Time: 0:00:20 477.84 kB/s
   conda-env-2.5. 100% |################################| Time: 0:00:00 392.63 kB/s
   ruamel_yaml-0. 100% |################################| Time: 0:00:00 331.95 kB/s
   conda-4.1.8-py 100% |################################| Time: 0:00:00 457.60 kB/s
   Extracting packages ...
   [      COMPLETE      ]|###################################################| 100%
   Unlinking packages ...
   [      COMPLETE      ]|###################################################| 100%
   Linking packages ...
   [      COMPLETE      ]|###################################################| 100%
   

3. After install, close and then re-open the terminal window so the changes can take effect.

4. For a list of packages installed locally (in the currently active environment):

   conda list

   The “py36_1” in the list are pip installed.

   _license                  1.1                      py36_1  
   alabaster                 0.7.9                    py36_0  
   anaconda                  4.3.0               np111py36_0  
   

.bash_profile config

TODO: Instructions for both Miniconda and conda.

The path to Python should be the first in PATH:

1. Open a text editor to ~/.bash_profile and add:

   export PATH=”~/miniconda3/bin:$PATH”
   export PYTHON_PATH=~/miniconda3/bin/python

   When you’re done:

2. Activate the shell file :

   source ~/.bash_profile

   Troubleshoot brew install

3. To reset:

   brew link --overwrite python

   Warning: Already linked: /usr/local/Cellar/python/2.7.12_2
   
   To relink: brew unlink python && brew link python
   

Scripting Python versions

TODO: Use a shim so that a script can display the version of Python it is using:

python3
Python 3.6.7 (default, Nov 20 2018, 22:03:43) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import sys
>>> print(sys.version)  # parentheses necessary in python 3.

Some answers:

3.6.7 (default, Nov 20 2018, 22:03:43) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)]
 
2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)]

Per https://docs.python.org/3/library/sys.html#sys.hexversion

>>> sys.version_info
sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0)
(2, 5, 2, 'final', 0)
 
>>> sys.hexversion
50726896
>>> '%x' % sys.hexversion
'30607f0'
>>> sys.hexversion < 0x02060000
True

For a script to ensure that it’s running the version of Python intended:

#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
  sys.stderr.write("You need python 2.6 or later to run this script\n")
  exit(1)

Alternately:

>>> from platform import python_version
>>> print(python_version())
3.6.7

import platform
from distutils.version import StrictVersion 
assert(StrictVersion(platform.python_version()) >= "2.6")

Conda info

1. Get a list:

   conda info

   The response:

   Current conda install:
    
             platform : osx-64
          conda version : 4.3.9
       conda is private : False
      conda-env version : 4.3.9
    conda-build version : 2.1.2
         python version : 3.5.2.final.0
       requests version : 2.12.4
       root environment : /Users/mac/anaconda  (writable)
    default environment : /Users/mac/anaconda
       envs directories : /Users/mac/anaconda/envs
          package cache : /Users/mac/anaconda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/osx-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False
             user-agent : conda/4.3.9 requests/2.12.4 CPython/3.5.2 Darwin/16.4.0 OSX/10.12.3
                UID:GID : 501:20
       

   Conda environments

2. Get a list of Conda environments (from any folder) using the -e flag:

   conda info -e

   Alternately, if you like typing long options:

   conda info --env
   

   The response are like this:

   tensorflow_env           /Users/mac/anaconda/envs/tensorflow_env
   root                  *  /Users/mac/anaconda
   

3. Create a Conda environment named py2 using Python 2.7:

   conda create -n py2 python=2.7 anaconda
   

4. Press y to go ahead.

5. Add python packages, such as TensorFlow:

   conda install -n yourenvname tensorflowp
   conda install -c conda-forge tensorflow
   

6. Activate to use the environment:

   source activate
   

7. When done using TensorFlow, deactivate the environment:

   source deactivate
   

Conda pyenv

pyenv enables switch between multiple versions of Python on a single system (laptop).

1. Create two “named” Conda environments (one with Python2 and the other with Python3):

   conda create -n py3 python=3*
   conda create -n py2 python=2*
   
   
   

2. Set one of these as my default by adding to my terminal startup file ~/.bash_profile:

   source activate py3 
   

   Typically I only use these “named python” environments to run a Python REPL or do general Python tasks. I’ll create another conda environment named specifically for each real project I work on.

Uninstall

PROTIP: Delete Conda one folder at a time (without the –yes parameter).

anaconda-clean

Files such as:

.conda
   Backup directory: /Users/mac/.anaconda_backup/2017-03-07T051620
   .continuum
   .idlerc
   .ipython
   .jupyter
   .matplotlib
   

http://www.numericalexpert.com/blog/conda_tips

Install Python packages

From inside a conda environment:

1. NumPy at http://www.numpy.org needed by http://www.pymvpa.org/installation.html

   pip install numpy

2. Instead of downloading http://www.scipy.org/scipylib/download.html# linear algebra, standard distributions, signal processing, data IO

   pip install scipy

3. SKlearn

   pip install sklearn

4. Pandas based on VIDEO: How to install Pandas Miniconda:

   pip install pandas

   YouTube videos: Learn Pandas

5. matplotlib

   Other Python packages:

   • xlwings interfaces with Microsoft Excel spreadsheets
   • pygame develops GUI

OpenCV for computer vision

Follow the instructions below to install python2 + OpenCV in mac

http://www.pyimagesearch.com/2016/11/28/macos-install-opencv-3-and-python-2-7/

In case of python3 + OpenCV follow

http://www.pyimagesearch.com/2016/12/05/macos-install-opencv-3-and-python-3-5

Conda vs Pip

• Conda handles library dependencies outside of the Python packages as well as Python packages themselves.

• Conda installs from binary, meaning that someone (e.g., Continuum) has already done the hard work of compiling the package, making installation easier, and faster.

• pip can install anything from PyPI in one command.

• Conda requires at least three commands: skeleton, build, install, and possibly more.

• Conda uses its own format, which has some advantages (like being static, and again, Python agnostic).

Conda provides an alternative set of commands popular for scientific (Machine Learning) computing. See http://conda.pydata.org/docs/_downloads/conda-cheatsheet.pdf

This table lists the difference in commands between Conda and pip, a summary of the more detailed table is at https://conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands

PyCharm IDE

$195/year PyCharm is purpose-built by Jetbrains.com for Python coding:

1. VIDEO: Read:

   https://www.jetbrains.com/pycharm-edu/download/#section=mac

2. Install using Homebrew:

   brew install pycharm
   

3. Add Terraform and HCL plugin ( Windows)

   To run Terraform from IDE and autocompletion, rename refactoring, etc.

4. Configure in Settings -> Tools -> Terraform

   Explicitly add Terraform run configurations

   https://discussions.udacity.com/t/referencing-pygame-from-pycharm-with-anaconda/223711/13

5. Run Terraform from IDE File context menu -> Run…

   Alternately, click green ‘run’ next to resources

Turi (Dato) Python algorithms

GraphLab Create from Dato provides scalable “pre-implemented” ML algorithms using Anaconda. Entire courses on its use is at

• https://www.coursera.org/learn/ml-foundations
• https://www.turi.com/learn/userguide/
• https://www.turi.com/products/create/docs/
• https://github.com/learnml/machine-learning-specialization
• https://www.coursera.org/learn/ml-clustering-and-retrieval/supplement/iF7Ji/software-tools-you-ll-need-for-this-course

When the one-year free license is over, note scikit-learn also uses Python with Anaconda.

Python libraries

For matrix operations, use the Numpy open-source Python library for fast performance with data that fits in memory. Quickstart.

In a requirements.txt file:

tweepy (http://www.tweepy.org)

csv (https://pypi.python.org/pypi/csv)

textblob (https://textblob.readthedocs.io/en/dev/)

keras (https://keras.io)

bokeh Flask ipython jupyter matplotlib nose numpy pandas Pillow pymc requests scikit-image scikit-learn scipy seaborn statsmodel tensorflow virtualenv virtualenvwrapper

OpenCV3

brew tap homebrew/science && brew update
brew install opencv3 --with-contrib --with-python3 --without-python
ln -s /usr/local/opt/opencv3/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so /usr/local/lib/python3.6/site-packages/cv2.so
export PYTHONPATH=$PYTHONPATH:/usr/local/Cellar/opencv3/3.2.0/lib/python3.6/site-packages
   

Data Manipulation

SFrame is an open-source, highly-scalable Python library for data manipulation. Unlike Pandas, SFrame is not limited to datasets which can fit in memory, so it can deal with large datasets, even on a laptop.

http://stackoverflow.com/questions/509211/explain-pythons-slice-notation/

Conda

Conda is similar to virtualenv and pyenv, other popular environment managers.

https://virtualenv.pypa.io/en/stable/

https://github.com/yyuu/pyenv

https://www.continuum.io/downloads

conda install numpy pandas matplotlib

conda install jupyter notebook

1. List the packages installed, with its version number and what version of Python:

   conda list

Conda Environments

1. Create new environment for Python, specifying packages needed:

   conda create -n my_env python=3 numpy pandas

2. Enter an environment on Mac:

   source activate my_env

   On Windows:

   activate my_env

   When you’re in the environment, the environment’s name appears in the prompt:

   (my_env) ~ $.

3. Leave the environment

   source deactivate

   On Windows, it’s just deactivate.

4. Get back in again.

5. Create an environment file by piping the output from an export:

   conda env export > some_env.yaml

   When sharing your code on GitHub, it’s good practice to make an environment file and include it in the repository. This will make it easier for people to install all the dependencies for your code. I also usually include a pip requirements.txt file using pip freeze (learn more here) for people not using conda.

6. Load an environment metadata file:

   conda env create -f some_env.yaml

7. List environments created on your machine:

   conda env list

8. Remove an environment:

   conda env remove -n some_env

Read:

• https://conda.io/docs/using/index.html

• https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/

Where installed?

When Python is installed using pip, see where it’s installed:

python3 -m site --user-base

The response on my system:

/Users/[myusername]/Library/Python/3.7

Miscellaneous

http://stackoverflow.com/questions/990754/how-to-leave-exit-deactivate-a-python-virtualenv?rq=1

https://gist.github.com/alyssaq/f60393545173379e0f3f describes install of https://bootstrap.pypa.io/get-pip.py

https://docs.python.org/3/library/2to3.html 2to3 is a Python program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code.

References

https://joernhees.de/blog/2014/02/25/scientific-python-on-mac-os-x-10-9-with-homebrew/ recommends several pip libraries

https://linuxacademy.com/cp/socialize/index/type/community_post/id/14209 released 1/25/2017 by Michael Jenkins makes use of AWS Boto3 with Python2 to create S3 buckets, upload files, and delete S3 buckets in code examples as part of Linux Academy’s Red Hat Certified Specialist in Virtualization (EX318) Preparation video Course

Sikuli Python

1. To use SikuliX from real Python via py4j

   cd /usr/local/
   git clone https://github.com/RaiMan/sikulix4python
   cd sikulix4python
   

2. Get support on the Python:

   pip install py4j

   Collecting py4j
     Downloading https://files.pythonhosted.org/packages/04/de/2d314a921ef4c20b283e1de94e0780273678caac901564df06b948e4ba9b/py4j-0.10.8.1-py2.py3-none-any.whl (196kB)
     |████████████████████████████████| 204kB 402kB/s 
   Installing collected packages: py4j
   Successfully installed py4j-0.10.8.1
   

3. Download the Jython interpreter for Python scripting (the default) jython-standalone-2.7.1.jar

   curl -O https://repo1.maven.org/maven2/org/python/jython-standalone/2.7.1/jython-standalone-2.7.1.jar

4. Download the sikulixapi for use by Maven:

   https://launchpad.net/sikuli/sikulix/2.0.1/+download/sikulixapi-2.0.1.jar

Resources

macos-install-all/tests/firefox_pycon_search.py

https://blog.xoxzo.com/en/2020/05/09/making-sense-python-environment/

VIDEO COURSE: EC-Council’s CodeRed “Black Hat Python: Python For Pentesters” by Cody Jackson

More on Python

This is one of a series about Python:

1. Python install on MacOS
2. Python install on MacOS using Pyenv

3. Python install on Raspberry Pi for IoT

4. Python tutorials
5. Python Examples
6. Python coding notes
7. Pulumi controls cloud using Python, etc.

8. Jupyter Notebooks provide commentary to Python

9. Python certifications

10. Test Python using Pytest BDD Selenium framework
11. Test Python using Robot testing framework

12. Testing AI uses Python code

13. Microsoft Azure Machine Learning makes use of Python

14. Python REST API programming using the Flask library
15. Python coding for AWS Lambda Serverless programming
16. Streamlit visualization framework powered by Python
17. Web scraping using Scrapy, powered by Python
18. Neo4j graph databases accessed from Python