Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

Confusion and errors from many alternatives and options


Overview

This tutorial describes the different options to install and uninstall Python within various package managers (which helps you find and install Python packages).

Here I’m taking a “deep dive” approach because I haven’t seen one on the internet.

I’ve pulled out the various incantations suggested by others on StackOverlow and put them here in context.

I want you to feel confident that you’ve mastered this skill. That’s why this takes a hands-on approach where you type in commands and we explain the responses and possible troubleshooting. This is a “deep dive” because all details are presented.

Like a good music DJ, I’ve carefully arranged the presentation of concepts into a sequence for easy learning, so you don’t have to spend as much time as me making sense of the flood of material around this subject.

Sentences that begin with PROTIP are a high point of this website to point out wisdom and advice from experience. NOTE point out observations that many miss. Search for them if you only want “TL;DR” (Too Long Didn’t Read) highlights.

Stuck? Contact me and I or one of my friends will help you.

TL;DR Summary

PROTIP: Various methods of installing Python are incompatible with each other.

There are two separate versions of Python such that some Python functions in one version do not work with commands in another version.

The version of Python that comes with Apple Mac OSX is obsolete and needs to be updated with XCode.

Version 2 of Python must be the default Python command for Apple’s use.

Despite all this hassle, Python is the preferred language of Artificial Intelligence and Machine Learning at the forefront of computer science innovation today.

The heavy use of math in AI and ML means it’s best to install Anaconda and use conda commands (instad of Miniconda or pip with virtualenv). These provide for custom virtual environments that each contain a desired version of Python (2.7 vs 3.6) for a specific purpose/project, plus specific versions of add-on Python packages (such as TensorFlow).

PIP install is troublesome, often because they are more recent than those in Conda.

Python Install Options

There is what can be a confusing conflict of choice here for installing Python and its package manager.

The “traditional” approach:

Not recommended. That would be too easy.

  • pip (Python Installation Packager) is built on top of setuptools which is what downloads and installs Python software over a network (usually the Internet) with a single command (easy_install). It itself is installed using easy_install.

  • easy_install is an environment manager.

One writes: “Avoid easy_install or pip to install a Python package that needs a library used by non Python programs, such as Qt bindings (PySide)”.

Alternatively, there are these alternatives:

  • Conda is the command-line interface (CLI) tool that combines functionality of pip and virtualenv

  • Miniconda is a lightweight distribution of Conda.

So miniconda users also use conda commands.

  • Anaconda contains a curated collection of over 720 “common” packages for scientific Python users. It goes on top of miniconda.

  • Install using Homebrew, then add homebrew science for scientific work (according to this).

  • MacPorts is an alternative to Homebrew that is more compatible with other Linux. However, not all packages are available in MacPorts.

CAUTION: MacPorts, Fink, and Homebrew do not coexist on a single machine.

Differences among them:

  • Pip is a package manager.
  • Virtualenv is an environment manager.
  • Conda is both.

  • Conda handles library dependencies outside of the Python packages as well as the Python packages themselves.

  • Conda installs from binary, meaning that someone (e.g., Continuum) has already done the hard work of compiling the package, making installation easier, and faster.

  • pip can install anything from PyPI in one command.
  • Conda requires at least three commands: skeleton, build, install, and possibly more

  • Conda uses its own format, which has some advantages (like being static, and again, Python agnostic).

Conda provides an alternative set of commands popular for scientific (Machine Learning) computing.

http://conda.pydata.org/docs/_downloads/conda-cheatsheet.pdf

This table lists the difference in commands between Conda and pip:

Task Conda package and environment manager command Pip package manager command Virtualenv environment manager command
Install a package conda install $PACKAGE_NAME pip install $PACKAGE_NAME -
Update a package conda update --name $ENVIRONMENT_NAME $PACKAGE_NAME pip install --upgrade $PACKAGE_NAME -
Update package manager conda update conda Linux/OSX: pip install -U pip Win: python -m pip install -U pip -
Uninstall a package conda remove --name $ENVIRONMENT_NAME $PACKAGE_NAME pip uninstall $PACKAGE_NAME -
Create an environment conda create --name $ENVIRONMENT_NAME python - cd $ENV_BASE_DIR; virtualenv $ENVIRONMENT_NAME
Activate an environment source activate $ENVIRONMENT_NAME - source $ENV_BASE_DIR/$ENVIRONMENT_NAME
/bin/activate
Deactivate an environment source deactivate - deactivate
Search available packages conda search $SEARCH_TERM pip search $SEARCH_TERM -
Install package from specific source conda install --channel $URL $PACKAGE_NAME pip install --index-url $URL $PACKAGE_NAME -
List installed packages conda list --name $ENVIRONMENT_NAME pip list -
Create requirements file conda list --export pip freeze -
List all environments conda info --envs - Install virtualenv wrapper,
then lsvirtualenv
Install other package manager conda install pip pip install conda -
Install Python conda install python=x.x - -
Update Python conda update python * - -

Python comes with Mac OSX

Ever since the Mavericks version of Mac OSX, Python 2 comes installed on MacOS machines.

(Use the index at the right if you want to jump ahead)

  1. Open a Terminal shell window and issue command:

    python --version

    The response on a freshly installed El Capitan version:

    Python 2.7.6
    

    CAUTION: The sub-version that comes installed with MacOS may be obsolete and needs to be upgraded. But keep to version 2, not version 3 of Python.

Interactive Python on CLI

  1. List your present working directory:

    pwd

    We next use a Python interactive command to obtain this again.

  2. Open a Python command-line prompt:

    python

    The response:

    Python 2.7.12 (default, Oct 16 2016, 19:01:27) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    

    The response on a new El Capital machine:

    Python 2.7.6 (default, Sep  9 2014, 15:04:36)
    [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 
    

    >>> is the Python interpreter prompt.

    Don’t enter or copy >>> when typeing or copying Python commands.

    Current Working Directory

  3. Display Python’s current working directory:

    >>> import os
    >>> os.getcwd()

    In the response, substitue my “mac” user name with yours:

    '/Users/mac/Downloads'
    

    The above should be the same as the path obtained from pwd before entering Python.

  4. Futhermore…

    >>> import os, sys
    >>> os.path.dirname(os.path.abspath(sys.argv[0]))

    The response is your home folder (substitue “mac” with your user name):

    ‘/Users/mac’

    Exit Python

  5. Exit the Python interpreter by typing the exit function (with the parentheses symbols) :

    exit()

    NOTE: MacOS does not come installed with a package manager for Python.

    IPython (Jupyter) Notebook enables a “notebook” interface to re-run commands. http://sjbyrnes.com/python/ notes

  6. Enter Python again for the instructions to follow.

    Start a HTTP Server Using Python

    A simple HTTP server service can be started with command:

    python -m SimpleHTTPServer

    For Python3:

    python3 -m http.server

    The response:

    Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...   
    

Don’t Uninstall Python 2.7

The version of Python that comes with Mac OSX should not be removed because some Apple system software have hard-coded references to it.

That is the reason why elevated privilages (sudo) is necessary to remove Python from your Mac:

Nevertheless here’s the bad advice to harm yourself:


   sudo rm -rf ~/Library/Frameworks/Python.framework/Versions/2.7
   sudo rm -rf "/Applications/Python 2.7"
   

Remove symbolic links pointing to the python version see


   cd /usr/local/bin/
   ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework/Versions/2.7' | awk '{print $9}' | tr -d @ | xargs rm
   

Remove references to deleted paths in PATH environment variable within shell profile files.
Depending on which shell you use, any of the following files may have been modified:

   ~/.bash_login, 
   ~/.bash_profile, 
   ~/.cshrc, 
   ~/.profile, 
   ~/.tcshrc,
   ~/.zprofile
   
  1. List symbolic links pointing to the python version:

    
    ls -l /usr/bin/python
    

    On El Capitan, this should display: a sym link such as:

    -rwxr-xr-x  2 root  wheel 58416 Jul 14  2015 /usr/bin/python
    

    If instead you followed some bad advice and see something like this:

    lrwxr-xr-x  1 root  wheel  18 Feb  7 20:54 /usr/bin/python -> /usr/bin/python2.7
    
  2. List symbolic links pointing to the python version:

    
    cd ~
    ls -l /usr/bin/python/python2.7
    

    Response:

    lrwxr-xr-x  1 root  wheel  75 Oct  8 10:46 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
    

    The “../../” means that it’s above your HOME folder, in the root of the Mac OS.

  3. So let’s go there:

     
    cd /System/Library/Frameworks/Python.framework/Versions/2.7/bin/
    

    There are the executables “python” and “python2.7” plus others.

  4. Run:

    python2.7

    The response:

    Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 
    
  5. Exit

  6. Run the generic python generically:

    
    python2.7
    

    The response is a newer Python:

    Python 2.7.12 (default, Oct 16 2016, 19:01:27) 
    [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> exit()
    
  7. List symbolic links pointing to the Python version:

    
    ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework/Versions/2.7'
    

    According to this:

    Alias to the rescue

    CAUTION: I’m still working on the following:

    If you tried to commit suicide like the above, the work-around is an alias, which the operating system resolves before going down PATH.

  8. Get a Python 2.7 installed. For example, at:

    ../Cellar/python/2.7.12_2/bin/python2.7

  9. Verify the sym link:

    
    ls -l /usr/local/bin
    

    The response:

    lrwxr-xr-x  1 mac  staff  39 Mar  5 00:41 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.12_2/bin/python2.7
    
  10. Edit ~/.bash_shell to add a shell alias:

    alias python=/usr/local/bin/python2.7
    
  11. Close and open another Terminal.
  12. Verify the version.

    
    cd ~
    python --version
    

    The response should be the newer sub-version:

    2.7.12 

XCode

To build Python on a machine requires a GCC compiler. One comes with command-line tools installed with Apple’s XCode IDE. Newer versions also installs a Git client.

XCode install

  1. Get the installation location in a Terminal window:

    
    xcode-select -p
    

    The answer:

    /Applications/Xcode.app/Contents/Developer
    

It used to be that one can enter a command:

xcode-select --install

The response on my Sierra machine is:

   xcode-select: error: command line tools are already installed, use "Software Update" to install updates
   

So below is the “Software Update” approach:

  1. Use an internet browser to https://developer.apple.com/xcode

  2. Provide your Apple ID and password. Get one if you don’t already have one.

  3. Go through Apple’s location verification if prompted.

  4. Click “Download” and provide your Apple ID. You’ll need to establish an Apple ID.

  5. Select the version to download:

    File Date Download Unpacked
    XCode_8.2.1 8C1002 2016-12-19 ? GB ? GB
    XCode_8_beta_2.xip 2016-07-05 5.9 GB 12.32 GB
    XCode_7.31 2016-05-03 3.8 GB ?
    XCode_4.1 2014- 2.9 GB ?
    XCode_3.2.4 2014- 2.? GB ?

    NOTE: These are massive files that may take a while to download if you don’t have a fast internet connection.

    CAUTION: Even more important, make sure that your machine will have enough free space available.

  6. Open App Store. Click Open.

  7. Remember to delete the installer after you’re done, then

    XCode version

  8. Confirm the version installed.

    
    /usr/bin/xcodebuild -version
    

    The answer:

    Xcode 8.2.1
    Build version 8C1002
    

    The should match up with the Build Number on the Apple web page.

    Alternately, for a more precise version number and other info (Mavericks and up):

    
    pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
    

    This is a specific version of:

    pkgutil --pkgs | grep -i tools
    

    The response:

    package-id: com.apple.pkg.CLTools_Executables
    version: 8.2.0.0.1.1480973914
    volume: /
    location: /
    install-time: 1484969093
    groups: com.apple.FindSystemFiles.pkg-group 
    
  9. Get the version of GCC installed:

    gcc --version
    

    The answer:

    Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
    Apple LLVM version 8.0.0 (clang-800.0.42.1)
    Target: x86_64-apple-darwin16.4.0
    Thread model: posix
    InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
    

Download Python installer

If you must do it the hard way, bareback, etc:

  1. In an internet browser at
    https://www.python.org/downloads/mac-osx

  2. Click the Latest link at the top or a
    specific “Mac OS X 64-bit/32-bit installer”.

    File Date Download
    python-2.7.12-macosx10.6.pkg 2016-06-25 21.3 MB
    python-2.7.11-macosx10.6.pkg 2015-12-05 21.1 MB
    python-2.7.10-macosx10.6.pkg 2015-05-23 21.1 MB
    python-2.7.09-macosx10.6.pkg 2014-12-10 21.0 MB
    python-2.7.08-macosx10.6.pkg 2014-07-02 19.4 MB
    python-2.7.07-macosx10.6.pkg 2014-05-31 19.4 MB
    python-2.7.06-macosx10.6.pkg 2013-11-10 19.2 MB

    This table shows the growth in download size over time, an analysis unique to this page.


Upgrade pip and setuptools

Install pip homebrew without setuptools

Instead of following instructions such as this with:

pip install -U pip setuptools

BTW, on Windows it’s:

python -m pip install -U pip setuptools

  1. On a Mac use the Mac system package manager Homebrew to install pip (as recommended by this site and https://pip.readthedocs.io/en/stable/installing):

    brew install pip

    Conda installs outside the standard structure, so

  2. Run:

    brew doctor

    this warning appears (which can be safely ignored):

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
    /Users/mac/miniconda2/bin/python-config
    /Users/mac/miniconda2/bin/python2-config
    /Users/mac/miniconda2/bin/python2.7-config

QUESTION: Is there a way to suppress these messages?

Requirements.txt

In a GitHub repo cloned locally, if you see a file Requirements.txt, it is likely a list of Python packages needed by the application:

# pip install awscli mkdocs
awscli==1.7.26
bcdoc==0.14.0
botocore==0.107.0
certifi==2015.4.28
colorama==0.3.3
docutils==0.12
ghp-import==0.4.1
Jinja2==2.7.3
jmespath==0.7.1
livereload==2.3.2
Markdown==2.6.2
MarkupSafe==0.23
mkdocs==0.14.0
pyasn1==0.1.7
python-dateutil==2.4.2
PyYAML==3.11
rsa==3.1.4
six==1.9.0
tornado==4.1
   

Notice each specific version is specified.

  1. In a Terminal shell window, change the directory to where app resides.

  2. Just for laughs:

    pip install </strong>

    The response:

    You must give at least one requirement to install (see "pip help install")
    

    The use of the word “requirement” in this message is partly why lists of Python package dependencies are in a requirements.txt file.

    Virtual Environments

    Examples of instructions for installing a requirements.txt file are typically preceded by a source bin/activate command which executes the activate script in the project’s bin folder.

    An activate script is placed in each virtual environment established to store different sets of dependencies required by different projects in separate isolated locaations.

    This solves the "Project X depends on version 1.x but, Project Y needs 4.x" dilemma, and keeps your global site-packages directory clean and manageable.


before the pip install command: Automatically download the packages listed (after you manually change the /path/to)

source bin/activate
pip install -r /path/to/requirements.txt
</strong>

This downloads dependencies from PyPI (the Python Package Index), a public repository of software for the Python programming language at https://pypi.python.org/pypi.

However, adding the “--no-index” option would not use it.

NOTE: pip compiles everything from source if a “wheel” is not available for it. Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist), especially when a project contains compiled (C language) extensions.

pip install --use-wheel package

### pip scikit-learn #

  1. In a Terminal on any folder, globally install dependencies libraries:

    pip install -U scikit-learn

  2. Edit the Python script to add at the top:

    from sklearn import tree
    

    pip iPython Jupyter

  3. iPython is the kernel of Jupyter.

    pip install ipython


Virtualenv and Docker

Hynek Schlawack recommends

  • Don’t pip-install anything into its global site-packages beyond virtualenv.

  • Install both virtualenv and system isolation (they are not mutually exclusive):

    • Isolate your application server’s OS from its host using Docker/lxc/jails/zones/kvm/VMware/… to one container/vm per application.

    • inside of them also do isolate your Python environment using virtualenv from unexpected surprises in the system site-packages.

Remove hassles from managing per-project virtualenvs by using one of these, depending on the shell and operating system used:

  • virtualenvwrapper
  • virtualenvwrapper-win on MS Windows
  • virtualfish

PIP (Python Installation Packager)

As of Python 2.7.9 and Python 3.4.x, python.org installers for OS X install pip as well from Activestate.com and download ActivePython. It’s a simple install that gives you both Python and pip.

According to https://www.python.org/download/mac/tcltk/, download from http://www.activestate.com/activetcl/downloads file ActiveTcl8.6.3.1.298624-macosx10.5-i386-x86_64-threaded.dmg

After install, the ActiveTcl User Guide is popped up.

Jesse Noller notes in So you want to use python on the Mac:

“Now, some people may recommend you install Macports or Fink: these are both “sorta” package managers for OS/X, and while I do have Macports installed, I do not use it for Python work. I prefer compilation and self management.”

Easy_install

Others use easy_install (with setuptools) to install packages from the web.

sudo easy_install pip

The response:

Password:
Searching for pip
Best match: pip 8.1.2
Adding pip 8.1.2 to easy-install.pth file
Installing pip script to /Users/mac/miniconda2/bin
Installing pip3.5 script to /Users/mac/miniconda2/bin
Installing pip3 script to /Users/mac/miniconda2/bin
 
Using /Users/mac/miniconda2/lib/python2.7/site-packages
Processing dependencies for pip
Finished processing dependencies for pip
   

Virtual pip environments

The best way to have painless and reproducible deployments is to package whole virtual environments of the application you want to deploy including all dependencies but without configuration.

In the world of Python, an environment is a folder (directory) containing everything that a Python project (application) needs to run in an organised, isolated fashion.

When it is initiated, it automatically comes with its own Python interpreter

  • a copy of the one used to create it - alongside its very own pip.

The ability to work with either version 3 or 2.7 on the same machine is needed because, as this MacWorld article points out, Mac Mavericks and Yosemite are installed with Python 2.7, cannot run python3 scripts.

You can work on a Python project which requires Django 1.3
while also maintaining a project which requires Django 1.0.

It’s done by creating isolated Python environments using virtualenv (Virtual python environment builder).

sudo pip install virtualenv

As the reponse requests, activate:

source /usr/local/opt/autoenv/activate.sh

This does not issue a response.

  1. Instead of "venv", substitute your project name to to create:

    cd my_project_folder
    virtualenv venv

    Exclude the virtual environment folder from source control by adding it to the git ignore list.

    New python executable in /Users/mac/gits/wilsonmar/shippable/one/bin/python
    Installing setuptools, pip, wheel...
      Complete output from command /Users/mac/gits/wils...pable/one/bin/python - setuptools pip wheel:
      Traceback (most recent call last):
      File "", line 4, in 
      File "/Users/mac/miniconda2/lib/python2.7/tempfile.py", line 32, in 
     import io as _io
      File "/Users/mac/miniconda2/lib/python2.7/io.py", line 51, in 
     import _io
    ImportError: dlopen(/Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
      Referenced from: /Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so
      Expected in: flat namespace
     in /Users/mac/gits/wilsonmar/shippable/one/lib/python2.7/lib-dynload/_io.so
    ----------------------------------------
    ...Installing setuptools, pip, wheel...done.
    Traceback (most recent call last):
      File "/Users/mac/miniconda2/bin/virtualenv", line 11, in 
     sys.exit(main())
      File "/Users/mac/miniconda2/lib/python2.7/site-packages/virtualenv.py", line 711, in main
     symlink=options.symlink)
      File "/Users/mac/miniconda2/lib/python2.7/site-packages/virtualenv.py", line 944, in create_environment
     download=download,
      File "/Users/mac/miniconda2/lib/python2.7/site-packages/virtualenv.py", line 900, in install_wheel
     call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
      File "/Users/mac/miniconda2/lib/python2.7/site-packages/virtualenv.py", line 795, in call_subprocess
     % (cmd_desc, proc.returncode))
    OSError: Command /Users/mac/gits/wils...pable/one/bin/python - setuptools pip wheel failed with error code 1
    </pre>
    
    QUESTION: How to fix this?
    This occurs when virtualenv was installed with easy_install (or "python setup.py install")
    
    
  2. List all virtual environments:

    lsvirtualenv

  3. To use a particular Python interpreter:

    virtualenv -p /usr/bin/python2.7 venv

  4. Activate your project:

    source venv/bin/activate

    The name of the current virtual environment should now appear on the left of the prompt (e.g. (venv)Your-Computer:your_project UserName$).

    From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

    autoenv

  5. To automatically activate an vironment when you cd into it:

    brew install autoenv

    The response:

    ==> Downloading https://github.com/kennethreitz/autoenv/archive/v0.1.0.tar.gz
    ==> Downloading from https://codeload.github.com/kennethreitz/autoenv/tar.gz/v0.
    ######################################################################## 100.0%
    ==> Caveats
    To finish the installation, source activate.sh in your shell:
      source /usr/local/opt/autoenv/activate.sh
    ==> Summary
    🍺  /usr/local/Cellar/autoenv/0.1.0: 4 files, 5K, built in 2 seconds
    
  6. Install packages as usual, for example:

    pip install request

  7. When you are done working in the virtual environment for the moment:

    deactivate

    The above puts you back to the system’s default Python interpreter with all its installed libraries.

  8. To delete a virtual environment, delete its folder. In this case, it would be:

    rm -rf venv

  9. To keep your environment consistent,

    freeze

    the current state of the environment packages:

    pip freeze > requirements.txt

    This creates (or overwrites) a requirements.txt file containing a simple list of all the packages in the current environment and their respective versions. This file would make it easier to re-create the environment and to install the same packages using the same versions:

    pip install -r requirements.txt

    This ensures consistency across installations, deployments, and developers.

As noted in http://docs.python-guide.org/en/latest/dev/virtualenvs/ this create a folder in the current directory containing the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything; omitting the name will place the files in the current directory instead.

  1. Open an internet browser to https://www.python.org/downloads/ and download file python-3.4.2-macosx10.6.pkg.

    See http://docs.python-guide.org/en/latest/dev/virtualenvs/ https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

  2. To list what packages have been installed:

    pip list

  3. Look for packages by keyword:

    pip search django

Use pip to install Tensorflow

As Siraj shows in his video, on a Mac Terminal, define an environment variable that points to the download URL:

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.10.0rc0-py3-none-any.whl
   

(This is for Python 3 on a Mac.)

Install it using PIP and the variable:

sudo pip3 install --upgrade $TF_BINARY_URL
   

Vagrant

This explanation forks another.

Setup a provider VM solution to store the image (like VMware, AWS, Hyper-V). https://www.virtualbox.org/wiki/Downloads click “x86/amd64” next to VirtualBox 4.3.20 for OS X hosts to download VirtualBox-4.3.20.96.dmg (109 MB) Read the 368 page User Manual.

Install https://github.com/dotless-de/vagrant-vbguest to keep VirtualBox guest additions up to date. http://schof.org/2014/03/31/working-around-virtualbox-bug-12879/

In the Finder, in the Applications folder, drag and drop it onto your Dock for quicker use later. Double-click on VirtualBox for the VirtualBox Manager.

Download the 224.3 MB vagrant_1.7.1.dmg

The binary gets installed in the Applications folder with a link to the /usr/bin so it is added to the shell path.

List commands:

vagrant
vagrant list-commands

Change directory to where you want to store the Vagrant project and run

vagrant init

The response:

A Vagrantfile has been placed in this directory. You are now ready to vagrant up your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on vagrantup.com for more information on using Vagrant.

ls

This shows Vagrantfile (with capital V).

https://www.vagrantup.com/downloads.html

vagrant up --provider=PROVIDER

https://vagrantcloud.com/boxes/search lists boxes created by the community.

I am pulling the box from ATT M2X https://m2x.att.com/developer/sample-code This Repo provides a Vagrant virtual machine that contains several demo applications (Ruby and Python) that report data to AT&T M2X. https://github.com/attm2x/m2x-demo-vagrant

git clone https://github.com/attm2x/m2x-demo-vagrant.git

vagrant box add chef/centos-6.5

bootstrap.bash

User data for Vagrant is filed in the directory from which vagrant was used and is stored in an invisible directory .vagrant.d

Python 2 executables

This advice from 2010.

### Path to executables #

  1. To see what MacOS

    which python

    If you have MiniConda installed:

    /Users/mac/miniconda2/bin/python
    

    If you have Anaconda installed:

    /Users/mac/anaconda/bin/python
    

    If you are inside a conda activated environment:

    /Users/mac/anaconda/envs/py2/bin/python
    
  2. For a list of what Python executes:

    ls ~/miniconda2/bin

    The response begins with this:

    2to3        openssl        python2-config
    activate    pip         python2.7
    aws         pip2        python2.7-config
    

    Type python

  3. Find where you are picking up Python from?

    type python

    If Python was installed:

    python is hashed (/usr/bin/python)

    Alternately, if Conda was installed:

    python is hashed (/Users/mac/miniconda2/bin/python)

  4. Open a Terminal shell window and issue command:

    python --version

    The response is its version. My Mac Yosemite default of Python shows this:

    Python 2.7.6 (default, Sep  9 2014, 15:04:36)
    [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
    

    With Miniconda installed on El Capitan, it’s this instead:

    Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:43:17) 
    [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>>
    
  5. Get the folder:

    python -m site --user-site

The response I got is this:

/Users/mac/Library/Python/2.7/lib/python/site-packages

Alternately:

/Users/mac/.local/lib/python2.7/site-packages

Miniconda install

Below is a more “hands-on” description than what pydata.org and Kyle Purdon offers.

  1. Use an internet browser to visit the Miniconda Download page at http://conda.pydata.org/miniconda.html

    Alternately, download:

    cd ~/Downloads
    wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh

  2. Select version to download. For Python 2.7:

    Version File Size
    Python 2.7 Miniconda2-latest-MacOSX-x86_64.sh 20.3 MB
    Python 3.5 Miniconda3-latest-MacOSX-x86_64.sh 23.4 MB

    NOTE: Python3 is not backward compatible with Version 2.

    Notice the “.sh” means these are shell scripts.

  3. Open a Terminal shell window to navigate to your Downloads folder and run the Python 2.7 script:

    cd ~/Downloads
    bash Miniconda2-latest-MacOSX-x86_64.sh -b

    PROTIP: The “-b” option specifies unattended with defaults.

    The response:

    Welcome to Miniconda2 4.0.5 (by Continuum Analytics, Inc.)
     
    In order to continue the installation process, please review the license
    agreement.
    Please, press ENTER to continue
    >>> 
    
  4. Press Enter to accept the license.
  5. Press “q” to the “:” prompt.
  6. Type yes.

    “ERROR: File or directory already exists:” appears if miniconda was already installed.

  7. Confirm conda version.

Anaconda Install

This video by Corey Schafer explains it well.

https://docs.continuum.io/anaconda/install

  1. Go to web page:

    https://www.continuum.io/downloads

    QUESTION: Is there a brew anaconda?

  2. Click on the operating system icon (Mac, Windows, Linux) or scroll down and press the tab.
  3. Click to download the “command-line installer”.

    Version File Installer Installed
    Python 3.6 Anaconda3-4.3.0-MacOSX-x86_64 363 MB -
    Python 2.7 Anaconda2-4.3.0-MacOSX-x86_64 358 MB 1.41 GB
  4. In a (bash) Terminal:

    cd Downloads
    chmod 555 ./Anaconda3-4.3.0-MacOSX-x86_64.sh 
    ./Anaconda3-4.3.0-MacOSX-x86_64.sh 
    

    The response:

    Welcome to Anaconda3 4.3.0 (by Continuum Analytics, Inc.)
     
    In order to continue the installation process, please review the license
    agreement.
    Please, press ENTER to continue
    
  5. Press Enter (as if you cared).
  6. Press Tab until you’re exhausted.
  7. Type yes and press Enter.

    Anaconda3 will now be installed into this location: /Users/mac/anaconda3

  • Press ENTER to confirm the location
  • Press CTRL-C to abort the installation
  • Or specify a different location below
  1. Press Enter.

    PREFIX=/Users/mac/anaconda3

  2. Wait for it to come back to you.

    installing: _license-1.1-py36_1 ...
    installing: alabaster-0.7.9-py36_0 ...
    installing: anaconda-client-1.6.0-py36_0 ...
    installing: anaconda-navigator-1.4.3-py36_0 ...
    installing: appnope-0.1.0-py36_0 ...
    installing: appscript-1.0.1-py36_0 ...
    installing: astroid-1.4.9-py36_0 ...
    installing: astropy-1.3-np111py36_0 ...
    installing: babel-2.3.4-py36_0 ...
    ...
    installation finished.
    Do you wish the installer to prepend the Anaconda3 install location
    to PATH in your /Users/mac/.bash_profile ? [yes|no]
    [yes] >>> yes
    
  3. Type yes.

You may wish to edit your .bashrc or prepend the Anaconda3 install location:

$ export PATH=/Users/mac/anaconda3/bin:$PATH

Thank you for installing Anaconda3!

Share your notebooks and packages on Anaconda Cloud! Sign up for free: https://anaconda.org


Conda verson

See https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/

  1. For the version of conda installed, specify the upper-case V:

    conda -V

    Alternately:

    conda --version

    The response is like:

    conda 4.3.9
    

Update conda

  1. To update miniconda’s version, use the conda command line installed above:

    conda update conda

    The response is a list of packages to be updated if you agree:

    Fetching package metadata: ....
    .Solving package specifications: ..........
     
    Package plan for installation in environment /Users/mac/miniconda2:
     
    The following packages will be downloaded:
     
     package                    |            build
     ---------------------------|-----------------
     sqlite-3.13.0              |                0         1.4 MB
     python-2.7.12              |                1         9.5 MB
     conda-env-2.5.1            |           py27_0          26 KB
     ruamel_yaml-0.11.7         |           py27_0         174 KB
     conda-4.1.8                |           py27_0         201 KB
     ------------------------------------------------------------
                                            Total:        11.2 MB
     
    The following NEW packages will be INSTALLED:
     
     ruamel_yaml: 0.11.7-py27_0
     
    The following packages will be UPDATED:
     
     conda:       4.0.4-py27_0 --> 4.1.8-py27_0 
     conda-env:   2.4.5-py27_0 --> 2.5.1-py27_0 
     python:      2.7.11-0     --> 2.7.12-1     
     sqlite:      3.9.2-0      --> 3.13.0-0     
     
    Proceed ([y]/n)? 
    
  2. Press “y” to proceed. A sample response:

    Fetching packages ...
    sqlite-3.13.0- 100% |################################| Time: 0:00:02 483.10 kB/s
    python-2.7.12- 100% |################################| Time: 0:00:20 477.84 kB/s
    conda-env-2.5. 100% |################################| Time: 0:00:00 392.63 kB/s
    ruamel_yaml-0. 100% |################################| Time: 0:00:00 331.95 kB/s
    conda-4.1.8-py 100% |################################| Time: 0:00:00 457.60 kB/s
    Extracting packages ...
    [      COMPLETE      ]|###################################################| 100%
    Unlinking packages ...
    [      COMPLETE      ]|###################################################| 100%
    Linking packages ...
    [      COMPLETE      ]|###################################################| 100%
    
  3. After install, close and then re-open the terminal window so the changes can take effect.

  4. For a list of packages installed locally (in the currently active environment):

    conda list

    The “py36_1” in the list are pip installed.

    _license                  1.1                      py36_1  
    alabaster                 0.7.9                    py36_0  
    anaconda                  4.3.0               np111py36_0  
    

.bash_profile config

TODO: Instructions for both Miniconda and conda.

The path to Python should be the first in PATH:

  1. Open a text editor to ~/.bash_profile and add:

    export PATH=”~/miniconda2/bin:$PATH”
    export PYTHON_PATH=~/miniconda2/bin/python

    When you’re done:

  2. Activate the shell file :

    source ~/.bash_profile

    Troubleshoot

  3. To reset:

    brew link –overwrite python

    Warning: Already linked: /usr/local/Cellar/python/2.7.12_2
    To relink: brew unlink python && brew link python

    Conda info

  4. Get a list:

    conda info

    The response:

    Current conda install:
     
              platform : osx-64
           conda version : 4.3.9
        conda is private : False
       conda-env version : 4.3.9
     conda-build version : 2.1.2
          python version : 3.5.2.final.0
        requests version : 2.12.4
        root environment : /Users/mac/anaconda  (writable)
     default environment : /Users/mac/anaconda
        envs directories : /Users/mac/anaconda/envs
           package cache : /Users/mac/anaconda/pkgs
            channel URLs : https://repo.continuum.io/pkgs/free/osx-64
                           https://repo.continuum.io/pkgs/free/noarch
                           https://repo.continuum.io/pkgs/r/osx-64
                           https://repo.continuum.io/pkgs/r/noarch
                           https://repo.continuum.io/pkgs/pro/osx-64
                           https://repo.continuum.io/pkgs/pro/noarch
             config file : None
            offline mode : False
              user-agent : conda/4.3.9 requests/2.12.4 CPython/3.5.2 Darwin/16.4.0 OSX/10.12.3
                 UID:GID : 501:20
        

    Conda environments

  5. Get a list of Conda environments (from any folder) using the -e flag:

    conda info -e

    Alternately, if you like typing long options:

    conda info –env

    The response are like this:

    tensorflow_env           /Users/mac/anaconda/envs/tensorflow_env
    root                  *  /Users/mac/anaconda
    
  6. Create a Conda environment named py2 using Python 2.7:

    
    conda create -n py2 python=2.7 anaconda
    
  7. Press y to go ahead.

  8. Add python packages, such as TensorFlow:

    
    conda install -n yourenvname tensorflowp
    conda install -c conda-forge tensorflow
    
  9. Activate to use the environment:

    source activate
    
  10. When done using TensorFlow, deactivate the environment:

    source deactivate
    

Conda pyenv

pyenv enables switch between multiple versions of Python on a single system (laptop).

  1. Create two “named” Conda environments (one with Python2 and the other with Python3):

    
    conda create -n py3 python=3*
    conda create -n py2 python=2*
    
    
    
  2. Set one of these as my default by adding to my terminal startup file ~/.bash_profile:

    
    source activate py3 
    

    Typically I only use these “named python” environments to run a Python REPL or do general Python tasks. I’ll create another conda environment named specifically for each real project I work on.

Uninstall

PROTIP: Delete Conda one folder at a time (without the –yes parameter).

anaconda-clean

Files such as:

   .conda
   Backup directory: /Users/mac/.anaconda_backup/2017-03-07T051620
   .continuum
   .idlerc
   .ipython
   .jupyter
   .matplotlib
   

http://www.numericalexpert.com/blog/conda_tips/


### Install Python packages #

From inside a conda environment:

  1. NumPy at http://www.numpy.org/ needed by http://www.pymvpa.org/installation.html

    pip install numpy

  2. Instead of downloading http://www.scipy.org/scipylib/download.html# linear algebra, standard distributions, signal processing, data IO

    pip install scipy

  3. SKlearn

    pip install sklearn

  4. Pandas based on VIDEO: How to install Pandas Miniconda:

    pip install pandas

    YouTube videos: Learn Pandas

  5. matplotlib

Other Python packages:

  • xlwings interfaces with Microsoft Excel spreadsheets
  • pygame develops GUI

OpenCV for computer vision

Follow the instructions below to install python2 + OpenCV in mac

http://www.pyimagesearch.com/2016/11/28/macos-install-opencv-3-and-python-2-7/

In case of python3 + OpenCV follow

http://www.pyimagesearch.com/2016/12/05/macos-install-opencv-3-and-python-3-5/


Python 3 vs. 2

Sure, they say “all new Python code should be written for version 3. There are so many new features in Python 3 that it doesn’t make much sense to stick with Python 2 unless you’re working with old code.”

Most new features introduced with Python 3 versions not backwards compatible with version 2.

### Using Python 3

  1. After installing Python3, obtain the Python 3 command line with:

    python3

    The response I got:

    Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22)
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    

Here are the steps to remove Python3 from your Mac:

   ls -l ~/Library/Frameworks/Python.framework/Versions/3.4/bin/python3
   

sudo rm -rf that.

Floating point

In Python 2, type:

1/2

The response is:

0

In Python3, type:

1/2

The response is:

0.5

Print

For the most part, Python 2 code works with Python 3.

Where Python 2 code fails most often is the print statement. Printing in Python 2 is done like so:

print "Hello", "world!"

The response:

Hello world!

If you input the above in Python 3, the response is:

SyntaxError: Missing parentheses in call to ‘print’

This is because Python 3 uses a function:

print("Hello", "world!")

So in Python 2.6+, use the future module to back-port:

from __future__ import print_function
   print("Hello", "world!")
   

Try this:


   import sys
   print('Python: {}'.format(sys.version))
   

The response:

Python: 3.5.2 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:52:12) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]
   

BLAH: This doesn’t work for me:


import numpy
print('Numpy: {}'.format(numpy.version))
   

The response:

Numpy: <module 'numpy.version' from '/Users/mac/anaconda/lib/python3.5/site-packages/numpy/version.py'>
   

PyCharm

https://discussions.udacity.com/t/referencing-pygame-from-pycharm-with-anaconda/223711/13

Turi (Dato) Python algorithms

GraphLab Create from Dato provides scalable “pre-implemented” ML algorithms using Python installed using Anaconda. Entire courses on its use is at

  • https://www.coursera.org/learn/ml-foundations
  • https://www.turi.com/learn/userguide/
  • https://www.turi.com/products/create/docs/
  • https://github.com/learnml/machine-learning-specialization
  • https://www.coursera.org/learn/ml-clustering-and-retrieval/supplement/iF7Ji/software-tools-you-ll-need-for-this-course

When the one-year free license is over, note scikit-learn also uses Python with Anaconda.

Python libraries

For matrix operations, use the Numpy open-source Python library for fast performance with data that fits in memory. Quickstart.

In a requirements.txt file:

tweepy (http://www.tweepy.org)

csv (https://pypi.python.org/pypi/csv)

textblob (https://textblob.readthedocs.io/en/dev/)

keras (https://keras.io)

bokeh Flask ipython jupyter matplotlib nose numpy pandas Pillow pymc requests scikit-image scikit-learn scipy seaborn statsmodel tensorflow virtualenv virtualenvwrapper

OpenCV3

brew tap homebrew/science && brew update
brew install opencv3 --with-contrib --with-python3 --without-python
ln -s /usr/local/opt/opencv3/lib/python3.6/site-packages/cv2.cpython-36m-darwin.so /usr/local/lib/python3.6/site-packages/cv2.so
export PYTHONPATH=$PYTHONPATH:/usr/local/Cellar/opencv3/3.2.0/lib/python3.6/site-packages
   

Data Manipulation

SFrame is an open-source, highly-scalable Python library for data manipulation. Unlike Pandas, SFrame is not limited to datasets which can fit in memory, so it can deal with large datasets, even on a laptop.

http://stackoverflow.com/questions/509211/explain-pythons-slice-notation/


Miscellaneous

http://stackoverflow.com/questions/990754/how-to-leave-exit-deactivate-a-python-virtualenv?rq=1

https://gist.github.com/alyssaq/f60393545173379e0f3f describes install of https://bootstrap.pypa.io/get-pip.py

More on OSX

This is one of a series on Mac OSX: