Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

PROTIPs and tricks from various learning resources

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Here are various coding tips I’ve seen while going through Python programming classes after installing Python and Jupyter.

Reserved Keywords

Listed alphabetically below are words that Python’s reserved for itself, so you can’t use them as custom variables.

PROTIP: Research and find out what each is about:

  • and
  • as
  • assert
  • async
  • await
  • break
  • class
  • continue
  • def
  • del
  • elif
  • else
  • except
  • False
  • finally
  • for
  • from
  • global
  • if
  • import
  • in
  • is
  • lambda
  • None
  • nonlocal
  • not
  • or
  • pass
  • raise
  • return
  • True
  • try
  • while
  • with
  • yield

The list above can be retrieved (as an array) by this code after typing python for the REPL (Read Evaluate Print Loop) interactive prompt:

python
>>> import keyword
>>> keyword.kwlist
>>> exit()

Press control+D to exit anytime.

Use Not None Reserved Word

Returning 0 on error can be confused with the number 0 as a valid response.

To avoid the confusion, return the Python reserved word “None”:

result = safe_square_root(4)
if result is not None:   # happy path:
   value = result.pop()  # pop up from stack.
   print(value)
else:  # notice we're not checking for None.
    # calling function does not need to handle error:
    # an error occurred, but encapsulated to be forwarded and processed upstream:
    print("unable to compute square root")

Function:

def safe_square_root(x)
    try:
        return [math.sqrt(x)]   # in a stack.
    except ValueError:
        return None   # using reserved word.

Environment Variables

To read a file named “.env” at the $HOME folder, and obtain the value from “MY_EMAIL”:

import os
env_vars = !cat ~/.env
for var in env_vars:
    key, value = var.split('=')
    os.environ[key] = value
 
print(os.environ.get('MY_EMAIL'))   # containing "johndoe@gmail.com"

This code is important because it keeps secrets in your $HOME folder, away from folders that get pushed up to GitHub.

There is the “load_dotenv” package that can do the above, but using native commands mean less exposure to potential attacks.

String operations

To return just the first 3 characters of a string:

letters = "abcdef"
first_part = letters[:3]

The above code provides flexibility for alternatives such as Cyrillic (Russian) character set.

Alternatively, this function defines a dictionary to covert an Excel column number to a number:*

def letter_to_number(letters):
    letters = letters.lower()
    dictionary = {'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7,'h':8,'i':9,'j':10,'k':11,'l':12,'m':13,'n':14,'o':15,'p':16,'q':17,'r':18,'s':19,'t':20,'u':21,'v':22,'w':23,'x':24,'y':25,'z':26}
    strlen = len(letters)
    if strlen == 1:
        number = dictionary[letters]
    elif strlen == 2:
        first_letter = letters[0]
        first_number = dictionary[first_letter]
        second_letter = letters[1]
        second_number = dictionary[second_letter]
        number = (first_number * 26) + second_number
    elif strlen == 3:
        first_letter = letters[0]
        first_number = dictionary[first_letter]
        second_letter = letters[1]
        second_number = dictionary[second_letter]
        third_letter = letters[2]
        third_number = dictionary[third_letter]
        number = (first_number * 26 * 26) + (second_number * 26) + third_number
    return number

But instead of defining a dictionary, you can use a property of the ASCII character set, in that the Latin alphabet begins from its 65th position for “A” and its 97th character for “a”, obtained using the ordinal function:

ord('a')  # returns 97
ord('A')  # returns 65

This returns ‘a’ :

chr(97)

There is a Python library to work with Excel spreadsheets: https://xlsxwriter.readthedocs.io/working_with_cell_notation.html#cell-utility which translates between Excel cell addresses (such as “A1”) and zero-based Python array tuple:

str = xl_rowcol_to_cell(0, 0, row_abs=True, col_abs=True)  # $A$1
(row, col) = xl_cell_to_rowcol('A1')    # (0, 0)
column = xl_col_to_name(1, True)   # $B

File open() modes

PROTIP: Be explicit about using text (vs. binary) mode.

CharacterMeaning
bbinary (text mode is default)
ttext mode (default)
rread-only (the default)
+open for updating (read and write)
wwrite-only after truncating the file
aappend
xopen for exclusive creation, failing if file already exists
Uuniversal newlines mode (used to upgrade older code)

write() returns the count of codepoints (characters in the string), not the number of bytes. So don’t use it’s count.

read() returns line endings (\n) in string lines.

readlines() shows the whole file.

File Copy commands

The shutil package provides fine-grained control for copying files:

    import shutil

This table summarizes the differences among shutil commands:

 Dest. dir.Copies metadataPreserve permissionsAccepts file object
shutil.copyfile----
shutil.copyfileobj---Yes
shutil.copyYes-Yes-
shutil.copy2YesYesYes-

See https://docs.python.org/3/library/filesys.html

File Metadata

Metadata includes Last modified and Last accessed info (mtime and atime). Such information is maintained at the folder level.

For all commands, if the destination location is not writable, an IOError exception is raised.

  • To copy a file within the same folder as the source file:

    shutil.copyfile(src, dst)

    buffer cannot be when copying to another folder.

  • To copy a file within the same folder and buffer file-like objects (with a read or write method, such as StringIO):

    shutil.copyfileobj(src, dst)

Notice both individual file copy commands do not copy over permissions from the source file. Both folder-level copy commands below carry over permissions.

CAUTION: folder-level copy commands do not buffer.

  • PROTIP: To copy a file to another folder and retain metadata:

    file_src = 'source.txt'  
    f_src = open(file_src, 'rb')
    file_dest = 'destination.txt'  
    f_dest = open(file_dest, 'wb')
    shutil.copyfileobj(f_src, f_dest)  
     

    The destination needs to specify a full path.

  • To copy a file to another folder and NOT retain metadata:

    shutil.copy2(src, "/usr", *, follow_symlinks=True)
  • You can use the operating system shell copy command, but there is the overhead of opening a pipe, system shell, or subprocess, plus poses a potential security risk.

    # In Unix/Linux
    os.system('cp source.txt destination.txt')  \# https://docs.python.org/3/library/os.html#os.system
    status = subprocess.call('cp source.txt destination.txt', shell=True) 
     
    # In Windows
    os.system('copy source.txt destination.txt')
    status = subprocess.call('copy source.txt destination.txt', shell=True)  \# https://docs.python.org/3/library/subprocess.html
    
  • Pipe open has been deprecated. https://docs.python.org/3/library/os.html#os.popen

    # In Unix/Linux
    os.popen('cp source.txt destination.txt')
     
    # In Windows
    os.popen('copy source.txt destination.txt')
    

Error Exception handling

Handle file not found exception : :

# if file doesn't exist in folder, create it:
import os
import sys
 
def make_at(path p, dir_name)
    original_path = os.getcwd()
    try:
        os.chdir(path)
        os.makedir(dir_name)
    except OSError as e:
        print(e, file=sys.stderr)
        raise
    finally:  #clean-up no matter what:
        os.chdir(original_path)

Operating system

There are platform-specific modules:

  • Windows msvcrt (Visual C run-time)
  • MacOS sys, tty, termios, etc.

To determine what operating system to wait for a keypress, use sys.platform, which has finer granularity than sys.name because it uses uname:

# https://docs.python.org/library/sys.html#sys.platform
from sys import platform
if platform == "linux" or platform == "linux2":
    # linux
elif platform == "darwin":
    # OS X
elif platform == "win32":
    # Windows
elif platform == "cygwin":
    # Windows running cygwin Linux emulator
   

http://code.google.com/p/psutil/ to do more in-depth research.

Command generator

docopt from https://github.com/docopt/docopt is described at http://docopt.org creates custom CLI commands by parsing a command help text into cli code that implements it.

Brilliant.

Click

Dan Bader recommends the use of click.pocoo.org/6/why click custom package (from Armin Ronacher) instead of the argparse package that comes with Python 3.2+ (and the optparse package that comes with Python 2).

Click provides decorators such as the “@click.command()” below:

\# cli.py
import click
 
@click.command()
def main():
    print("I'm a beautiful CLI ✨")
 
if __name__ == "__main__":
    main()
   

List comprehension

squares = [x * x for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The Playbook of code shown on What Does It Take To Be An Expert At Python? [1:52:02] presented by by James Powell at the PyData conference on Aug 2, 2017.

Abhishake Gupta’s pyTest at https://github.com/letspython3x/code_examples

https://www.codementor.io/alibabacloud/ how-to-create-and-deploy-a-pre-trained-word2vec-deep-learning-rest-api-oekpbfqpj

https://www.learnpython.org/en/Classes_and_Objects

Math Operators

11%5 use the (percent sign), the modulo operator to divide 11 by the quotient 5 in order to return 1 because two 5s can go into 11, leaving 1 left over, the remainder.

11//5 uses “floor division” to return just the integer of 2, discarding the remainder. It returns the integral part of the quotient.

To avoid divide by zero errors by returning 0:

def weird_division(n, d):
    return n / d if d else 0

Testing

Being a dynamic language, errors in Python code can appear only when run rather than when compiled.

PROTIP: Create a test .py file to go with each py file.

There are several libraries to support testing.

  1. unittest

    Described at Python Tutorial: Unit Testing Your Code with the unittest Module Aug 16, 2017 by Corey Schafer

  2. pyTest

    pip3 install pytest
    import file_ab_session as fas
    def test_add_function_given_two_arguments():
     RESULT = fas.add(2,3)
     EXPECTED_RESULT = 5
     assert RESULT == EXPECTED_RESULT
    

Applicable to both:

  • Name all test classes with a name beginning with “test”.

  • Tests are not run from top to bottom, so each test needs to be stand-alone.

  • To do stuff before the tests:

@classmethod
   def setupClass(cls)
       print('in setupClass')
 
   @classmethod
   def tearDownClass(cls)
       print('in tearDownClass')
   

Tools for Debugging Python code

  • Python Tutor - an excellent way to actually visualize how the interpreter actually reads and executes your code

  • DiffChecker - compares two sets of text and shows you which lines are different

  • Debugging in Python - steps you can take to try to debug your program

  • Mocking of API end-points when the actual service is not available.

Python in the Cloud

On AWS:

On Azure:

pip install azure
  • https://docs.microsoft.com/python/azure/
  • https://azure.microsoft.com/resources/samples/?platform=python
  • https://github.com/Azure/azure-sdk-for-python/wiki/Contributing-to-the-tests
  • https://azure.microsoft.com/en-us/support/community/

Sets

Day of week Set handling

day_of_week_en = ["Sun","Mon","Tue","Wed","Thu","Fri","Sat"]
day_of_week_en.append("Luv")
days_in_week=len(day_of_week_en)
print(f"{days_in_week} days a week" )
print(day_of_week_en)
 
x=0
for index in range(8):
    print("{0}={1}".format(day_of_week_en[x],x))
    x += 1

set([3,2,3,1,5]) # auto-renumbers with duplicates removed

Tuple

person = ('john', 'doe', 40)
person
type(person)

Tuples are fixed-sized collections of related items (akin to a “struct” in Java or “record”) used to pass multiple values of various types to or from a function.

Use a list instead for a collection of similar objects.

PROTIP: When adding a single value, include a comma at the end to avoid it being classified as a string:

person = 'john',
type(person)

More about Python

This is one of a series about Python:

  1. Python install on MacOS using Pyenv
  2. Python install on MacOS
  3. Python install on Raspberry Pi for IoT

  4. Test Python using Pytest BDD Selenium framework
  5. Test Python using Robot testing framework

  6. Python certifications
  7. Python tutorials
  8. Python coding notes
  9. Jupyter Notebooks provide commentary to Python

  10. Pulumi controls cloud using Python, etc.
  11. Microsoft Azure Machine Learning makes use of Python
  12. Testing AI uses Python code

  13. Python REST API programming using the Flask library
  14. Python coding for AWS Lambda Serverless programming
  15. Streamlit visualization framework powered by Python
  16. Web scraping using Scrapy, powered by Python
  17. Neo4j graph databases accessed from Python