My collection of useful ways to use Python securely in a production setting

This is implementation of suggestions in my article on “How to shine at coding challenges”:

It’s no longer impressive to have open-source libraries under your name. Concern about malicious libraries and their unsecure transitive dependencies has forced products such as Walmart Labs’ Hapi.js to not use any external libraries.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

It has been reported that much sample code lacks security, editing, internationalization, etc. So, along with some friends, we created this repo containing coding that has various features all (safely) working together in one program file:

https://github.com/wilsonmar/python-samples

This page contains “deep-dive” commentary which references code in that repos, which contain these files:

• api-sample.py - the star of this show - the coding

• python-samples.sh - a shell script which sets up the environment with Conda and installs Python packages needed. Common Python installation and configuration issues and procedures are described in my blog article wilsonmar.github.io/python-install.

• api-sample.env which stores environment variables used by the Python program.

• .gitignore contains

• workflow files for use within GitHub Actions

Other Python project templates:

• https://github.com/MartinHeinz/python-project-blueprint by Martin Heinz in Russia

• https://dev.to/codemouse92/dead-simple-python-project-structure-and-imports-38c6

Install

Before being able to run the code on a particular machine (laptop), several utilities need to be installed on top of the Operating System.

On a macOS Terminal or

A. To view the code online, use a browser at address:

https://github.com/wilsonmar/python-samples/blob/master/api-sample.py

B. Alternately, edit the code online using Cloud9

C. Alternately, to work with the whole repo on your laptop,

1. navigate to where you want the repo added and

2. Open your macOS Terminal which has been installed a git program, and:

   git clone https://github.com/wilsonmar/python-samples/blob/master/api-sample.py

3. Navigate into the folder created:

   cd Python-samples

4. Using “code” (VSCode), or other editor (IDE) to open the whole folder by specifying a dot:

   code . 

   PyCharm,

5. Within your editor, in the left menu, click on api-sample.py to open it for edit.

Virtualenv

A virtual environment enables a specific set of Python dependencies to be installed, so no weird, difficult-to-debug dependency issues arise.

When installing venv:

created virtual environment CPython3.9.8.final.0-64 in 5940ms
  creator CPython3Posix(dest=/Users/wilson_mar/gmail_acct/python-samples/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/wilson_mar/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==58.5.3, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

“venv” is the preferred name of an environment. But variable my_venv_folder is used in case you want customization.

1. Detect whether the folder (defined by variable my_venv_folder) has been created:

   PROTIP: Python code running a Linux operating system command.

   if run("which python3").find(my_venv_folder) == -1:  # not found:
   # Such as /Users/wilsonmar/miniconda3/envs/py3k/bin/python3
   # So create the folder inside the program's folder:
   python3 -m venv ${my_venv_folder}
   

2. Activation is necessary. To activate on a Mac:

   source "{my_venv_folder}"/bin/activate

   On Windows:

   venv\Scripts\activate.bat

3. To check if a virtual environment is active, In CLI, (venv) appears. The path of the venv folder should appear:

   echo ${VIRTUAL_ENV}

   Within Python: check whether the VIRTUAL_ENV environment variable is set to the path of the virtual environment:

   • Outside a virtual environment, sys.prefix points to the system python installation and sys.real_prefix is not defined.

   • Inside a virtual environment, sys.prefix points to the virtual environment python installation and sys.real_prefix points to the system python installation.

conda info --envs

Bandit vulnerability checker

• https://soshace.com/how-to-secure-python-web-app-using-bandit/
• https://bandit.readthedocs.io/en/latest/plugins/index.html
  
  

Bandit transforms code into an abstract syntax tree (AST) to analyze vulnerabilities https://snyk.io/learn/security-vulnerability-exploits-threats/

Running in debug mode

Running with debug=True exposes the Werkzeug debugger which can execute arbitrary code.

Bandit flags this with “201:flask_debug_true”.

On production servers, set debug == False which disables assert statements.

Access Google Sheets

• https://medium.com/analytics-vidhya/how-to-read-and-write-data-to-google-spreadsheet-using-python-ebf54d51a72c
  
  

One way to reduce risk of data loss is to store data in cloud environments managed by pros.

Here is how to do it within Google Cloud (whose business is to sell metadata about their users):

1. If you haven’t already, get a Google account (with a email address Gmail.com).

2. Get in the Google API Console at https://console.developers.google.com or https://console.cloud.google.com

   You will be brought to your default project, such as:
   https://console.cloud.google.com/apis/dashboard?pli=1&project=weather-454da

3. Click your project name at the upper-right for the “Select a project”.

4. Click “NEW PROJECT”. CREATE. The name you provide is combined with the project code generated by Google to create the globally unique service name, which is like an email address. Such as:

   XXX-compute@developer.gserviceaccount.com

5. Click “ENABLE APIS AND SERVICES”.
6. Search for Google Drive API. Click on it among results. Click “ENABLE”.

7. In the sidebar, click Credentials.

   

8. Click CREATE CREDENTIALS at the top. Select “Service account” (Enables server-to-server, app-level authentication using robot accounts).

   

9. VIDEO: Click the “hamberger” icon at the upper-left for “IAM & Admin” -> “Service Accounts” -> “CREATE SERVICE ACCOUNT”.

10. For “Service account name”, come up with a name.

    The “Service account ID” and email are generated based on your input above.

11. Type a Service account description.
12. Click “CREATE AND CONTINUE”.
13. Select a role. Full access is granted to Basic -> Owner.
14. CONTINUE. DONE.
15. Click the three-dot icon to the right of the service account just created to
    select Manage Keys.
16. ADD KEY to select “Create new key”. Select “JSON” (instead of P12). CREATE.

17. Save it to your personal folder, not within a GitHub repo. CLOSE the confirmation pop-up.

    PROTIP: If you have several Google projects, keep a list of the names Google assigned.

    CAUTION: Notice that the Key expiration date is year “9999”.

    PROTIP: Instead of using a service account key, use workload identity federation to grant Google Cloud access to external multi-cloud workloads (AWS, Azure OIDC, SAML 2.0, etc.). It uses temporary tokens from workload identity pools.

18. ??? Principal??? Select Compute Engine service default, JSON, hit create.

19. Open up the JSON file, share your spreadsheet with the API ID.

20. Save the credentials file (formatted in JSON) for the Python program below to read. For example:

    /Users/johndoe/github-johndoe/repo1/creds.json

Google provides the API for:

• Bulk document creation
• Content Management
• Workflow Management
  
  

Python code samples make use of the Pandas library.

TODO: In a Google Doc, find a marker such as “QUESTION:” and extract the sentence next to it, then paste that sentence in a Google Sheet. Perhaps using the SpaCy library, https://spacy.io/ - a Python library for Natural Language Processing (NLP) to recognize the meaning of tokens within each sentence, as described in https://towardsdatascience.com/how-to-extract-structured-information-from-a-text-through-python-spacy-749b311161e using https://github.com/alod83/data-science/tree/master/TextAnalysis

Access Google Docs

* https://developers.google.com/docs

• https://developers.google.com/docs/api/quickstart/python
• https://towardsdatascience.com/how-to-load-the-content-of-a-google-document-in-python-1d23fd8d8e52 </em>
  
  

Access Google Sheets

• https://developers.google.com/sheets/api/quickstart/python
• https://erikrood.com/Posts/py_gsheets.html
• https://www.youtube.com/watch?v=cnPlKLEGR7E
• https://blog.coupler.io/python-to-google-sheets/#Python_script_to_export_Excel_to_Google_Sheets
• https://www.youtube.com/watch?v=4ssigWmExak </em>
  
  

1. Examine the library: https://github.com/nithinmurali/pygsheets

2. On a Mac Terminal with Python installed,

   pip install pygsheets
   

3. Create a Google Sheet.

4. The code

   import pygsheets
   import pandas as pd
   GOOGLE_CREDS_FILE="/Users/erikrood/desktop/QS_Model/creds.json"
   #authorization
   gc = pygsheets.authorize(service_file=GOOGLE_CREDS_FILE)
    
   # Create empty dataframe
   df = pd.DataFrame()
    
   # Create a column
   df['name'] = ['John', 'Steve', 'Sarah']
    
   #open the google spreadsheet (where 'PY to Gsheet Test' is the name of my sheet)
   sh = gc.open('PY to Gsheet Test')
    
   #select the first sheet in the file:
   wks = sh[0]
    
   #update the first sheet with df, starting at cell B2. 
   wks.set_dataframe(df,(1,1))
   

Image to Text

https://developer.ibm.com/tutorials/document-scanner/

Program Execution Feature Flags

In a terminal:

python api-sample.py
   

No parameters need to be specified because the program has hard-coded defaults for each feature flag, with ALL features enabled.

Inside the program are feature flags which the program references to determine whether each feature is executed during a particular run.

Sections of code (and their feature flags)

1. Define program attributes.
2. Import libraries
3. Capture starting time and set default global values
4. Parse arguments that control program operation
5. Define utilities for printing (in color), logging
6. Local machine in-memory SQL database = SQLLite
7. Obtain run control data from .env file in the user’s $HOME folder
8. Define Localization (to translate text to the specified locale)
9. Define utilities for managing data storage folders and files
   1. Create, navigate to, and remove local working folders
10. Display run conditions: datetime, OS, Python version, etc.
    1. Retrieve client IP address = get_ipaddr
    2. Lookup geolocation info from IP Address = lookup_ipaddr
    3. Obtain Zip Code to retrieve Weather info = lookup_zipinfo
    4. Retrieve Weather info from zip code or lat/long = show_weather
11. Generate various calculations for hashing, encryption, etc.
    1. Generate Hash from a file & text = gen_hash
    2. Generate a random salt = gen_salt
    3. Generate a random percent of 100 = gen_1_in_100
    4. Convert between Roman numerals & decimal = process_romans (case structure)
    5. Generate JWT (Json Web Token) = gen_jwt
    6. Generate Lotto America Numbers = gen_lotto
    7. Generate Magic 8-ball numbers = gen_magic_8ball

12. Get in the cloud: 1. Retrieve secrets from Azure Key Vault = use_azure 2. Retrieve secrets from AWS KMS = use_aws 3. Retrieve secrets from GCP = use_gcp 4. Retrieve secrets from HashiCorp Vault = use_vault

13. Applications processing user input with persistance: 1. Calculte BMI using units of measure based on country = categorize_bmi 2. Generate Fibonacci with memoization = gen_fibonacci 3. Make change using Dynamic Programming = make_change 4. Fill knapsack = fill_knapsack

14. Make use of cloud services: 1. Create/Reuse container folder for img app to use
15. Download img application files = download_imgs
16. Manipulate image (OpenCV OCR extract) = process_img
17. Send message to Slack = send_slack_msgs (TODO:)

18. Send email = send_email (TODO:)

19. Remove (clean-up) folder/files created = cleanup_img_files
20. Display run time stats at end of program = display_run_stats

Input Data specification

The program precedence of override:

1. Prompts of the user from inside the running program (such as for Zip Code) overrides
2. parameter specifications at run-time, which overrides
3. key text retrieved from OS Keyring, which overrides
4. what is retrieved from Azure, AWS, GCP, HashiCorp Vault, which overrides
5. what is specified in persistent environment (.env) file, which overrides
6. what is (can safely be) hard-coded in program code, which overides
7. what is obtained from the operating system.
   
   

Output specification

What the program outputs to the Terminal can be precisely specified.

The default sample output:

*** env_path LOCALE 'en_EN' overrides OS LOCALE ('en_US', 'UTF-8')
 
*** api-sample.py v0.0.33 Created: Saturday 27 Nov 2021 01:23:18 PM   
*** at /Users/wilsonmar/gmail_acct/python-samples/api-sample.py 
*** on /Users/wilsonmar/miniconda3/envs/py3k/lib/python3.8/site-packages 
*** Started Saturday 27 Nov 2021 08:33:41 PM   (epoch=1638070421.006971) 
*** macOS version=10.16 ['Big Sur', 2020] process ID=10298
*** Disk space free: 42.0 / 122.1 GB 
*** Python version="3.8.12 | packaged by conda-forge | (default, Sep 29 2021, 19:44:33) 
[Clang 11.1.0 ]
  
*** env_path=/Users/wilsonmar/python-samples.env
 
*** Lotto America: 5 lucky numbers between 1 and 52 and 1 Star number between 1 and 10:
*** 20 6 24 4 38 6 
  
*** uuid.uuid4()=3d9a8c08-c354-4712-8e7d-d8dae320a1be 
*** x.time=509684474424495112 
*** Path: "/Users/wilsonmar/Projects" 
*** Directory "Images" created Thursday 25 Nov 2021 09:23:20 PM MST -0700
 
*** Longitude: -97.822 Latitude: 37.751 in US America/Chicago USD (VPN).
*** Using hard-coded default zip code "59041".
*** Longitude: -108.9922 Latitude: 45.4941 in US Joliet 59041 
*** Minimum temperature: 44.42°F (6.90°C), Sunrise: 2021-11-23 07:26:23 AM 
*** Currently: 49.57°F (9.76°C), 26% humidity, overcast clouds, visibility: 10000 feet
*** Maximum temperature: 56.61°F (13.67°C),  Sunset: 2021-11-23 04:38:57 PM 
*** Wind Speed: 1.97 (Gusts: 4.14) mph from direction: WNW (259/360) 
*** Atmospheric pressure: 1000 hPa (hectopascals) 
  
*** Script executing at path: '/Users/wilsonmar/Projects' 
*** Downloading to directory: '/Users/wilsonmar/Projects/Images' 
*** Directory "Images" created Thursday 25 Nov 2021 09:23:21 PM MST -0700
Images/
    google.ico
*** Downloading to file path: '/Users/wilsonmar/Projects/Images/google.ico' 
*** No downloading as file can be accessed.
*** Download of 5,430-byte google.ico 
*** After this run: /Users/wilsonmar/Projects/Images 
Images/
    google.ico
 
*** Ended Saturday 27 Nov 2022 08:34:26 PM   (epoch=1638070466.103145) 
*** api-sample.py done in 0.77 seconds. 

Top of coding file in api-sample.py

1. QUESTION: Why is the top line needed?

   #!/usr/bin/env python

Imports

1. Internal or external Python modules need to be specified for their classes, functions, and methods to be referenced.

   import unittest
   from safe_module import package, function, class
   

   Utilities:

   • https://pyup.io tracks thousands of Python packages for vulnerabilities and makes pull requests to your codebase automatically when there are updates.
   • Bandit is static linter which alerts developers to common security issues in code.
   • Chef InSpec audits installed packages and their versions.
   • Pysa is static analyzer, open sourced by Facebook (Meta).
   • sqreen.com (until its purchase by Datadog) checks each application for packages with malicious code and checks for legitimate packages with known problems or outdated versions.
     
     

   Known security issues have been found in Pickle & cPickle for its serialization from representation on disk or over the network interface, because constructors and methods contain executable code. For example, website cookies have a reduce method that can be modified to contain malicious code. A better approach would be to use json.loads() or yaml_safe_load().

   To verify payload integrity using cryptographic signatures, use pycryptodome. Don’t use PyCrypto because the project hasn’t been since Jun 20, 2014 and no security update has been released to fix vulnerability CVE-2013-7459.
   
   

   There have been a number of Trojan horse cases from malicious code in Python packages, specifically PyPi. Moreover, some were not detected for a year.

   pythonsecurity.org, an OWASP project aimed at creating a security-hardened version of Python, explains security concerns in modules and functions (from 2014). Other lists: * https://packetstormsecurity.com/files/tags/python/ * https://codehandbook.org/secure-coding-in-python/ * https://www.whitesourcesoftware.com/vulnerability-database/
   
   

   OWSAP had (until 2015) a curated list of Python libraries for security applications.

   There are two types of import paths in Python: absolute and relative.

   Absolute imports specifies the path of the resource to be imported using its full path from the project’s root folder.

   from package1 import module1
   from package1.module2 import function1
   

   Relative import specifies the resource to be imported relative to the current location of the project the folder of the import statement. There are two types of relative imports: implicit and explicit.

   from .some_module import some_class
   from ..some_package import some_function
   

   Implicit relative imports have been removed in Python 3 because if the module specified is found in the system path, it will be imported with vulnerabilities. If the malicious module is found before the real module it will be imported and could be used to exploit applications that has it in their dependency tree. If a malicious module specified is found in the system path, it will be imported into your program.

LinkedIn.com video course by Ronnie Sheer

Date and Time Handling

There are several modules which handle date, time, timezones, etc.:

• date – Manipulate just date ( Month, day, year)
• time – Time independent of the day (Hour, minute, second, microsecond)
• datetime – Combination of time and date (Month, day, year, hour, second, microsecond)
• timedelta — A duration of time used for manipulating dates
• tzinfo — An abstract class for dealing with at https://www.iana.org/time-zones”>time zones

• see https://www.wikiwand.com/en/List_of_tz_database_time_zones 2012 Best Current Practice for Maintaining the Time Zone (TZ) Database is by a group of volunteers. Geographical boundaries in the form of coordinate sets are not part of the tz database, but boundaries are published in the form of vector polygon shapefiles. Using these vector polygons, one can determine, for each place on the globe, the tz database zone in which it is located.

• Use the time Module to Convert Epoch to Datetime in Python
• Use the datetime Module to Convert Epoch to Datetime in Python

datetime.datetime is a subclass of datetime.date.

Location-based data processing

Each country is assigned a specific block of IP addresses for use within that country. Services such as Netflix block services based on IP address. So some users send traffic through VPN gateways in various countries to mask their origin.

Each country not only have its own language, it also has a different way to display date and time.

Various programs maintain a table of 110 countries:

• https://gist.github.com/mlconnor/1887156 .csv needs Palestine and Bangladesh
• https://gist.github.com/mlconnor/1878290 is a Java program to use the csv file.
  
  

The issue with a table by country is that there are several country identifier codes:

• ISO 3166 Country Code (“USA”) used by Windows
• ISO639-2 Country Code (“US”) used by Linux
  
  

Information associated with each country:

• ISO 3166 Language Codes (“eng” or “spa” for Spanish)
• ISO639-2 Lang
• Currency code
• Telephone prefix
  
  

The preferred Date/Time format is based on LOCALE (language within country).

TODO: To determine data format, this program does a lookup of IP Address to obtain the country code. It then retrieves a country_info.csv file to do a lookup based on the MY_COUNTRY two-letter code.

LOCALE

PROTIP: LOCALE has different values on Windows vs. Linux and other systems:

if sys.platform == 'win32':
    locale.setlocale(locale.LC_ALL, 'rus_rus')  # ISO3166
else:
    locale.setlocale(locale.LC_ALL, 'ru_RU.UTF-8')  # ISO639-2 country and language
print(datetime.date.today().strftime("%B %Y"))

PROTIP: To format various locales cleanly without changing your OS locale, use an external package:

• Babel package http://babel.pocoo.org/en/latest/dates.html
• Arrow package https://arrow.readthedocs.io/en/latest/
  
  

Babel:

from datetime import date, datetime, time
from babel.dates import format_date, format_datetime, format_time
d = date(2007, 4, 1)
format_date(d, locale='en')     # u'Apr 1, 2007'
format_date(d, locale='de_DE')  # u'01.04.2007'
   

https://unicode-org.github.io/icu/userguide/format_parse/datetime/

TODO: With Unicode:

locale.setlocale(locale.LC_ALL, lang)
format_ = datetime.datetime.today().strftime('%a, %x %X')
format_u = format_.decode(locale.getlocale()[1])

• Bulgarian пет, 14.11.2014 г. 11:21:10 ч.
• Czech pá, 14.11.2014 11:21:10
• Danish fr, 14-11-2014 11:21:10
• German Fr, 14.11.2014 11:21:10
• Greek Παρ, 14/11/2014 11:21:10 πμ
• English Fri, 11/14/2014 11:21:10 AM
• Spanish vie, 14/11/2014 11:21:10
• Estonian R, 14.11.2014 11:21:10
• Finnish pe, 14.11.2014 11:21:10
• French ven., 14/11/2014 11:21:10
• Croatian pet, 14.11.2014. 11:21:10
• Hungarian P, 2014.11.14. 11:21:10
• Italian ven, 14/11/2014 11:21:10
• Lithuanian Pn, 2014.11.14 11:21:10
• Latvian pk, 2014.11.14. 11:21:10
• Dutch vr, 14-11-2014 11:21:10
• Norwegian fr, 14.11.2014 11:21:10
• Polish Pt, 2014-11-14 11:21:10
• Portuguese sex, 14/11/2014 11:21:10
• Romanian V, 14.11.2014 11:21:10
• Russian Пт, 14.11.2014 11:21:10
• Slovak pi, 14. 11. 2014 11:21:10
• Slovenian pet, 14.11.2014 11:21:10
• Swedish fr, 2014-11-14 11:21:10
• Turkish Cum, 14.11.2014 11:21:10
• Chinese 周五, 2014/11/14 11:21:10

Sample for import contain something like:

class TestMakingChange(unittest.TestCase):
 
    def setUp(self):
        self.american_coins = [25, 10, 5, 1]
        self.random_coins = [10, 6, 1]
 
        self.testcases = [(self.american_coins, 1, 1), (self.american_coins, 6, 2), (self.american_coins, 47, 5), (
            self.random_coins, 1, 1), (self.random_coins, 8, 3), (self.random_coins, 11, 2), (self.random_coins, 12, 2)]
   

Block comments

CODING CONVENTION: Block comments about the program as a whole and each function defined.

“Dunder” variables

At the top of the program file I added metadata about the file:

__repository__ = "https://github.com/wilsonmar/python-samples"
__author__ = "Wilson Mar"
__copyright__ = "See the file LICENSE for copyright and license info"
__license__ = "See the file LICENSE for copyright and license info"
__version__ = "0.0.58"  # change on every push - Semver.org format per PEP440
__linkedin__ = "https://linkedin.com/in/WilsonMar"

Variables defined with double underlines are commonly called “dunder” variables.

import datetime
print(dir(datetime))

yields attributes:

['MAXYEAR', 'MINYEAR', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_divide_and_round', 'date', 'datetime', 'datetime_CAPI', 'time', 'timedelta', 'timezone', 'tzinfo']

1. Import of Libraries

CODING CONVENTION: In our program imports are listesd in alphabetical order to make them easier to find. Most IDEs would detect when you don’t have an imported coded.

SECURITY CONSIDERATION: Generally, minimize the number of external dependencies to a small number of trusted ones from Microsoft, Amazon, etc.

Quit/Exit Program

Whenever a program runs in Python, the site module is automatically loaded into memory. So it does not need to be imported before issuing its quit() and exit() functions which raise a SystemExit exception to exit the program.

Because quit() works with the interactive interpreter, they should not be used in production code.

Also not recommended is using os._exit() method of the os module which exits a process without calling any cleanup handlers or flushing stdio buffers, which is not a very graceful.

PROTIP: The recommended way to exit production code is to use sys.exit() which also raises a SystemExit exception when executed:

import sys
print("exiting the program")
print(sys.exit())

2. Define starting time and default global values

• https://pythonguides.com/python-epoch-to-datetime/ provides examples of how to convert from one date format to any other.
  
  

This would be the first command:

start_epoch_time = time.time()

Notice that to avoid confusion, only one timestamp is captured. epoch time is obtained, then reformatted to datetime:

# start_datetime = _datetime.datetime.now()

CAUTION: This code is “naive” and not timezone aware. The time is relative to local time only.

ALTERNATIVE: For timezone-aware (rather than naive) datetime, use arrow library: see https://arrow.readthedocs.io/en/latest/

import arrow start_epoch_time = time_start=arrow.now()

Sleep

To pause/suspend the calling thread’s execution for a specified number of seconds:

import time
time.sleep(1.5)       # seconds
time.sleep(400/1000)  # milliseconds

Pause execution

CAUTION: To pause program execution until the user does not press any key, but this method only works on Windows, so wrap the command:

import os
os.system("pause")

To pause program execution for user input, in Python 3:

name = input("Please enter your name: ")
print("Name:", name)

Back in Python 2:

name = raw_input("Please enter your name: ")
print("Name:", name)

Threading Timer

Alternately, set a Timer to wait for a specific time before calling a thread that calls a function:

from threading import Timer
def nextfunction():
    print("Next function is called!")
...
t = Timer(0.5, nextfunction)
t.start()

3. Parse arguments that control program operation

Since api-sample.py was written to be used as the starting point for building other programs, it has a large scope of features coded.

• The IP Address is obtained using the requests library.
• Geolocation information based on IP address is obtained using an API usig the urllib2 library.
  
  

Included in the code are conversions of dates, floats, and formatting floats.

https://learnpython.com/blog/9-best-python-online-resources-start-learning/

Show or not

Additionally, our custom print statements make use of global variables:

show_warning = True    # -wx  Don't display warning
show_info = True       # -qq  Display app's informational status and results for end-users
show_heading = True    # -q  Don't display step headings before attempting actions
show_verbose = True    # -v  Display technical program run conditions
show_trace = True      # -vv Display responses from API calls for debugging code
   

Verbosity flags

The above sample reflects these default verbosity variables, which can be changed in the code:

The output above are issued in order of execution, explained below.

TODO: A “dev” and “prod” mode which establishes whole sets of switches.

4. Define utilities for printing (in color), logging, etc.

Printing in Color

Different colors in print output on CLI Terminal make it clear what type of information is being convayed:

• Red for failure conditions
• Yellow for warnings
• Green or White for normal information (in BOLD type)
  
  

There are external libraries (such as colorama) to enable coding to incorporate colors:

print(colored('Hello, World!', 'green', 'on_red'))

However, PROTIP: We prefer not to type names of colors (such as “GREEN”) in app code because in the future we may want to change the color scheme within changing every print() line of code. Different font codes are needed in dark backgrounds than in white backgrounds.

Ideally, we would specify text to print using a custom function that automatically incorporates the appropriate colors in the output:

print_info("Buy {widgets_to_buy} widgets")
print_warning("Free disk space on {disk_id} low: {disk_pct_free}%")
print_fail("Code {some_code} not recognized in program.")
   

# FIXME: Pull in text_in containing {}.

Internally the print_info() function would use statements that is the equivalent of:

print("*** %s %s=%s" % (my_os_platform, localize_blob("version"), platform.mac_ver()[0]),end=" ") print("%s process ID=%s" % ( my_os_name, os.getpid() ))

PROTIP: The ,end=” “ at the end of the first statement removes the line break (new line) normally issued by Python print() statements.

PROTIP: Defining statics in a class requires each to be referenced with the class name, which provides context about what that static is used for.

So rather than coding colors in every print statement, such as this:

from colorama import Fore, Back, Style

from termcolor import colored

Logging

PROTIP: Per RFC 5848, append log entries with the identity of intermediary handlers along the log custody chain.

7.2. Local in-memory SQLLite database = use_sqlite

• Functions to use SQLLite
  
  

Each country can be referenced using identifiers of either two or three characters (“US” or “USA”). So it would be useful to make use of a SQL database with an index to each type of identifier.

PROTIP: A SQL database locally created from within a Python program is as transitory (temporary) as the program instance itself. SQLite (C-language) runs inside the same process as the application.

WARNING: SQLite connection objects are not thread-safe, so no sharing connections between threads.

The Python sqlite3 module adheres to the Python Database API Specification v2.0 (PEP 249).

https://pynative.com/python-sqlite/

SQLCipher is an open-source library that applies to SQLite databases transparent 256-bit AES encryption (in CBC mode) for mobile devices (Swift, Java, Xamarin). In addition to a free Community BSD-license, Zetetic offers paid Commercial and Enterprise licenses which is “3-4X faster”. It makes use of OpenSSL. Users of the peewee ORM would use the sqlcipher playhouse module. Python driver

CAUTION: If data stored is sensitive, encryption of data in transit and at rest is still needed on “scratch” databases. For more persistant storage which lives to serve many different instances of a program, use a proper database established in a cloud enviornment.

The code has a way to figure out why a sqlite3 db script might not be working. Like the comments say, it uses 3 phases, checks if a path exist, checks if the path is a file, checks if that file’s header is a sqlite3 header.

References:

• https://www.youtube.com/watch?v=byHcYRpMgI4 from FreeCodeCamp.org/Codemy.com is the most through
• https://www.youtube.com/watch?v=pd-0G0MigUA
• https://www.youtube.com/watch?v=KHc2iiLEDoQ by Telusko
• https://www.youtube.com/watch?v=E7aY1XJX1og intro by Bryan Cafferky
• https://python-course.eu/applications-python/sql-python.php
• https://codereview.stackexchange.com/questions/182700/python-class-to-manage-a-table-in-sqlite
• https://charlesleifer.com/blog/encrypted-sqlite-databases-with-python-and-sqlcipher/
  
  

If the database is created as part of program initiation, it would minimize the time users wait for the database to be created when needed. However, this consumes more memory.

If a connection to the database remains open, it would minimize the time users wait for a connection to be established when needed. However, this may leave the database vulnerable.

https://www.youtube.com/watch?v=byHcYRpMgI4&list=RDCMUC8butISFwT-Wl7EV0hUK0BQ&start_radio=1&rv=byHcYRpMgI4

5. Obtain run control data from .env file in the user’s $HOME folder

Code in this section is used to obtain values that control a run, such as override of the LOCALE, cloud region, zip code, and other variable specs.

Be aware of unhandled conditions in Python locale.

This is needed for testing.

The code reads a file in an “.env” file in the user’s $HOME folder because that folder is away from GitHub. That file’s name by hard-coded default is:

env_file = 'python-samples.env'

The following example of the .env file contents is not put in the code because that would trigger findings in utilities that look for secrets in code.

#MY_IP_ADDRESS=""     # override of lookup done by program
IPFIND_API_KEY="12345678-abcd-4460-a7d7-b5f6983a33c7"
#MY_COUNTRY="US"      # For use in whether to use metric
LOCALE="en_US"  # "en_EN", "ar_EG", "ja_JP", "zh_CN", "zh_TW", "hi" (Hindi), "sv_SE" #swedish
#MY_ENCODING="UTF-8"  # "ISO-8859-1"
 
#MY_ZIP_CODE="59041"  # use to lookup country, US state, long/lat, etc.
#MY_US_STATE="MT"
#MY_LONGITUDE = ""
#MY_LATITUDE = ""
#MY_TIMEZONE = ""
#MY_CURRENCY = ""
#MY_LANGUGES = ""
 
OPENWEATHERMAP_API_KEY="12345678901234567890123456789012"
 
AZURE_SUBSCRIPTION_ID="12345678901234567890123456789012"   # access to info behind this requires user credentials
AZURE_REGION="eastus"
KEY_VAULT_NAME="howdy-from-azure-eastus"
 
AWS_REGION="us-east-1"
AWS_CMK_DESCRIPTION="My Customer Master Key"   # this is not a secret, but still does not belong here.
KEY_ALIAS = 'alias/hands-on-cloud-kms-alias'   # 
 
GCP_PROJECT_ID="123456etc?"
GCP_REGION="east1?"
 
VAULT_TOKEN=3340a910-0d87-bb50-0385-a7a3e387f2a8   # secret
VAULT_URL=http://localhost:8200
 
IMG_PROJECT_ROOT="$HOME"  # or "~" on macOS="/Users/wilsonmar/" or Windows: "D:\\"
IMG_PROJECT_FOLDER="Projects"

TODO: The program downloads file “python-samples.env” from GitHub to the user’s $HOME folder for reference:

Such run variables can be overridden by specifications in the program’s invocation parameters or real-time UI specification.

Some use this mechanism to retrieve API keys to services that do not ask for personal information and credit cards (such as weather apps). But storing any secret in a clear-text (unencrypted) file containing is not recommended.

CAUTION: Leaving secrets anywhere on a laptop is dangerous. One click on a malicious website and it can be stolen. It’s safer to use a cloud vault such as Amazon KMS, Azure, HashiCorp Vault after signing in.

• https://blog.gruntwork.io/a-comprehensive-guide-to-managing-secrets-in-your-terraform-code-1d586955ace1#bebe
• https://vault-cli.readthedocs.io/en/latest/discussions.html#why-not-vault-hvac-or-hvac-cli
  
  

Putting secrets in an .env file is better than putting secrets in ~/.bash_profile on macOS. See https://python-secrets.readthedocs.io/en/latest/readme.html

6. Localization

NOTE: For localized presentation, use these specialized functions which understands LOCALE localization : # atof (convert a string to a floating point number), # atoi (convert a string to integer), # str (formats a floating point number using the same format as the # built-in function str(float) but takes the decimal point into account).

Use Language Code Identifier (LCID) https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c?redirectedfrom=MSDN

my_encoding = “utf-8” # default: or “cp860” or “latin” or “ascii” “ISO-8859-1” used in SQLite.

7. Display run conditions: datetime, OS, Python version, etc.

Get = get_ipaddr

There are several ways to get the IP address addressed by the program.

7. Define utilities for managing data storage folders and files

7.1. Create, navigate to, and remove local working folders

Front-end

To obtain user input:

• PythonCard

9. Generate various calculations for hashing, encryption, etc.

Hashing is a one-way operation. Hashing works by mapping a value (such as a password) with a salt to a new, scrambled value. Ideally, there should not be a way of mapping the hashed value / password back to the original value / password.

By contrast, an encrypted value can possibly be (eventually) decrypted to its clear-text value.

So when storing passwords in databases, hashing (with a strong salt) is considered more secure than encryption and decryption (2-way operations). When a user provides a password for authentication, a hash of it is created the same way, then compared with the hash stored in the database.

https://www.python.org/dev/peps/pep-0506/

Validate each password

• Minimum 10 characters
• No all-number values
• Not Common values
  
  

Credential stuffing using credentials stolen from other websites.

Rate-limiting HTTP 403 error

Passlib

http://theautomatic.net/2020/04/28/how-to-hide-a-password-in-a-python-script/ discusses passlib, which is an external package requiring

pip install -U passlib

from passlib.context import CryptContext
   

Select one of several CryptContext objects obtained by an additional package:

• argon2 (with argon2_cffi package)
• bcrypt
• pbkdf2_sha256
• pbkdf2_sha512
• sha256_crypt
• sha512_crypt
  
  

The scheme used is specified in code such as:

password_in = "test_password"
...   
# Create CryptContext object:
context = CryptContext(
        schemes=["pbkdf2_sha256"],
        default="pbkdf2_sha256",
        pbkdf2_sha256__default_rounds=50000
) 
# hash password:
hashed_password = context.hash(password_in)
...
# Verify hashed password:
context.verify(password_in, hashed_password)
   

Creating a CryptoContext requires specifying the number of “rounds” – the number of times that a function (algorithm) runs to map a password to its hashed version. Each scheme involves several collections operations. The more rounds the more scrambling and thus more secure from brute force. More rounds also take longer to complete. Also, bcrypt and argon2 are slower to produce hashed values, and therefore, usually considered more secure.

https://github.com/python/cpython/blob/3.6/Lib/random.py

https://martinheinz.dev/blog/59 - The xkcdpass library generates strong passphrase made of words from a word/dictionary file on your system such as /usr/dict/words

Base64 Salt

A “salt” provides a random string that is appended to a password before hashing for safer storage in a database. It makes the password more random and therefore harder to guess (using rainbow tables).

Since modern computer hardware grows ever more powerful, attempt billions of hashes per second, purposely slow hash functions now need to be used for password hashing, to make it more inefficient for attackers to brute-force a password. Thus a timer on salt hash calculations.

There is the passlib library.

1. To create a random string of n bytes that is Base64 encoded for use in URLs:

   secrets.token_urlsafe([nbytes=None])

   n=16
   token_urlsafe(16)  
       # 'Drmhze6EPcv0fN_81Bj-nA'
   

   On average each n byte results in approximately 1.3 characters. If nbytes is None or not supplied, a reasonable default is used.

Python 3.6 introduced a secrets module, which “provides access to the most secure source of randomness that your operating system provides.”

https://docs.python.org/3.6/library/secrets.html

secrets.token_bytes([nbytes=None])

Return a random byte string containing nbytes number of bytes. If nbytes is None or not supplied, a reasonable default is used.

>>> token_bytes(16)  
b'\xebr\x17D*t\xae\xd4\xe3S\xb6\xe2\xebP1\x8b'

secrets.token_hex([nbytes=None])

Return a random text string, in hexadecimal. The string has nbytes random bytes, each byte converted to two hex digits. If nbytes is None or not supplied, a reasonable default is used.

>>> token_hex(16)  
'f9bf78b9a18ce6d46a0cd2b0b86df9da'

In order to generate some cryptographically secure numbers, you can call secrets.randbelow().

secrets.randbelow()
n=10
rand_num = secrets.randbelow(n)  # returns a number between 0 and n.
print(f'*** {rand_num} ')
 
# from random import SystemRandom
cryptogen = SystemRandom()
[cryptogen.randrange(3) for i in range(20)] # random ints in range(3)
    # [2, 2, 2, 2, 1, 2, 1, 2, 1, 0, 0, 1, 1, 0, 0, 2, 0, 0, 0, 0]
[cryptogen.random() for i in range(3)]  # random floats in [0., 1.)
   # [0.2710009745425236, 0.016722063038868695, 0.8207742461236148]

The above replaces use of standard pseudo-random generators os.urandom() which are not suitable for security/cryptographic purposes.

salt_size = 32
password_salt = os.urandom(salt_size).hex()  # returns a byte string
   # m\xd4\x94\x00x7\xbe\x04\xa2R'    
map(ord, os.urandom(10))
   # [65, 120, 218, 135, 66, 134, 141, 140, 178, 25]

References:

• https://tonyarcieri.com/4-fatal-flaws-in-deterministic-password-managers

Encryption and decryption

See https://www.geeksforgeeks.org/encrypt-and-decrypt-files-using-python/

1. Import required module:

   import cryptography
   from cryptography.fernet import Fernet
   

2. Enable generation of a key to encrpyt our password:

   key = Fernet.generate_key()

3. Perform encryption (Hashing is recommended because it is generally more secure):

   enc = f.encrypt(b"test_password")

   Note that the password needs to be passed in bytes.

4. Decrpyt the encrypted password using the decrypt method.

   f.decrypt(enc)

9.4. Generate a fibonacci number recursion = gen_fibonacci

The Fibonacci sequence is a sequence of numbers which is the sum of the two preceding numbers.

fibonacci_memoized_cache = {0: 0, 1: 1, 2: 2, 3: 3, 4: 5, 5: 8, 6: 13, 7: 21, 8: 34, 9: 55, 10: 89, 11: 144, 12: 233, 13: 377, 14: 610}

This was identified by Leonardo Fibonacci (1175 A.D. - 1250 A.D). BTW Fibonacci found that the quotient of the adjacent number has a proportion, roughly 1.6180, or its inverse 0.6180, also called the “golden ratio”.

An example is how quickly rabbits reproduce, starting with one pair of rabbits (male and female). It takes one month until they can mate. At the end of the second month the female gives birth to a new pair of rabbits, etc.

There are actually practical uses for Fibonacci sequences in financial technical analysis. Specifically, retracements:

• https://www.investopedia.com/articles/technical/04/033104.asp
• https://www.investopedia.com/terms/f/fibonaccilines.asp
• https://www.investopedia.com/terms/f/fibonaccitimezones.asp
  
  

https://python-course.eu/applications-python/fibonacci-to-music-score.php

def fibonacci_recursive(n):
        """Calculate using brute-force across all - for O(n) time complexity
        This is also called a "naive" implementation.
        """
        if n in {0, 1, 2}:   # the first 3 result values (0, 1, 2) are the same as the request value.
            return n
        # recursive = function calls itself:
        return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)   

The “Dynamic programming” approach is to start out with a cache of pre-calculated solutions from previous runs, such as the 15th number being 610:

The increase in Fibonucci return values grow exponentially.

def fibonacci_memoized(n):
      """Calculate using saved lookup - for O(1) time complexity"""
      if n in fibonacci_memoized_cache:  # Base case
            return fibonacci_memoized_cache[n]
      # else:  # add entry in fibonacci_memoized_cache and save to Redis/Kafka (see below)
         # TODO: If Redis is not found, issue API calls to create it.
      fibonacci_memoized_cache[n] = fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
      return fibonacci_memoized_cache[n]

Based on bottom_up_fib(n) in https://github.com/samgh/DynamicProgrammingEbook/blob/master/python/Fibonacci.py

External caching in Azure Cache for Redis

In production systems that does this kind of workload, consider the use of a Redis/Kafka in-memory cache so that several instances can be running to calculate numbers. In such a case, we want each instance to contribute to the pool of numbers.

In the memoized example, the program first checks if the value is already in the local cache. If not, it gets the whole cache set from Redis Cache.

If it’s not in the Redis Cache, calculate the new Fibonacii value and update the local cache and also adds an entry to the Redis Cache. TODO: Add the cache set to long-term storage (SQL)?

“Caching typically works well with data that is immutable or that changes infrequently. Examples include reference information such as product and pricing information in an e-commerce application, or shared static resources that are costly to construct.” – See https://docs.microsoft.com/en-us/azure/architecture/best-practices/caching

Redis would allow duplicates of a program to run in several locations, and update a central momoized_cache. The example here uses Azure Cache for Redis, a highly-scalable SaaS service fully-managed by Azure based on the open-source Redis in-memory key-value database.

It helps to write to a more permanent location (in a JSON database) because in order to scale down a Cache, a new instance needs to be setup. When cached data expires, it’s removed from the cache, and the application must retrieve the data from the original data store (it can put the newly fetched information back into cache).

Redis is designed to run inside a trusted environment that can be accessed only by trusted clients.

The Premium plan is needed for access inside a private virtual network.

https://azure.microsoft.com/en-us/services/cache/ is the marketing home page

Pricing begins at $0.022/hour ($0.528/day or $15.84/month) for the “C0” Basic service to a maximum of 256 client connections referencing up to 250 MB in the US.

For enterprise plans, the DNS name ends with …westus2.redisenterprise.cache.azure.net

To view values in Azure, there is no GUI. So install the Azure Cache for Visual Studio Code (in Preview as of Dec. 2021).

1. Click on the link above.
2. Within VSCode, click “Install” and other steps described.
3. Press Shift+Control+A or, on the left menu, click the icon with three dots if you don’t see the Azure icon.
4. In the CACHES section, click “Sign in to Azure”. This is equivalent to “az login”.
5. Close the browser tab opened automatically.
6. Select an Azure Subscription.
7. Click on the “>” next to the cache to expand it.
8. Click the filter icon to the right of a “DB” line.
9. Type in “*” (asterisk) for all key values and press Enter. In the Fibonucci memoization example, 15, 16, 17, etc. would appear under the “DB”.
10. Click on each key for another tab to appear.

References:

• https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-python-get-started
• https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/
• https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-overview

9.9 Make change using Dynamic Programming = make_change

This “Coin Changing problem” was a PDF: Codility challenge to returning change for the smallest number of bills/coins,

The call to the function is:

make_change_dynamic(34,[100,50,20,10,5,1])

The function’s signature:

def make_change_dynamic(k, C):
    # k is the amount you want back in bills/change
    # C is an array of the denominations of the currency, such as [100,50,20,10,5,1]
    # (assuming there is an unlimited amount of each bill/coin available)
    n = len(C)  # the number of items in array C
    print(f'*** make_change_dynamic: k={k} C="{C}" n={n} ')

In the array of denominations, the largest denomination appears first because we want to give out the largest bills first. For example, if k is 200, we would give back two $100 bills, not a stack of $1 bills. This is called the “greedy” method.

The plainly (“brute force”) approach is to iteratively pick the largest denomination from array C (such as 100), with each turn. It returns an array of each denomination given back as change = [20, 10, 1, 1, 1, 1]

*** make_change_dynamic: C="[100, 50, 20, 10, 5, 1]" n=6 
*** turn=0 k=34 to start 
*** turn=1 k=14 after denom=20 change 
*** turn=2 k=4 after denom=10 change 
*** turn=3 k=3 after denom=1 change 
*** turn=4 k=2 after denom=1 change 
*** turn=5 k=1 after denom=1 change 
*** turn=6 k=0 after denom=1 change 
*** After 6 turns, k=0 remaining ...
*** make_change: change_back=[20, 10, 1, 1, 1, 1] 

The assumption is an infinite number of each denomination (kind of bill/coin).

Make Change Dynamically

To optimize, we can reduce the number of “turns” and the extent of repetitive lookup of denominations. The brute-force solution does not consider those aspects.

Dynamic Programming involves breaking down a problem into solutions to sub-problems. There is a “top-down” and “bottom-up” approach to solving the problem.

PROTIP: Many of the “Dynamic Programming” solutions (such as Fibonacci) involves caching and then referencing a pre-calculated array (using memoization) instead of prepeatedly performing calculations interrively. In other words, dynamic programming generally uses more memory space to take less time.

A “bottom-up” (iterative) solution is often faster than a “top-down” (recursive) approach, but not always.

add that to output list “dp” for what is given out, and subtract it from “k”.

The global value going into the function is MAX_INT which defines an arbitrary maximum number of bills/coins for use the empty state starting point.

    dp = [0] + [MAX_INT] * k   # array of bills given out
    for i in xrange(1, n + 1):   # 
       for j in xrange(C[i - 1], k + 1):
           dp[j] = min(dp[j - C[i - 1]] + 1, dp[j])
    return dp

to use Space Complexity from O(n · k) to O(k).

References:

• VIDEO by Stephen O’Neil of codebelts.com

• VIDEO: mnemonic “FAST” method by Sam Gavis-Hughson at Byte by Byte, author of DP (Dynamic Programming) ebook for Python.

• VIDEO by Derrick Sherrill
• VIDEO by “Back to Back SWE https://b2bswe.co/change-making-problem
• VIDEO by Kevin Naughton Jr.
• VIDEO by NeetCode: 1
• VIDEO by NeetCode: 2
• VIDEO by CS Dojo from Facebook
• VIDEO: by Paul

• VIDEO by interiewing.io

• VIDEO by Hackerrank on the Coin Change problem using Memoization and DP
• https://www.hackerrank.com/domains/python

9.10 Fill knapsack = fill_knapsack

The “Knapsack” optimization challenge has many uses in the real world. (Except for Mary Poppins), every knapsack can carry a limited amount of weight. Given a value and weight for each item, how do we maximize the amount of value carried in the knapsack?

items = {(w:2, v:6), (w:2, v:10), (w:3, v:12)}  # w=weight, v=value
max weight = 5
knapsack(items, max weight) = 22  # maximum

Right away, limit our combinations that are less than the maximum weight.

The brute-force approach is to examine every possible combination of items in the knapsack. Ignore items too heavy to fill remaining space in the bag.

This Knapsack problem is the quintessential example of a dynamic programming problem. Other dynamic programming problems are variations of it.

References:

• VIDEO: ?
• https://youtu.be/Mjy4hd2xgrs
• https://www.youtube.com/watch?v=xOlhR_2QCXY
• VIDEO: by Back to Back SWE

9.8. Convert between Roman numerals & decimal = process_romans

Sample output:

*** process_romans: roman_to_int: MMXXI => 2021 
*** process_romans: int_to_roman: 2021 ==> MMXXI 

Alternative: Pure Python GeoIP API = https://github.com/appliedsec/pygeoip

12. Obtain Zip Code to retrieve Weather info = lookup_zipinfo

PROTIP: Users don’t have to provide information which can be looked up based in an API given a Zip Code: ‘country’: ‘United States’, ‘country abbreviation’: ‘US’, … ‘state’: ‘Montana’, ‘state abbreviation’: ‘MT’

# NOTE: Several place names can be associated with a Zip Code.
# TODO: Loop through a list of zip codes.
# TODO: Repeat every x minutes for updates
# TODO: Save results (in CSV or SQLite DB) for time series analysis

# Alternately:
# city_name="New York"
# city_name = input("Enter city name : ")

There is a function for code to obtain zip code. It uses a potentially problematic infinite while loop to request user input. We also use the built-in input() function because we want to minimize use of 3rd-party libraries such as PyInputPlus. See https://medium.com/code-85/the-best-way-to-request-user-input-in-python-e072a808dc82 The PyInputPlus module has a module for each data type – inputStr(), inputNum(), inputMenu() – to apply appropriate edits to input entered.

13. Retrieve Weather info from zip code or lat/long = show_weather

Weather reports report on the Kelvin scale, which is converted to Celcius and Fahrenheit scales by the program.

NOTE: The Fahrenheit metric is shown in parentheses because there are very few nations in the world that use the Fahrenheit unit of temperature.

Countries and territories that use the Fahrenheit scale are:

• United States
• Bahamas
• Cayman Islands
• Liberia
• Palau
• The Federated States of Micronesia
• Marshall Islands
  
  

A few nations use BOTH Fahrenheit and Celsius:

• Antigua and Barbuda
• Saint Kitts and Nevis
• British Virgin Islands
• Montserrat
• Belize
• Bermuda
• Turks and Caicos
  
  

All other nations in the world exclusively use the Celsius scale when measuring temperature. The Celsius is named for the Swedish astronomer Anders Celsius, who developed a scale in 1742. The 100-degree range of the Celsius scale – from freezing at 0 degrees to boiling at 100 (at sea level)– made the Celsius scale a natural fit to be a part of the metric system.

The equivalent of a Celsius temperature of 21.1 is 70 on the Fahrenheit scale.

BTW The Fahrenheit scale was initially proposed in 1724 by the Dutch-German-Polish physicist physicist Daniel Gabriel Fahrenheit. The scale is defined by two fixed points: 32 °F (the freezing point of water) and 212 °F (the boiling point of water).

There are other API’s for weather and other services:

• https://www.timeanddate.com/services/api/ has moon phases as well as astronomical positions like altitude, distance, and azimuth for every location on Earth.
• https://api.tidesandcurrents.noaa.gov/api/prod/
• https://tidesandcurrents.noaa.gov/moon_phases.shtml?year=2021&data_type=monDec

14. Retrieve secrets from local OS Keyring = use_keyring

Based on https://martinheinz.dev/blog/59

A Python application can use the operating system’s Keyring utility that stores secure credentials in an encrypted file within your $HOME directory.

The Keyvault file (by default) uses your user account login password for encryption, so it gets automatically unlocked when you login and you therefore don’t have worry about extra password.

In the code, we start by checking location of keyring config file, which is the place where you can make some configuration adjustments if needed.

We then check the active keyring and proceed with adding a password into it. Each entry has 3 attributes - service, username and password, where service acts as a namespace, which in this case would be a name of an application. To create and retrieve an entry, we can just use set_password and get_password respectively. In addition to that, also get_credential can be used - it returns a credential object which has an attribute for username and password.

14. Retrieve secrets from GitHub Action = use_github_actions

14. Retrieve secrets from Azure Key Vault = use_azure

https://docs.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python

Azure Active Directory identity library

https://github.com/rjmax/ArmExamples/tree/master/keyvaultexamples from 2015.

from azure.identity import DefaultAzureCredential

• https://pypi.org/project/azure-identity/
  
  

from azure.keyvault.secrets import SecretClient:

• see https://pypi.python.org/pypi/azure-keyvault-secrets
  
  

1. If you already have a key vault, make sure it allows template deployments:

   az keyvault update –name ExampleVault –enabled-for-template-deployment true

2. Create a new key vault and add a secret:

   az group create --name ExampleGroup --location centralus
   az keyvault create \
   --name ExampleVault \
   --resource-group ExampleGroup \
   --location centralus \
   --enabled-for-template-deployment true
   az keyvault secret set --vault-name ExampleVault --name "ExamplePassword" --value "hVFkk965BuUv"
   

3. As the owner of the key vault, you automatically have access to create secrets. If you need to let another user create secrets:

   az keyvault set-policy \
     --upn  \
     --name ExampleVault \
     --secret-permissions set delete get list
   </pre>
   
   

4. The user who deploys the template must have the Microsoft.KeyVault/vaults/deploy/action permission for the scope of the resource group and key vault, Replace “00000000-0000-0000-0000-000000000000” with the subscription ID:

   {
     "Name": "Key Vault resource manager template deployment operator",
     "IsCustom": true,
     "Description": "Lets you deploy a resource manager template with the access to the secrets in the Key Vault.",
     "Actions": [
    "Microsoft.KeyVault/vaults/deploy/action"
     ],
     "NotActions": [],
     "DataActions": [],
     "NotDataActions": [],
     "AssignableScopes": [
    "/subscriptions/00000000-0000-0000-0000-000000000000"
     ]
   }
   

5. Assign the custom role above to the user on the resource group level:

   az role definition create --role-definition ""
   az role assignment create \
     --role "Key Vault resource manager template deployment operator" \
     --assignee  \
     --resource-group ExampleGroup
   </pre>
   
   
   

References:

• https://www.youtube.com/watch?v=BErur8WwAsg - Getting Started with Microsoft Azure in Python by Jie Jenn
• https://www.youtube.com/watch?v=k2VYcYS3EIA

• https://www.youtube.com/watch?v=gC4wmZf7dAI - Enable Zero Trust with Azure AD PIM (Privileged Identity Management) and Azure Lighthouse for MSPs (Managed Service Providers)

  Azure Friday

Azure SDK for Python:

• Alternative: see my blog about Pulumi
• https://docs.microsoft.com/en-us/azure/developer/python/azure-sdk-install?tabs=pip
• https://www.youtube.com/watch?v=4xoJLCFP4_4 - Introducing the Azure SDK for Python

• https://www.youtube.com/watch?v=WER5X_zm6Aw - An introduction to the unified Azure SDK

  Azure Friday

• https://www.youtube.com/watch?v=5oIcT0HCrvI - Microsoft Azure Overview: The Azure Python SDK by Sigma Coding
• https://www.youtube.com/watch?v=_qQq6oHskUQ - Machine Learning and Python with Microsoft Azure - http://aka.ms/azuredevstreams by https://twitch.tv/enceladosaurus

References:

• https://docs.microsoft.com/en-us/python/api/overview/azure/keyvault-keys-readme?view=azure-python
• https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/key-vault-parameter?tabs=azure-cli

14.2 Use Azure Redis Cache for Memoization

“Dynamic Programming” is the strategy of reducing the “Time Complexity” of code by increasing “Storage Complexity”. Practically, rather than repeating a computation, lookup the results of the computation stored on disk or memory.

The example here is comparing the time needed to calculate Fibonacci numbers recursively versus lookup in an Azure Redis Cache instance.

14.5 Microsoft Graph API

https://devblogs.microsoft.com/microsoft365dev/new-python-quick-start-and-samples-for-microsoft-graph/

15. Retrieve secrets from AWS KMS = use_aws

To generate, encrypt, and decrypt data keys that can be used outside of AWS KMS, AWS uses two types of CMK (Customer Master Key) to encrypt up to 4KB of data:

• Symmetric CMK: 256-bit symmetric key that never leaves AWS KMS unencrypted.

• Asymmetric CMK: AWS KMS generates a key pair where the private key never leaves AWS KMS unencrypted.

References:

• https://www.learnaws.org/2021/02/20/aws-kms-boto3-guide/
• https://towardsdatascience.com/python-and-aws-ssm-parameter-store-7f0e211bb91e AWS Systems Manager’s Parameter Store
• https://aws.amazon.com/blogs/security/how-to-securely-provide-database-credentials-to-lambda-functions-by-using-aws-secrets-manager/
• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html
• https://towardsdatascience.com/python-and-aws-ssm-parameter-store-7f0e211bb91e
• Rotating IAM Keys using Boto3
• stop/start EC2 instances on a scheduled basis to save cost using AWS Lambda and CloudWatch

16. Retrieve secrets from GCP Secret Manager = use_gcp

• https://cloud.google.com/docs/authentication
• https://cloud.google.com/docs/authentication#strategies
• https://cloud.google.com/docs/authentication/getting-started
• “How to create a Google Cloud Service Account and download client json file”by Jie Jenn Jul 25, 2021

GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application.

https://cloud.google.com/docs/authentication/production - using service accounts

https://cloud.google.com/code/docs/vscode/secret-manager

https://stackoverflow.com/questions/35159967/setting-google-application-credentials-for-bigquery-python-cli

Attaching service accounts to resources for Google Cloud services is more convenient and secure than manually passing credentials. Google Cloud Client Library Application Default Credentials (ADC) automatically finds service account credentials

json-credentials-path=os.environ["GOOGLE_APPLICATION_CREDENTIALS"]

1. To create credentials to a service account setup according to https://cloud.google.com/docs/authentication/production

   export GCP_PROJECT_ID="weather-454da"
   export GCP_SVC_ACCT_NAME="memyselfandi"
   gcloud iam service-accounts create "${GCP_SVC_ACCT_NAME}" --project "${GCP_PROJECT_ID}"
   gcloud projects add-iam-policy-binding "${GCP_PROJECT_ID}" \
    --member="serviceAccount:${GCP_SVC_ACCT_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/owner"
    # Response is list of several member serviceAccount:
   gcloud iam service-accounts keys create "${GCP_SVC_ACCT_NAME}.json" --iam-account="${GCP_SVC_ACCT_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"
    # created key [3dd743379e48adb5c020de0a4ef04f0b5930fbd5] of type [json] as [wilson-svc-2112140232.json] for [wilson-svc-2112140232@weather-454da.iam.gserviceaccount.com]
   

2. For on-going reference:

   export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"

   CAUTION: PROTIP: Service Account Keys are like passwords. Whoever posseses one will be able to use it to login the account.

3. Rotate keys. Generate old keys, refer to them, delete old keys.

4. Assign Roles for Secret Manager: https://console.cloud.google.com/iam-admin/serviceaccounts?project=weather-454da&supportedpurview=project

   • Secret Manager Admin
   • Secret Manager Secret Accessor
   • Secret Manager Secret Version Adder
   • Secret Manager Secret Version Manager
   • Secret Manager Viewer
     
     

Keyless GCP Workload Identity Federation

Workload Identity Federation (g.co/workloadidentityfederation) provides keyless authentication for service accounts used by applications services (programs rather than people) outside the GCP cloud. It replaces long-lived service account access keys with short-lived (temporary) access tokens which impersonate a service account. Vidoes:

• “Service Account keys and impersonation” Mar 19, 2021 by Google Cloud Tech

• “Security Risks of Service Accounts in Google Cloud: How to Mitigate Them” by Out of DevOps

• “GitHub Workflow and Workload Identity Federation” Oct 9, 2021 by Out of DevOps

• “GCP - Workload Identity Federation - Access GCS Bucket From AWS Lambda Function” Apr 30, 2021 by Cloud Monkey
• “Keyless Entry: Securely Access GCP Services From Kubernetes” (at Cloud Next Aug. ‘19) by Google Cloud Tech
  
  

1. After installing gcloud, in a CLI Terminal, login

   gcloud login

   The above command causes the default browser to open for you to designate an account and provide its password.

2. Define values:

   # Identity pool groups for least privilege and enviornment: dev, stage, prod :
   gcloud_identity_pool_id="weather-identity-pool-211216"
   gcloud_identity_pool_desc="???"
   gcloud_identity_pool_display_name="???"
   gcloud_provider_id="external-provider-1"  # such as aws, azure, AD, Okta, K8s
   gcloud_issuer-uri="???"  # 
   gcloud_project_number="123412342341234"
   gcloud_svc_acct_email="???@"
   gcloud_identity_subject="???"
   

3. Create a Workload Identity Pool:

   gcloud iam workload-identity-pools create "${gcloud_identity_pool_id}" \
   --location="global" \
   --description="${gcloud_identity_pool_desc}" \
   --display-name="${gcloud_identity_pool_display_name}"
   

   NOTE: Several can be created.

4. Create a one-way trust:

   gcloud iam workload-identity-pools providers create-oidc "${gcloud_provider_id}" \
   --workload-identity-pool="${gcloud_identity_pool_id}" \
   --location="global" \
   --issuer-uri="${gcloud_issuer-uri}" \
   --attribute-mapping="google.subject=assertion.sub"
   

5. Create an IAM policy for impersonation:

   gcloud iam service-accounts add-iam-policy-binding "${gcloud_svc_acct_email}" \
   --role roles/iam.workloadIdentityUser \
   --member "principal://iam.googleapis.com/projects/${project_number}/locations \
   "/global/workloadIdentityPools/${gcloud_identity_pool_id}/subject/${gcloud_identity_subject}"
   

Google Text-to-Speech

The engine is powered by DeepMind.

• “Google Cloud Text-to-Speech AI API in Python - Getting Started (Part 1)” by Jie Jenn

• https://www.youtube.com/watch?v=7tDGxUxKUAo”>

https://www.youtube.com/watch?v=lKra6E_tp5U&t=0s

“Text to Speech Converter - FREE & No Limits” by Kevin Stratvert

SECTION 17. Retrieve secrets from HashiCorp Vault

See my wilsonmar.github.io/hashicorp-vault.

Local Test instance

1. Install Vault

2. Install GPG

On Ubuntu:

1. Add the HashiCorp GPG key for response “OK”.

   curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
   

2. Add the official HashiCorp Linux repository:

   apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
   

   Get:6 https://apt.releases.hashicorp.com focal/main amd64 Packages [50.1 kB]

   Fetched 59.6 kB in 1s (81.2 kB/s)

3. Update the package cache and install Vault.

   apt update && apt install vault
   

   NOTE:The package scripts prepare the vault binary to execute with memory locking capabilities with the setcap utility by adding the +ep cap_ipc_lock capabilities as described in in the Vault Configuration documentation.

4. Configure Vault with an integrated storage backend in the next challenge, and it is not recommended to use memory locking with that storage backend.

5. Remove the memory locking capabilities from the vault binary.

   setcap -r /usr/bin/vault

6. Verify the installation by obtaining the vault version locally:

   vault version
   vault -h
   

7. https://www.vaultproject.io/docs/agent

   setcap -r /usr/bin/vault

8. Get the URL of the HashiCorp Vault instance you’ll be using.

   To use Docker Compose to run a contaiinerized Vault instance on your laptop for testing, see https://modularsystems.io/blog/securing-secrets-python-vault/

   git clone https://github.com/ryanhartje/containers.git
   cd containers/consul-vault/
   docker-compose up -d
   

   Alternately,

   Hands-on lab to deploy Vault into a real environment

9. Add to python-samples.env, for example:

   VAULT_TOKEN=3340a910-0d87-bb50-0385-a7a3e387f2a8 
   VAULT_URL=http://localhost:8200
   

   PROTIP: Tokens have an associated TTL (Time to Live).

   Setup Instances

10. Create a KVv2 secrets path (called “secrets”):

    vault secrets enable kv-v2
    

11. Create a secret on the “myapp” path.

    App coding for single instance

    https://github.com/dmcteer/vault-ha/blob/master/vault-noha.py does not cache secrets in memory to reduce frequent requests to retrieve them nor manage TTLs inside of the application so that a secret is only requested at the time that it is expected to change.

12. Create a new AppRole auth method (called “myapp”).

13. Create and apply a policy to allow the “myapp” role to read the secret from the “myapp” path.

14. Generate a role ID and a secret ID for the “myapp” role.

    These two values need to be protected (e.g. in correctly permissioned files) because anyone who can get hold of them would be able to login to Vault themselves. So wrap the token and delete files immediately after being read by your app.

15. Use AppRole to login to Vault by passing a role ID & secret ID to the application. Make the role ID and the secret ID available to the application through environment variables on the system where the application is running. This can be executed securely in production environments through the use of a trusted configuration management tool for standalone hosts, or init/sidecar processes in Kubernetes.

16. Renew the token (by the running application) while it is running.

17. Write an automated script to make calls repeateded.

18. Adjust retry timers for secret retrieval when there are no available Vault clusters. This helps to prevent applications from overwhelming the Vault service.

19. Rotate secrets using Consul https://github.com/hashicorp/consul-template

20. Setup a monitoring tool such as Splunk to ingest audit logs and build notifications about who isn’t following the method outlined above, allowing you to proactively reach out to users and help them build this resiliency into their applications.

21. Conduct chaos engineering by downing the Vault server.

    High Availability

22. To improve reliability (reduce downtime) through High Availability (HA), setup an additional performance (active/active) replica cluster in a separate geographic az or region to constantly replicate data from the primary so that identical secret data exists in both places. The anticipated RTO would be a few minutes of downtime during recovery.

    

23. Either the load balancer or the application would need to:

    1. recognize when a cluster is no longer available and
    2. begin routing requests to a healthy cluster.
       
       

24. Set allowed CIDR ranges.

25. Use the additional 19 lines of code

    https://github.com/dmcteer/vault-ha/blob/master/vault-ha.py

26. Conduct “chaos engineering”:

    • Down the primary and measure the RTO time it takes for the secondary to take over
    • Down the secondary and measure the time it takes to bring up a replacement instance

References:

• https://medium.com/hashicorp-engineering/coding-for-secrets-reliability-with-hashicorp-vault-2090dd8667e

• https://www.youtube.com/watch?v=SLB_c_ayRMo - Terraform Course - Automate your AWS cloud infrastructure on freeCodeCamp.org

• https://www.youtube.com/watch?v=-leJQ20Nu0c - HashiCorp Vault without Hassle - Eric Feliksik by TheSmartbit (2017)

• https://www.youtube.com/watch?v=YGs438aJtZg - HashiCorp Vault Azure Secrets Engine Demo

• https://www.youtube.com/watch?v=ZWaKF-UXtx8 - HashiCorp Vault PKI Secrets Engine Demo for Certificate Management by TeKanAid

• https://www.youtube.com/watch?v=G46ovYs_9hs - CloudAcademy HashiCorp: Vault Identity
• https://fakrul.wordpress.com/2020/06/06/python-script-credentials-stored-in-hashicorp-vault/
• https://learn.hashicorp.com/tutorials/vault/static-secrets
• https://discuss.hashicorp.com/t/python-code-to-access-static-secret-to-access-snowflake-database/23059
• https://stackoverflow.com/questions/62606388/get-secrets-from-enterprise-vault-using-python
• https://www.youtube.com/watch?v=KxQVlrFy3Gc - using GitLab

16. Retrieve secrets from GCP = use_gcp

1. See my https://wilsonmar.github.com/gcp about getting an account, creating a project, and getting into https://console.cloud.google.com and Cloud Shell.

2. Edit the python-samples.env file with:

   PROJECT_ID="1234etc"

3. Enable billing for project

4. Use the Cloud Shell to enable the Secret Manager API:

   gcloud services enable secretmanager.googleapis.com
   

   You should see output like this:

   Operation "operations/acf.cc11852d-40af-47ad-9d59-477a12847c9e" finished successfully.

5. On your laptop, install the Secret Manager Client Library:

   pip3 install --user google-cloud-secret-manager==2.5.0
   

6. Enter the Jupyter enviornment:

   ipython

To use serverless Google Cloud Functions, specify in the requirements.txt of your Python project folder:

google-cloud-secret-manager==2.5.0

In Secret Manager, a secret is a wrapper around a collection of secret versions.

The secret stores metadata such as labels and replication, but it does not contain the actual secret.

A secret version contains the actual contents of a secret.

A secret version can be enabled, disabled, or destroyed.w

To change the contents of a secret, create a new version.

References:

• https://wilsonmar.github.io/gcp/

• Python on Google Cloud: https://cloud.google.com/python/

• Secret Manager: https://cloud.google.com/secret-manager/
• https://googleapis.dev/python/secretmanager/latest/index.html

• https://googleapis.dev/python/secretmanager/1.0.0/gapic/v1/api.html

• https://cloud.google.com/secret-manager/docs/reference/libraries#client-libraries-install-python

• Cloud Client Libraries for Python: https://googlecloudplatform.github.io/google-cloud-python/

17. Retrieve secrets from HashiCorp Vault = use_vault

https://www.amazon.com/Running-HashiCorp-Vault-Production-McTeer-ebook/dp/B08JJLGMZ3/

6.1 Calculte BMI using units of measure based on country = categorize_bmi

More APIs

• TODO: Authentication: see https://github.com/public-apis/public-apis#authentication

• TODO: OpenID Connect (OIDC): A simple identity layer on top of the OAuth framework.

• TODO: Send SMS text via Twillo

• https://hunter.io/api to find emails (25 free/month)

• TODO: Domain validator - https://developers.google.com/safe-browsing/v4
• TODO: WayBackMachine archiving https://archive.org/wayback/available?url=google.com

• TODO: https://github.com/public-apis/public-apis#url-shorteners

• TODO: Alexa https://www.youtube.com/watch?v=j8d8PQTi6uA&list=RDCMUCdiBpPE07MZ4TfFjsNh59Nw&start_radio=1&rv=j8d8PQTi6uA&t=89

• TODO: Google Assistant https://www.c-sharpcorner.com/article/creating-a-voice-assistant-using-python-and-its-libraries/

• TODO: Generate Random number, face: https://thispersondoesnotexist.com/
• TODO: https://bible-api.com/ (no-Auth like Lorem Ipsum text)
• TODO: https://developers.google.com/calendar/api/v3/reference/calendars

• TODO: Currency conversion

• TODO: Twitter account: sentiment analysis (AI)

• Picture annotation

• TODO: Flight status - https://skyscanner.github.io/slate/#api-documentation

• TODO: UPS code lookup https://github.com/public-apis/public-apis#shopping

• Facebook apps?

”””

””” Test runs:

Multi-platform?

• on MacOS Monterey - see https://wilsonmar.github.io/apple-mac-osx-versions/
• TODO: on Windows.
• TODO: on Linux Centos, Red Hat Enterprise Linux 8
• TODO: on Linux Ubuntu
• TODO: on Linux2 within AWS

Resilient? To ensure exceptions are handled properly:

• TODO: without .env file
• TODO: without wi-fi

”””

Proof by linking hash to a blockchain

Blockchains are an unalterable chain of events with time stamps.

“Chainpoint is an open standard for creating a timestamp proof of any data, file, or series of events.” One use case is to store predictions. Another use case is are agreements such as Leases since the content is hashed and thus unalterable.

The Chainpoint Hash API Gateway from Tierion.com enables regular applications to record hashes of data in the blockchain. The data itself is kept private. Use of the Hash API is free up to 3 records per second or 1,000 records per hour.

No 3rd-party library is needed, as we use import requests and import hashlib.

BLOG:

JSON Web Token (JWT) is used for authentication to https://hashapi.tierion.com/v1/auth/token The response from https://app.pluralsight.com/guides/using-the-tierion-hash-api-with-python is good for one hour.

• https://github.com/chainpoint/chainpoint-start
• https://github.com/chainpoint/chainpoint-gateway/wiki/Gateway-HTTP-API

Asana API

Asana.com is one of the early SaaS providers of team project management software.

1. Several apps integrate with it using their API:

   https://asana.com/apps

2. Specific example with how-to:

   https://anvil.works/articles/using-the-asana-api

3. Their API Docs:

   https://asana.com/guide/help/api/api

4. Sign up for an account. They have free ones.

   https://asana.com/developers

5. On their API Explorer page, get an API key

   https://developers.asana.com/explorer

6. Their OpenAPI (Swagger) specification list query perameters:

   https://asana.github.io/developer_docs_prerelease/api_reference/

   It’s generated using https://github.com/Asana/developer-docs https://github.com/Asana/api-explorer

7. Their Python library on GitHub:

   https://github.com/Asana/python-asana

• https://developers.asana.com/docs

Verify Email

In this program is code to validate email addresses.

PROTIP: Email validation is by API calls, so can be invoked immediately by JavaScript when someone types in an email on a form.

There are several email validator API services available. They all check for fake DNS domains as well as use regex functions to check whether email addresses have accepted characters, the right length, etc.

• https://mailboxlayer.com/product offers 30 API Requests/minute free. CAUTION: the API ACCESS KEY is part of the URL, which is unsafe:

  https://apilayer.net/api/check?access_key = YOUR_ACCESS_KEY & email = support@apilayer.com
   

• https://www.zerobounce.net/email-validation-pricing is free up to 100 emails per month.

• Hunter.io offers a free monthly plan of 50 email verifications with domain searches.

• Twilio’s SendGrid service has no free level.

• https://trumail.io/ has no free level:

  {
    "address": "wilsonmar@somewhere.com",
    "username": "wilsonmar",
    "domain": "gmail.com",
    "md5Hash": "17e996e1bbf467e0b15196ffdc185317",
    "suggestion": "",
    "validFormat": true,
    "deliverable": true,
    "fullInbox": false,
    "hostExists": true,
    "catchAll": false,
    "gravatar": false,
    "role": false,
    "disposable": false,
    "free": true
  }
   

• https://documentation.mailgun.com/en/latest/api-email-validation.html#email-validation returns HTTP 429 error if too many requests. An example of a response:

  {
    "address": "nonexistentemail@realdomain.com",
    "is_disposable_address": false,
    "is_role_address": false,
    "reason": [mailbox_does_not_exist],
    "result": "undeliverable",
    "risk": "high"
  }
   

  • https://rapidapi.com/auth/sign-up?referral=/pozzad/api/email-validator-1

  • https://mailchimp.com/developer/

temp-email.io

Gravatar from MD5 Hash of email

Early pioneers created, for use with WordPress, a website where people can associate (register) their email address with a picture (“avatar” image) at http://gravatar.com. It’s now used by many websites.

In this program, the feature flag view_gravatar controls whether a Gravatar lookup is attempted for an email address. The API call is made after calculating from the email address an MD5 hash (such as “5f2f71a59bd9e62b0cc5fe4cd7216968”), using hexdigest within the hashlib module.

In this program, the feature is tested using an email obtained from the env file:

some_email=os.environ.get('MY_EMAIL')  # "johnsmith@example.com"
print_verbose( some_email)
get_gravatar_url( some_email )
   

PROTIP: Here is an example of defaults specified for a function so not every parameters needs to be specified with an argument value.

def get_gravatar_url(email, size, default, rating):
    # Commentary of this is at https://wilsonmar.github.io/python-samples#view_gravatar
    hash = hashlib.md5(email.encode('utf-8')).hexdigest()
    url = "https://secure.gravatar.com/avatar/"
    ... # Validate size, rating
    url_string = url + hash +"&size="+ str(size) +"&d="+ default +"&r="+ rating
    return url_string

Gravatar.com responds to both unencrypted HTTP and HTTPS, but a different URL is used for each.

The “print_info()” function of this program outputs a URL you can copy to paste on your browser’s Address bar to get the avatar image for the email address johnsmith@example.com:

https://secure.gravatar.com/avatar/5f2f71a59bd9e62b0cc5fe4cd7216968

“default” refers to the default generated image Gravatar.com returns if absent an avatar registered to the email address specified. “identicon” is a randomly generated assortment of shapes that is specific to a commenter’s email (or IP address). Identicons allow visual representations of commenters without requiring any external sites or user interactions. “With 40 possible shapes (about 70 with inversions) in 3 possible positions, around 8000 distinguishable colors and four different rotations for each part, there should be several billion possible shape combinations which, even with the increasing chance of overlap with each additional user, should be quite enough for almost any blog.”

https://en.gravatar.com/site/implement/images/ explains the other parameters:

• size is the number of pixels, up to 2048px.
• rating uses a code like the ones on movies - it specifies the most “explicit” level based on the self-rating supplied whoever uploaded the image into Gravatar.com. “G” (“g”) is the default.
  
  

This program opens the default browser program with the URL returned:

import webbrowser
print_verbose("Opening web browser to view gravatar image of "+ some_email)
webbrowser.open(some_email_gravatar, new=2)

Test with another email by changing in the code the value of some_email.

TODO: Text Readability Score

https://github.com/brbcoding/Readability

• Automated Readability Index
• SMOG Index
• Gunning Fog
• Coleman Liau
• Flesch Kincaid
  
  

Dynamic Programming

https://www.byte-by-byte.com/dpbook/

Stand-alone execution

PROTIP: Test the program running as an stand-alone executable after bundling everything into a single .exe using the py2exe.

BTW Delphi has always been the best product for producing stand-alone .EXEs for Windows. Lazarus IDE using FreePascal

Difference between two images

https://www.youtube.com/watch?v=fUfvBnREBFc&list=RDCMUCvVZ19DRSLIC2-RUOeWx8ug&index=30

96. Text to Speech file and play mp3

Google’s Text-To-Speech API

• https://cloud.google.com/text-to-speech?hl=en
• https://cloud.google.com/text-to-speech/docs/quickstart-protocol
• https://www.youtube.com/watch?v=-AzGZ_CHzJk”>Python Voice Assistant Tutorial #1 - Playing Sound with gTTS (Google Text to Speech), by Tech With Tim Aug 31, 2019
• https://www.youtube.com/watch?v=tIFEe0W0BEA by Let’s Learn About
• https://www.youtube.com/watch?v=X9rxXFjoWzg with pyttsx3 by Rishabh Narayan
• https://www.youtube.com/watch?v=_bScjMgipkk with pyttsx3 by Parwiz Forogh
  
  

1. Select a GCP project
2. Create a service account
3. Enable Text-to-Speech

Enable project billing for Text-to-Speech.

Create and/or assign one or more service accounts to Text-to-Speech.

Download a service account credential key.

Set your authentication environment variable.

Alternately, perform audio manipulation and storing audio in a byte-sized object.

https://cloud.google.com/text-to-speech/docs/libraries

• pip install –upgrade google-cloud-texttospeech
• pip install –upgrade gtts

The gTTS function creates an object which reads the text and convert it to an audio object and mp3 audio-format file.

We can use many parameters with this function. We can reduce the speed of the output using the slow argument. The lang parameter specifies multiple languages.

Play mp3 file using the https://github.com/TaylorSMarks/playsound

Mp3 files can also be played by calling the VLC player using the VLC Python module https://wiki.videolan.org/Python_bindings

UI Libraries

The wxwidgets framework was started in 1992 in C++ for portability across Unix and Windows. It now support macOS as well. It’s free open-sourced at https://github.com/wxWidgets/wxWidgets.

Tutorial: https://zetcode.com/gui/wxwidgets/

Getting started: https://www.youtube.com/watch?v=L3IXsa9Yyr4

Native user interfaces on Windows, Macs, and Linux can be programmed in Python using https://www.wxpython.org/

Elixir after this install:

WXWIDGETS_PATH="$(brew edit wxwidgets --print-path)"
# Add the \-\-enable-compat28 flag for compatibility with Erlang's :observer
cat "$WXWIDGETS_PATH" | sed 's/args = \[/args = \["--enable-compat28",/' >> "$WXWIDGETS_PATH"
brew install wxwidgets --build-from-source

For Python: VIDEO

Coursera Google IT Automation Professional Certificate

On Coursera.com is the “Google IT Automation with Python” Professional Certificate from 2019 consists of six courses, each set over 4 weeks, using Qwiklabs. Worth it? My notes while taking it are below.

1. Crash Course on Python

2. Using Python to Interact with the Operating System

   • The official language reference may seem unintelligible to beginners. But it’s the definitive source for the most experienced devs.
   • BOOK: Automate the Boring Stuff with Python includes a lot of practical programming exercises for beginners. You can refer to this content to read more about some of the things that we’ll be discussing, and get inspired with more ideas of things that can be automated.

   • Hitchhiker’s Guide to Python

   • Installing Python 3 on MacOS with Homebrew
   • Python in the Microsoft Store for Windows 10
   • Installing Python 3 on Windows 10 with Chocolatey

   • Package management basics on Linux

   • Python IDEs and Code Editors (Guide)
   • Best Python IDEs and Code Editors

   • Top 5 Python IDEs for Data Science

   • Is it worth the time table
     
     
3. Introduction to Git and GitHub
   • Week 1: Introduction to Git and GitHub (Version Control)
   • Week 2: Advanced Git interaction, Undoing Things, Branching and Merging,
   • Week 3: Working with Remotes, Introduction to GitHub, Using a Remote Repository, Solving Conflicts

   • Week 4: Collaboration, Pull Requests, Code Reviews, Managing Projects

   • validations.py
     
     

4. Troubleshooting and Debugging Techniques (by Amanda Ballas, Google Security Systems Administrator)

   • od-cx/dev/urandom
   • top control+C
   • traffice shaping?

   • memory_profiler

   • https://realpython.com/python-concurrency/
   • https://hackernoon.com/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32
   • https://www.pluralsight.com/blog/tutorials/how-to-profile-memory-usage-in-python

   • https://www.linuxjournal.com/content/troubleshooting-network-problems

   • Eisenhower - Importance vs Urgent matrix
   • Technical Debt = make more work for ourselves later by taking shortcuts now

   • https://blog.rescuetime.com/how-to-prioritize/

   • https://simpleprogrammer.com/understanding-the-problem-domain-is-the-hardest-part-of-programming/
   • https://blog.turbonomic.com/blog/on-technology/thinking-like-an-architect-understanding-failure-domains

   • https://landing.google.com/sre/sre-book/chapters/effective-troubleshooting/

   • Download the file only once from the URL.
   • Pre-process it so that the same calculation doesn’t need to be done over and over again. This can be done in two ways (“Dynamic Programming”).
   • Create a dictionary with the start dates and then use the data in the dictionary instead of the complicated calculation.
   • Sort data by start_date and then go date by date.
     
     

5. Configuration Management and the Cloud by SRE includes Deploying Puppet 6 (“the current standard”), Automation in the Cloud, Managing Cloud Instances at Scale

   • https://quizlet.com/106907330/puppet-exam-questions-flash-cards/
   • IaC (Infrastructure as Code) - Using a version control system to deploy and manage node configurations
   • https://puppet.com/docs/puppet/latest/lang_resources.html
   • https://puppet.com/blog/deploy-packages-across-your-windows-estate-with-bolt-and-chocolatey/
   • Fact = A variable representing characteristics of a system
   • Idempotent = An action can be performed repeatedly without changing the system after the first run
   • https://en.wikipedia.org/wiki/Domain-specific_language
   • http://radar.oreilly.com/2015/04/the-puppet-design-philosophy.html
   • Puppet facts are stored in hashes. If we wanted to use a conditional statement to perform a specific action based on a fact value, what symbol must precede the facts variable for the Puppet DSL to recognize it?

   • 1:30:00 lab - Debug Puppet

   • hello-cloud.py “A simple Hello World type app which can serve on port 8000.”
     
     

6. “Automating Real-World Tasks with Python” by SRE Matt Gaunt - how to extend the capabilities of your python code by using some modules and libraries such as python image library (PIL) to work with images and also learn how to communicate with the world outside of your network such as using APIs and more.

   1. Use the Pillow Python Imaging Library (import PIL) to do the following to a batch of images: Open an image, Rotate an image, Resize an image, Save an image in a specific format in a separate directory.
   2. Python Requests two programs to communicate with each other. Data serialization to a Comma-Separated Value (CSV) or JSON (JavaScript Object Notation) file.
      
      
   • https://github.com/wilsonmar/systematic - a collection of OS level and programming tools for python, mainly just packing stuff I use all the time to one place.

References

Decrypting and encrypting strings in Python doesn’t work because strings and Integers are interned and thus persistent.

PROTIP: Use mutable bytearray() data structures elements that can be dynamically replaced.

To be safe, to dynamically resize a data structure, create a new one, copy data, and then write over the old one. Source: http://www.ibm.com/developerworks/library/s-data.html

def paranoid_add_character_to_list(ch, l):
  """Copy l, adding a new character, ch.  Erase l.  Return the result."""
  new_list = []
  for i in range(len(l)):
    new_list.append(0)
  new_list.append(ch)
  for i in range(len(l)):
    new_list[i] = l[i]
    l[i] = 0
  return new_list

For immutable data, mark data as sensitive using the memset C function within a C module. This is highly dependent on internal interpreter details such as: id having the same value as the object pointer, the offset of string data from the object pointer, etc. Incredibly brittle; do not recommend. On Linux:

import sys 
import ctypes
 
def nuke(var_to_nuke):
    strlen = len(var_to_nuke)
    offset = sys.getsizeof(var_to_nuke) - strlen - 1
    ctypes.memset(id(var_to_nuke) + offset, 0, strlen)
    del var_to_nuke               # derefrencing the pointer.

This provides an example of secure string handling

http://web.archive.org/web/20100929111257/http://www.codexon.com/posts/clearing-passwords-in-memory-with-python

https://app.pluralsight.com/library/courses/python-secure-coding-playbook/table-of-contents by Gavin Johnson-Lynn (@gav_jl, gavinjl.me)

API Security Top 10 Mobile Top 10 Internet of Things Top 10

Levenshtein/Edit Distance

The “Levenshtein Distance”, also known as “Edit Distance”, is a metric of the “distance” between two strings – the number of edit operations (substitutions and deletions) needed to transform one string into another one. Its mathematical definition is recursive (inefficient).

To compute the Levenshtein distance efficiently, use an algorithmic example of a bottom-up Dynamic Programming. A matrix containing the Levenshtein distances between all prefixes of the first string and all prefixes of the second one. We can dynamically compute the values in this matrix. The last value computed will be the distance between the two full strings. https://python-course.eu/applications-python/levenshtein-distance.php

Profiling Python Code

https://machinelearningmastery.com/profiling-python-code/

Knonwn-bad sample Python code

https://github.com/Contrast-Security-OSS/vulnpy is a library of purposely-vulnerable Python functions to install as extensions in a framework (Flask, Django, Pyramid, Falcon, WSGI, Bottle, FastAPI). These serve as a foundation for creating insecure web applications, to be used for security testing and demonstration of vulnerability scanners.

Parallel

dask.org Futures is used to scale Python libraries that you know and love (NumPy, pandas, and scikit-learn). Dask is used throughout the PyData ecosystem for parallel processing. It is included in many libraries today like Xarray, Prefect, RAPIDS, and XGBoost.

Adam Breindel held on OReilly “Scale your Python processing with Dask” Oct. 28, 2022. Dask array; Dask Bag.

References

https://towardsthecloud.com/update-macos-packages-single-command Update your macOS packages with a single command | Towards the Cloud

More about Python

This is one of a series about Python:

1. Python install on MacOS
2. Python install on MacOS using Pyenv

3. Python install on Raspberry Pi for IoT

4. Python tutorials
5. Python Examples
6. Python coding notes
7. Pulumi controls cloud using Python, etc.

8. Jupyter Notebooks provide commentary to Python

9. Python certifications

10. Test Python using Pytest BDD Selenium framework
11. Test Python using Robot testing framework

12. Testing AI uses Python code

13. Microsoft Azure Machine Learning makes use of Python

14. Python REST API programming using the Flask library
15. Python coding for AWS Lambda Serverless programming
16. Streamlit visualization framework powered by Python
17. Web scraping using Scrapy, powered by Python
18. Neo4j graph databases accessed from Python