Testing AI

What can possibly go wrong with robots smarter than humans?

Overview

Humans are limited
No rules
Predictions from Swarm
The role of humans
“Sophomoric”
Data quality
Output
Technical Debt
Coding
OpenAI
Articles
More about Python

Some companies at the “bleeding edge” are moving to “AI first” design.

For example, Foxconn is now assemblying iPhones using mostly mechanical robots rather than human robots. Tesla, BMW, and other auto manufacturers make heavy use of mechanical robots.

Simerlerly, Uber drivers are human until they are replaced by driverless cars.

By 2017, Artificial Intelligence programs have beaten world champions in Jeopardy, chess, go, and poker. Robots are at par with Olympic atheletes in table tennis.

Hit songs are written with emotional insights gained from scraping millions of conversations, newspaper headlines and speeches” “Not Easy” reached number four in the iTunes Hot Tracks chart, and number six in the alternative chart, within 48 hours of its release.

In offices, the trend is to replace people reading lines on screens. Instead of creating lines on various charts for analysis by people to make decisions, computers are making decisions.

Humans are limited

“Machines will be capable, within 20 years, of doing any work a man can do.” –Herbert Simon (1916-2001), Nobel Laureate

This is because a human can only focus on a few things at a time.

Because humans cannot respond quick enough, instead of sending alerts for people to take action, computers are beginning to take action automatically.

Self-driving cars by Tesla, comma.ai, and others are a manifestation of this trend.

A lot can also happen in a few seconds within a computer. So vigilant actions such as scanning for malware need to be done real-time.

A program can look for patterns in behavior and alert people to new threats detected.

In operation of computers, configuration settings are increasingly being updated by programs instead of people editing files.

No rules

Traditionally, programmers hand-code rules to detect and respond to known threats.

But this has not kept up.

AI (neural networks in particular) can now discover, in real-time, threats such as malware installation, phishing attacks, and brute-force intrusions which programmers did not anticipate.

They can do that because Big Data systems enable the analysis of massive floods of data quickly. Many computers in the cloud (with fast Google Fiber network links) now process data faster than in the past.

The increased scope of AI’s processing capabilities now means it can analyze many more variables quickly. For example, to identify malware, an AI program can quickly scan every email for phising by looking for clues such as the originating IP address, word choice and phrasing, and many other factors.

Predictions from Swarm

AI can be designed to make prediction based on data analyzed.

Startup Unanimous A.I. (uni.ai) has, since 2015, been making accurate predictions like who will win contests such as the Superbowl, March Madness, US presidential debates, the Kentucky Derby superfecta, Academy Awards, etc. It has been more accurate than individual experts.

Its software platform (called UNU) milks conclusions not from algorithms, but from the “collective wisdom” of group of breathing people who influence each other by their vote, in real-time, like a Ouiji board. (

The role of humans

In the above scenarios, the role of human operators is to make sure the data sets fed into an AI engine are accurate and robust.

Data quality is more important than ever to weed out false positives. The old adage “garbage in garbage out” applies even more today. Systems can only be as intelligent as the data it analyzes.

More importantly, AI adapts rules to deal with new threats.

AI does that by analyzing judgements human experts make.

“Sophomoric”

VIDEO: Deep Neural Networks are Easily Fooled by Evolving AI Lab

When an early-model Tay chatbot was first introduced in 2015, Microsoft shut it down a week after launch because it began spewing out racist and sexist texts because it lacked the filter that most human kids learn from parents.

Data quality

Normalizing Data
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/normalize-data

TanH
https://reference.wolfram.com/language/ref/Tanh.html

ZScore
http://stattrek.com/statistics/dictionary.aspx?definition=z-score
http://howto.commetrics.com/methodology/statistics/normalization/

Min Max
https://www.quora.com/What-is-the-meaning-of-min-max-normalization

PCA
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/principal-component-analysis
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/principal-component-analysis
https://stackoverflow.com/questions/9590114/importance-of-pca-or-svd-in-machine-learning

Singular Value Decomposition (SVD)
http://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/

Canonical-correlation analysis (CCA)
https://en.wikipedia.org/wiki/Canonical_correlation

http://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/</p>

Develop Machine Learning Models

Team Data Science
https://docs.microsoft.com/fi-fi/azure/machine-learning/team-data-science-process/python-data-access

K-Means
https://www.datascience.com/blog/k-means-clustering

Confusion Matrix
http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/
https://en.wikipedia.org/wiki/Confusion_matrix
https://en.wikipedia.org/wiki/F1_score

Ordinal Regression
https://en.wikipedia.org/wiki/Ordinal_regression

Poisson regression
https://en.wikipedia.org/wiki/Poisson_regression

Mean Absolute Error and Root Mean Squared Error
http://www.eumetrain.org/data/4/451/english/msg/ver_cont_var/uos3/uos3_ko1.htm

Cross Validation
https://towardsdatascience.com/cross-validation-in-machine-learning-72924a69872f

Output

Model training produces a checkpoint file that contains a model which already has parameters output from traning. Using checkpoint files means we can get straight to applying the model.

Technical Debt

https://www.ca.com/us/rewrite/articles/agile/accelerating-velocity-and-customer-value-with-agile-and-devops.register.html

Coding

https://www.youtube.com/watch?v=KkwX7FkLfug Neural Net in C++ Tutorial on Vimeo vinh nguyen

https://www.youtube.com/watch?v=AyzOUbkUf3M The Next Generation of Neural Networks GoogleTechTalks

https://www.youtube.com/watch?v=oYbVFhK_olY Deep Learning with Neural Networks and TensorFlow Introduction by sentdex

https://www.youtube.com/watch?v=ujBiM9stPHU Neural Network Calculation (Part 1): Feedforward Structure Jeff Heaton

OpenAI

https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618 Deep Learning (Adaptive Computation and Machine Learning series)</a> by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (of OpenAI) “is the only comprehensive book on the subject.”

Articles

http://www.computerworld.com/article/3163145/data-analytics/how-to-root-out-bias-in-your-data.html

https://blog.monkeylearn.com/sentiment-analysis-apis-benchmark/

https://medium.com/@jaredpolivka/machine-learning-with-humans-in-the-loop-lessons-from-stitchfix-300672904f80#.4n5ub8pt6

https://www.youtube.com/watch?v=zwm2C3V35Fw Artificial Intelligence - The Apex Technology of the Information Age: Goldman Sachs’ Heath Terry 2:41 general talk

More about Python

This is one of a series about Python:

Wilson Mar