Wilson Mar bio photo

Wilson Mar


Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Notes for before and after getting AI-900, AI-100, AI-102 certified, as we automate AI workflows in the Azure PaaS cloud

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


This article presents my notes toward a guided tour to introduce use of Microsoft’s Artificial Intelligence offerings running on the Azure cloud, which Microsoft calls “Cognitive services”.

TL;DR look for “PROTIP:” in this article highlight the author’s hard-won experience.You likely won’t find such information anywhere else. My contribution to the world (to you) is a less overwhelming learning sequence, one that starts with the least complex of technologies used, then more complex ones.

AI = Bing = Cortana = Cognitive Services?

Microsoft has published different lists for what services constitute its “Cognitive Services” brand name to achieve AI-enhanced solutions which mimic human intelligence.

“Cortana” was the brand-name for Microsoft’s AI. Cortana is the name of the fictional artificially intelligent character in the Halo video game series. Cortana was going to be Microsoft’s answer to Alexa, Siri, Hey Google, and other AI-powered personal assistants which respond to voice commands controlling skills that turn lights on and off, etc. However, since 2019, Cortana is considered a “skill” (app) that Amazon’s Alexa and Google Assistant can call, working across multiple platforms.

Microsoft has three service “Providers”:

Asset type Resource provider namespace/Entity Abbre- viation
Azure Cognitive Services Microsoft.CognitiveServices/accounts cog-
Azure Machine Learning workspace Microsoft.MachineLearningServices/workspaces mlw-
Azure Cognitive Search Microsoft.Search/searchServices srch-
  • Search (the “Bing” brand, before OpenAI) was separated out from the “Cognitive Services” to its own at https://docs.microsoft.com/en-us/azure/search, although it’s used in “Conversational AI” using an “agent” (Azure Bot Service) to participate in (natural) conversations. BTW: in 2019 Cortana decoupled from Windows 10 search.

    Since October 31st, 2020, Bing Search APIs transitioned from Azure Cognitive Services Platform to Azure Marketplace. The Bing Search v7 API subscription covers several Bing Search services (Bing Image Search, Bing News Search, Bing Video Search, Bing Visual Search, and Bing Web Search),

Azure IoT (Edge) Services are separate.

  • https://www.computerworld.com/article/3252218/cortana-explained-why-microsofts-virtual-assistant-is-wired-for-business.html

Azure OpenAI


  • https://azure.microsoft.com/en-us/services/openai/
  • https://learn.microsoft.com/en-us/training/paths/get-started-with-artificial-intelligence-on-azure/
  • https://wilsonmar.github.io/genai for definition of terms and tutorials about Generative AI, which produce new content based on what is described in the input. The OpenAI models are a collection of generative AI models that can produce language, code, and images.
  • https://azure.microsoft.com/pricing/details/cognitive-services/openai-service

In 2019 Microsoft invested $1 billion in the company as time on Azure cloud and to develop a large-scale AI supercomputer built exclusively for OpenAI’s research in Azure. Azure (with RBAC capabilities) power all of OpenAI’s workloads. OpenAI is a San Francisco-based artificial intelligence research laboratory. OpenAI was founded by Elon Musk, Sam Altman, Greg Brockman, and Ilya Sutskever in December 2015 (to compete with Google’s DeepMind acquisition). For that, in 2023, OpenAI exclusively licensed (closed-source) GPT-3 to Microsoft for their products and services. OpenAI’s mission is to artificial general intelligence (AGI). In 2022, OpenAI made available their GPT-3.5 foundation model for free trial, offering several categories of capabilities

  • ChatGPT: Generating natural language: summarizing complex text for different reading levels, suggesting alternative wording for sentences, etc.
  • Generating code: translating code from one programming language into another, identifying and troubleshooting bugs in code, etc.
  • Generating images from text prompts, based on OpenAI’s DALL-E model.

NOTE: Microsoft leverages OpenAI in ALL products and services (including Microsoft Teams):
Click picture for full-page view.

In the Microsoft AI Portfolio diagram above, Azure’s AI services are categorized into three groups:

  • Applied AI Services
  • Cognitive Services (the pillars of services: vision, speech, language, decision, plus the Azure OpenAI Service)
  • Azure’s Machine Learning (ML) platform

OpenAI is a “separate” service from Azure Cognitive Services because now “traditional” Azure Cognitive Services focuses on making predictions based textual and discrete data where OpenAI added to ML “attention” algorithms working on binary data (voice, images, and video) to enable it to perform “Generative AI” which produces new content based on what is described in the input.

  1. Assuming that you have a Azure account and subscription at the Azure portal https://portal.azure.com
  2. Consider pricing at https://azure.microsoft.com/pricing/details/cognitive-services/openai-service

    OpenAI put a paywall behind its more advanced GPT-4 models,

    Billing is based on 1,000 “tokens” increments, with the first 100,000 tokens per month free. Beyond that:

    “Standard” use of the older/more limited gpt-3.5-turbo model is then $0.002 per 1,000 tokens.

    Charges for the GPT-4 model has two dimensions: process stage and the size of the foundational model used :

    • The   “8K context” has   13 billion parameters.
    • The “32K context” has 175 billion parameters.

    • Prompt
    • Completions

    Images generated using DALL-E are $2 per 100 images.

available for free for the first 100,000 tokens per month, then $0.004 per 1,000 tokens. Doing the math,

  1. Apply for access to Asure OpenAI for your Region and Currency:


Competitive futures

Microsoft competes for talent with Google, Amazon, IBM, China’s Tencent.

BTW, by contrast, Bernard Marr identified four types of AI evolving:

  • “reactive” machines (such as Spam filters and the Netflix recommendation engine) are not able to learn or conceive of the past or future, so it responds to identical situations in the exact same way every time.

  • “limited memory” AI absorbs learning data and improve over time based on its experience, using historical data to make predictions. It’s similar to the way the human brain’s neurons connect. Deep-learning algorithms used today is the AI that is widely used and being perfected today.

  • “theory of mind” is when AI acquires decision-making capabilities equal to humans, and have the capability to recognize and remember emotions, and adjust behavior based on those emotions.

  • “self-aware”, also called artificial superintelligence (ASI), is “sentient” understanding of of its own needs and desires.


What can AI do?

Microsoft “democratizes” AI and Machine Learning by providing a front-end that hides some of the complexities, enabling them to be run possibly without programming.

  1. Microsoft’s Azure AI gallery of samples and users’ contributions:


    These are what are called “narrow” or “weak” AI.

  2. Microsoft hands-on DEMOS at:


Case studies of how people are already making use of AI/ML to save time and money:

Kinds of Cognitive Services

As of this writing, in various marketing and certification training DOCS, Azure Cognitive Services is categorized into these (which is the basis this article is arranged. Click on the bolded word to jump to it below):

  • Vision (Visual Perception) - interpret the world visually through cameras, videos, images

  • Speech - Text-to-Speech and Speech-to-Text to interpret written or spoken language, and respond in kind.

  • Language - aka Natural language Processing (NLP) to translate text (Text Analysis), etc.

  • Decision (Making) - supervised and unsupervised machine learning

  1. An alphabetical list of Microsoft’s AI/ML services at Cognitive Services pricing page at https://azure.microsoft.com/en-us/pricing/details/cognitive-services:


PROTIP: The reconciliation below between marketing Feature categories and the Kind of Cognitive service speecified in CLI az commands provides quick access to web pages and reveals some contradictions that need to be clarified, by category: The table also has a column to summarize free machine time.

Demo Kind Free API
global "CognitiveServices" See below -
? "Internal.AllInOne" ?
Features "ComputerVision" 2 TPC, 10K/mo. API
Features "CustomVision.Prediction" 10K predictions/mo. API
Features "CustomVision.Training" 2 TPS, 1 hour/mo. API
Features "Face" 20 TPM, 30K/mo. API
Features FormRecognizer 500 pages/mo. API
Features Video Indexer 600 minutes in portal, 2400 in API API
- Ink Recognizer" - -
Features "SpeechServices" (Speech-To-Text", Text-to-Speech") 1 concurrent, 5 hours/mo.* API
Features "SpeakerRecognition" 20/min, 5K/mo. API
Features Speech Translation 20/min, 5K/mo. API
Features "LUIS" 5 TPS, 10K trans/mo. API
Features "LUIS.Authoring" 5 TPS API
Features "QnAMaker" 3 TPS, 50K trans/mo, 3 docs/mo. API
Features "QnAMaker.v2" 3 TPS, 5K/mo. API
Features "TextAnalytics" (Detect Language, Sentiment, Named Entities, Key Phrases) 5K trans/mo. API
Features "TextTranslation" 2M chars/mo. API
Features "ImmersiveReader" 3M chars/mo. API
Decision (Making)
Features "AnomalyDetector" 2,000 trans/mo. API
Features "ContentModerator" (Evaluate, Find Faces, Match, OCR) 1 TPS API
Features "MetricsAdvisor" preview API
Features "Personalizer" 50K trans/mo. API
Features "Bing.CustomSearch" 50K trans/mo. -
Features "Bing.Search.v7" 3 TPS, 1000/mo API

Kinds with ? are known in websites but not listed by this az command (which you can run after you’re setup to run CLI commands:

    az cognitiveservices account list-kinds

PROTIP: CAUTION: Each service has a different maturity level in its documentation at azure.microsoft.com/en-us/downloads, such as SDK for Python open-sourced at github.com/azure/azure-sdk-for-python, described at docs.microsoft.com/en-us/azure/developer/python.

Also, individual products may be in preview or not available depnding on region.

Hybrid workflows

PROTIP: Although most Microsoft’s samples and tutorials usually focus on one service at a time, actual production work enjoyed by real end-users usually involves a pipeline of several services. For example: ingesting (stream processing) a newsfeed:


That and other flows are in Microsoft’s Architecture Center

IMPORTANT PROTIP: Microsoft allows its free “F0” tier to be applied to only a single Cognitive Service at a time. To remain free, you would need to rebuild a new Cognitive Service with a different “Kind” between steps.

Prequisites to this document

This document assumes that you have done the following:

  1. Get onboarded to a Microsoft Azure subscriptions and learn Portal GUI menu keyboard shortcuts.
  2. Setup a CLI scripting environment in shell.azure.com.
  3. Use CLI to Create a Cognitive Service to get keys to call the first REST API from among sample calls to many REST APIs: the Translator Text API.
  4. Setup PowerShell scripts

Learning Sequence: basics, simplest first

This document covers:

  1. Automatically shut down Resource Groups of a Subscription by creating a Logic App.

  2. Run an API connecting to an established endpoint (SaaS) you don’t need to setup: Bing Search.

  3. Create Functions

  4. Create a Workspace resource to run …

  5. Create a Workspace resource and
    Create Compute instance to run
    Automated ML of regression of bike-rentals.
  6. Create Compute instance to run a iPython notebook
  7. Create ML Workspace in Portal, then ml.azure.com
  8. Us cognitivevision.com to Create Custom Vision for …

  9. QnA Maker Conversational AI
  10. Train a Machine Learning model using iPython notebook
  11. IoT - “Hey Google, ask Azure to shut down all my compute instances”.

Automation necessary for PaaS

IMPORTANT PROTIP: As of this writing, Microsoft Azure does NOT have a full SaaS offering for every AI/ML service. You are required to create your own computer instances, and thus manage machine sizes (which is a hassle). Resources you create continue to cost money until you shut them down.

So after learning to set up the first compute service, we need to cover automation to shut them all down while you sleep.

So that you’re not tediously recreating everything everyday, this tutorial focuses on automation scripts (CLI Bash and PowerShell scripts) to create compute instances, publish results, then shut itself down. Each report run overwrites files from the previous run so you’re not constantly piling up storage costs.

Another reason for being able to rebuild is that you if you find that the pricing tier chosen is no longer suitable for your solution, you must create a new Azure Cognitive Search resource and recreate all indexes and objects.

When you use my Automation scripts at https://github.com/wilsonmar/azure-quickly/ to create resources the way you like, using “Infrastructure as Code”, so you can throw away any Subscription and begin anew quickly.

My scripts also makes use of a more secure way to store secrets than inserting them in code that can be checked back into GitHub.

Effective deletion hygiene is also good to see how your instances behave when it takes advantage of cheaper spot instances which can disappear at any time. This can also be used for “chaos engineering” efforts.

To verify resource status and to discuss with others, you still need skill at clicking through the Portal.azure.com, ML.azure.com, etc.


Principled AI vetting


Using Microsoft’s API algorithms and data (such as celebrity faces, landmarks, etc.) means there has be some vetting by Microsoft’s FATE (Fairness, Accountability, Transparency, and Ethics) research group: az-ai-principled-1126x610

Microsoft’s ethical principles guiding the development and use of artificial intelligence with people:

  • Fairness: AI systems should treat all people fairly.
  • Reliability & Safety: AI systems should perform reliably and safely.
  • Privacy & Security: AI systems should be secure and respect privacy.
  • Inclusiveness: AI systems should empower everyone and engage people.
  • Transparency: AI systems should be understandable.
  • Accountability: AI systems should have algorithmic accountability.

  1. DEMO: Hands on with AI/Guidelines for Human-AI Interaction: Click each card to see examples of each guideline

    https://aka.ms/hci-demo which redirects you to

    • Initially - make clear what the system can do & how well the system can do what it can do.
    • During interaction - Time services based on context; show contexually revelvant info; Match revelvant social norms; Migrate social biases.
    • When Wrong - support efficient invocation, dismissal, correction; Scope services when in doubt.
    • Over Time - remember recent interactions; learn from user behavior; update and adapt cautiously; encourage granular feedback

    PROTIP: Some fonts are real small. Zoom in to read it.


Azure AI certifications

Among Microsoft’s Azure professional certifications:

  • AI-900 is the entry-level exam ($99).

  • AI-102 (with free re-cert after 1-year) replaces
  • AI-100 (with free re-cert after 2-years) on June 30, 2021.

Both AI-100 and AI-102 are $165. The shift is from infrastructure (KeyVault, AKS, Stream Analytics) to programming C#, Python, and curl commands.

Previous exam 774 is now been retired. It was based on VIDEO: Azure Machine Learning Studio (classic) web services, which reflected “All Microsoft all the time” using proprietary “pickle” (pkl) model files. Classes referencing it are now obsolete.

DP-203 Data Engineering on Microsoft Azure goes into how to use machine learning in Azure Synapse Analytics.

LAB references DP-090 goes into implementing a Machine Learning Solution with Databricks.

Coursera video course: Developing AI Applications on Azure by Ronald J. Daskevich at LearnQuest is structured for 5 weeks. Coursera’s videos shows the text at each point of its videos. NOTE: It still sends people to https://notebooks.azure.com and covers Microsoft’s TDSP (Team Data Science Process) VIDEO:

  1. Business understanding (Charter, Objectives, Data sources, Data dictionaries)
  2. Data acquisition and understanding (Clean dataset, pipeline)
  3. Modeling
  4. Deployment model for use (testing) Flow: entry script to accept requests and score them, Deployment configs
  5. Customer Acceptance


PROTIP: Here’s a must-see website: http://aka.ms/AIFunPath which expands to Exam definitions are at Microsoft’s LEARN includes a free text-based tutorial called “Learning Paths” to learn skills:

The MS LEARN site refers to files in https://github.com/MicrosoftLearning/mslearn-ai900

Tim Warner has created several video courses on AI-900 and AI-100:

Practice tests:

  • https://www.whizlabs.com/learn/course/microsoft-azure-ai-900/
  • https://www.examtopics.com/exams/microsoft/ai-900/
  • https://www.whizlabs.com/learn/course/designing-and-implementing-an-azure-ai-solution/

2-hour AGuideToCloud video class by Susanth Sutheesh


AI-102 is intended for software developers wanting to build AI-infused applications.

AI-102 focuses on the use of pre-packaged services for AI development, not on the development of custom models using Azure Machine Learning, which is covered by exam DP-100.

First setup development environments:

  • Visual Studio Code and add-ons for C#, Python,
  • NodeJs for Bot Framework Composer and Bot Framework Emulator

AI-102 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

PROTIP: Unlike the AI-100 (which uses Python Notebooks), hands-on exercises in Microsoft’s 5-day live course AI-102T00: Designing and Implementing a Microsoft Azure AI Solution (with cloud time) consists of C# and Python programs at https://github.com/MicrosoftLearning/AI-102-AIEngineer (by Graeme Malcolm).

Modules in the course, shown at https://microsoftlearning.github.io/AI-102-AIEngineer/. Modules:

  1. Introduction to AI on Azure
  2. Developing AI Apps with Cognitive Services
  3. Getting Started with Natural Language Processing
  4. Building Speech-Enabled Applications
  5. Creating Language Understanding Solutions
  6. Building a QnA Solution
  7. Conversational AI and the Azure Bot Service
  8. Getting Started with Computer Vision
  9. Developing Custom Vision Solutions
  10. Detecting, Analyzing, and Recognizing Faces
  11. Reading Text in Images and Documents
  12. Creating a Knowledge Mining Solution


AI-100 Azure AI Engineer Associate

On June 30, 2021 Microsft retired the AI-100 exam in favor of AI-102 exam (avilable in $99 beta since Feb 2021). AI-100 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

  • Analyze solution requirements (25-30%)
  • Design AI solutions (40-45%)
  • Implement and monitor AI solutions (25-30%)



Raza Salehi (@zaalion) created on OReilly.com an AI-100 exam prep “crash course” which references his <a target=”_blank” href=”https://github.com/zaalion/oreilly-ai-100 and https://github.com/zaalion/uy-cognitve-services-crash-course

Guy Hummel’s CloudAcademy.com 7hr AI-100 video course.

McK Udemy by Anand Rao Nednur

Raza Salehi created on Pluralsight.com a series for Microsoft Azure AI Engineer (AI-100)

Practice tests:

  • https://www.whizlabs.com/learn/course/microsoft-azure-ai-100/

Sample ML Code

PROTIP: AI-102 is heavy on questions about coding.

Samples (unlike examples) are a more complete, best-practices solution for each of the snippets. They’re better for integrating into production code.

github.com/Azure-Samples from Microsoft offers samples code to use Cognitive Services REST API by each language:


A complete sample app is Microsoft’ Northwinds Traders consumer ecommerce store. install

Cognitive Services SDK Samples for:

Tim Warner’s https://github.com/timothywarner/ai100 includes Powershell scripts:

  • keyvault-soft-delete-purge.ps1
  • keyvault-storage-account.ps1
  • python-keyvault.py
  • ssh-to-aks.md - SSH into AKS cluster nodes
  • xiot-edge-windows.ps1
  • autoprice.py

Among Azure Machine Learning examples is a CLI at https://github.com/Azure/azureml-examples/tree/main/cli

Content Moderation




Response from the Text Moderation API include:

  • What type of potentially unwanted words were found (Profanity)
  • A list of potentially unwanted words found in the text.
  • Possible personally identifiable information (PII) found in the text.

Content Moderation (Evaluate, Find Faces, Match, OCR)

  1. G+/ to Content Moderator”
  2. “Create Custom Moderator”


    Could not create the marketplace item

    This marketplace item is not available.

    When working:

  3. Select Create.

  4. Enter a unique name for your resource, select a subscription, and select a location close to you.

  5. Select the pricing tier for this resource, and then select F0.

  6. Create a new resource group

  7. Select Create.

Create Cognitive Services

Azure Machine Learning 2.0 CLI (preview) examples


BTW https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-services-quickstart-code/cognitive-services-quickstart-code/ https://github.com/Azure-Samples/cognitive-services-sample-data-files

My script does the same as these manual steps:

  1. In Portal.azure.com
  2. G+\ Cognitive Services
  3. Click the Name you created.
  4. Click “Keys and Endpoint” in the left menu.
  5. Click the blue icon to the right of KEY 1 heading to copy it to your invisible Clipboard.

  6. Endpoint: https://tot.cognitiveservices.azure.com/

    TODO: DOCS: Automate above steps to create compute and server startpup script.

    PROTIP: These instructions are not in Microsoft LEARN’s tutorial.

    “Your document is currently not connected to a compute. Switch to a running compute or create a new compute to run a cell.”

  7. Click the Run triangle for “Your document is currently not connected to a compute.”
  8. “Create compute”
    • Virtual machine type: CPU or GPU
    • Virtual machine size: Select from all options (64 of them) QUESTION: What is the basis for “recommended”?
    • The cheapest is “Standard_F2s_v2” with “2 cores, 4GB RAM, 16GB storage” for Compute optimized at “$0.11/hr”. See Microsoft’s description of virtual machine types here.
    • Next
    • Compute name: PROTIP: Use 3-characters only, such as “wow” or “eat”.
    • Enable SSH access: leave unchecked
    • Create
    • Wait (5 minutes) for box to go from “Creating” to “Running”.

Kinds of Cognitive Services CLI

You would save money if you don’t leave servers running, racking up charges.

You can confidently delete Resource Groups and all resources attached if you have automation in CLI scripts that enable you to easily create them later.

Instead of the manual steps defined in this LAB, run my Bash script in CLI, as defined by this DOC:

  1. G+\ Cognitive Services.

  2. Click the +Create a resource button, search for Cognitive Services, and create a Cognitive Services resource with the following settings:
  3. Subscription: Your Azure subscription.
  4. Resource group: Select or create a resource group with a unique name.
  5. Region: Choose any available region:
  6. Name: Enter a unique name.
  7. Pricing tier: S0
  8. I confirm I have read and understood the notices: Selected.

TODO: Instead of putting plain text of cog_key in code, reference Azure Vault. Have the code in GitHub.

Azure has a cognitiveservices CLI subcommand.


Content moderator

Content Moderator services detect potentially offensive or unwanted content.


Metrics Advisor

Metrics Advisor monitors metrics and diagnoses issues.



Personalizer creates rich, personalized experiences for every user.

NLP (Natural Language Processing) services

Intro: Tutorial: https://docs.microsoft.com/en-us/learn/paths/explore-natural-language-processing

NLP enables the creation of software that can:

  • Analyze and interpret text in documents, email messages, and other sources.
  • Interpret spoken language, and synthesize speech responses.
  • Automatically translate spoken or written phrases between languages.
  • Interpret commands and determine appropriate actions.
  • Recognize Speaker based on audio.

Within Microoft, NLP consists of these Azure services (described below):


Speaker Recognition for authentication.

In contrast, Speaker Diarization groups segments of audio by speaker in a batch operation.

LUIS (Language Understanding Intelligent Service)

Think of “LUIS” as Amazon Alexa’s frienemy.

Utterances are input from the user that your app needs to interpret.


CHALLENGE: Add natural language capabilities to a picture-management bot.

  • Create a LUIS Service Resource (in Python or C#)
  • Add Intents
  • Add utterances
  • Work with Entities
  • Use Prebuilt Domains and Entities
  • Train and Publish the LUIS Model


LUIS Regional websites

Perhaps for less latency, create the LUIS app in the same geographic location where you created the service:

  • “(US) West US” in North America: www.luis.ai
  • “(Europe) West Europe” and “(Europe) Switzerland North”: eu.luis.ai
  • “(Asia Pacific) Australia East”: au.luis.ai

Create LUIS resource

NOTE: LUIS is not a service like “Cognitive Services”, but a Marketplace item:

  1. In the Azure portal, select + Create a resource.
  2. In the “Search services and Marketplace” box, type LUIS, and press Enter.
  3. In search results, select “Language Understanding from Microsoft”.
  4. Select Create.
  5. Leave the Create options set to Both (Authoring and Prediction).
  6. Choose a subscription.
  7. Create a new resource group named “LearnRG”.
  8. Enter a unique name for your LUIS service. This is a public sub-domain to a LUIS Regional websites (above)

    PROTIP: Different localities have different costs.

  9. For Authoring Location, choose the one nearest you.
  10. For Authoring pricing tier, select F0
  11. Set your Prediction location to the same region you chose for Authoring location
  12. For Prediction pricing tier, select F0. If F0 is not available, select S0 or other free/low cost tier.
  13. Select Review + Create.
  14. If the validation succeeds, select Create.
  15. After “Your deployment is complete” appears, click “Go to resource” (within Cognitive Services).
  16. Within the Resource Management section, select the “Keys and Endpoints” section.
  17. Click the blue button to the right of KEY 1’s value. That copies it to your Clipboard.
  18. If you are making use of my framework, open the environment variables file:

    code ../setmem.sh
  19. Locate the variables and replace the sample values:

    export MY_LUIS_AUTHORING_KEY=”abcdef1234567896b87f122281e9187e” export MY_LUIS_ENDPOINT=””

    SECURITY PROTIP: Putting key values in a global variables definition file separate from the code.

authoring_key = env:MY_LUIS_AUTHORING_KEY
   authoring_endpoint = env:MY_LUIS_ENDPOINT

  1. Select a LUIS Regional websites (above)
  2. Login.
  3. Choose an authoring resource (Subscription). If there are several listed, select the top one.
  4. Create a new authoring service (Azure service).

    Create a LUIS app in the LUIS portal

  5. Now at https://www.luis.ai/applications, notice at the upper-right “PictureBotLUIS (westus, F0)” where the Directory name usually appears in the Portal.

    The pages displayed will be different if you have already created a LUIS app or have no apps created at all. Select either Go to apps, or the apps option that is available on your initial LUIS page.

  6. Select “+ New app” for conversation Apps for the “Create new app” pop-up dialog.

    The LUIS user interface is updated on a regular basis and the actual options may change, in terms of the text used. The basic workflow is the same but you may need to adapt to the UI changes for the text on some elements or instructions given here.

    Take note of the other options such as the ability to import JSON or LU file that contains LUIS configuration options.

  7. Give your LUIS app a name, for example, PictureBotLUIS.

  8. For Culture, type “en-us” then select the appropriate choice for you language.

  9. Optionally, give your LUIS app a description so it’s clear what the app’s purpose is.
  10. Optionally, if you already created a Prediction Resource, you can select that in the drop-down for Prediction resource.
  11. Select Done.

  12. Dismiss the guidance dialog that may display.


    DEFINITION: Each bot action has an intent invoked by an utterance, which gets processed by all models. The top scoring model LUIS selects as the prediction.

    An utterance that don’t map to existing intents is called the catchall intent “None”.

    Intents with significantly more positive examples (“data imbalance” toward that intent) are more likely to receive positive predictions.

    Create a LUIS app with code

  13. On your local machine (laptop), install Visual Studio Code for your operating system.

  14. If you will be completing your coding with Python, ensure you have a Python environment installed locally. Once you have Python installed, you will need to install the extension for VS Code.

    Alternately, to use C# as your code language, start by installing the latest .NET Core package for your platform. You can choose Windows, Linux, or macOS from the drop-down on this page. Once you have .NET Core installed, you will need to add the C# Extension to VS Code. Select the Extensions option in the left nav pane, or press CTRL+SHIFT+X and enter C# in the search dialog.

  15. Create a folder within your local drive project folder to store the project files. Name the folder “LU_Python”. Open Visual Studio Code and Open the folder you just created. Create a Python file called “create_luis.py”.

    cd projects
    mkdir LU_Python
    cd LU_Python
    touch create_luis.py
    code .
  16. Install the LUIS package to gain access to the SDK.

    sudo pip install azure-cognitiveservices-language-luis
  17. Open an editor (Visual Studio Code)

    Create entities in the LUIS portal

  18. Be signed in to your LUIS app and on the Build tab.
  19. In the left column, select Entities, and then select + Create.
  20. Name the entity facet (to represent one way to identify an image).
  21. Select Machine learned for Type. Then select Create.

    Create a prediction key


  22. Sign in to your LUIS portal.
  23. Select the LUIS app that you want to create the prediction key for.
  24. Select the Manage option in the top toolbar.
  25. Select Azure Resources in the left tool bar.

    Unless you have already created a prediction key, your screen should look similar to this. The key information is obscured on purpose.

    A Starter Key provides 1000 prediction endpoint requests per month for free.

    Examples of utterances are on the Review endpoint utterances page on the Build tab.

    Install and run LUIS containers

    docker pull mcr.microsoft.com/azure-cognitive-services/luis:latest
  26. Specify values for Billing LUIS authoring endpoint URI and ApiKey:

    docker run --rm -it -p 5000:5000 --memory 4g --cpus 2 --mount type=bind,src=c:\input,target=/input
    --mount type=bind,src=c:\output\,target=/output

    ”–mount type=bind,src=c:\output\,target=/output” indicates where the LUIS app saves log files to the output directory. The log files contain the phrases entered when users hit the endpoint with queries.

  27. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -G \
    -d verbose=false \
    -d log=true \
    --data-urlencode "query=Can I get a 5x7 of this image?" \
  28. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -X GET \
    "http://localhost:5000/luis/v2.0/apps/{APP_ID}?q=can%20I%20get%20an%20a%205x7%20of%20this%20image&staging=false&timezoneOffset=0&verbose=false&log=true" \
    -H "accept: application/json"

DEMO JSON responses

  1. DEMO: voice control lighting in a virtual home.


  2. Select suggested utterances to see the JSON response:

    • Book me a flight to Cairo
    • Order me 2 pizza
    • Remind me to call my dad tomorrow
    • Where is the nearest club?

    Type instructions, use the microphone button to speak commands.

    LUIS identifies from your utterance your intents and entities.

    Entity types

    • List - fixed, closed set of related words (small, tiny, smallest). Case-sensitve
    • RegEx - (credit card numbers)
    • Prebuilt
    • Pattern.Any
    • Machine Learned


Run my az-luis-cli.sh.



  1. Create a Resource referenced when LUIS Authoring is defined. The resource name should be lower case as it is used for the endpoint URL, such as:


  2. The Bot Framework CLI requires Node.js.

    npm i -g npm
  3. Install Bot Framework Emulator per instructions at:

    Download to downloads folder at:

  4. Install Bot Framework Composer per instructions at:

    On a Mac, download the .dmg

  5. Use Node.js to install the latest version of Bot Framework CLI from the command line:

    npm i -g @microsoft/botframework-cli


    LUIS can generate from models a TypeScript or C# typw (program code).


  6. DEMO: https://www.luis.ai

  7. Sign in and create an authoring resource refercening the Resource Group.

    PROTIP: In the list of Cognitive services kinds, resources and subscription keys created for LUIS authoring are separate than ones for prediction runs so that utilization for the two can be tracked separately.

    https://aka.ms/AI900/Lab4 which redirects to
    Create a language model with Language Understanding which trains a (LUIS) language model that can understand spoken or text-based commands. He’s Alexa’s boyfriend, ha ha.

  8. Process Natural Lanaguage using Azure Cognitive Language Services

    https://github.com/MicrosoftLearning/AI-102-LUIS contains image files for reference by https://github.com/MicrosoftLearning/AI-102-Code-Repos https://github.com/MicrosoftLearning/AI-102-Process-Speech

    PROTIP: LUIS does not perform text summarization. That’s done by another service in the pipeline.


  • Utterance is the user’s input that a model needs to interpret, such as “turn the lights on”.
  • Entity is the word (or phrase) that is the focus of the utterance, such as “light” in our example.
  • Intent is the action or task that the user wants to execute. It reflects in utterance as a goal or purpose. For example, “TurnOn”.


Text Analytics API Programming

Techniques DOCS:

NOTE: The previous version references interactive Python Notebooks such as MS LEARN HANDS-ON LAB referencing “07 - Text Analytics.ipynb”.

  1. Look at the DEMO GUI at:


  2. Click “Next Step” through the various processing on a sentence:

    • Sentiment analysis (how positive or negative a document is)
    • Key phrase extraction

    • Entity Linking to show link in Wikipedia.
    • Bing Entity Search

    • Text to Speech services:
    • Language Detection (is it English, German, etc.)
    • Translator Text


Text Analytics API Reference

Some Text Analytics API services are synchronous and asynchronous

Cloud Academy lab “Using Text Analytics in the Azure Cognitive Services API”


Azure Text Analytics client library samples for JavaScript has step-by-step instructions:

  1. Logging into the Microsoft Azure Portal

    <a target=”_blank” href=”https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.CognitiveServices%2Faccounts”“>Cognitive Services</a>

  2. Select the service already defined for you.

    Retrieving Azure Cognitive Services API Credentials

  3. At the left menu click “Keys and Endpoint”. The Endpoint URL contains:


    The Endpoint is the location you’ll be able to make requests to in order to interact with the Cognitive Services API. The Key1 value is the key that will allow you to authenticate with the API. Without the Key1 value, you will receive unauthenticated errors.

  4. In the Azure Portal, type Function App into the search bar and click Function App:


  1. Click the only option:


You’ll be brought to the Function App blade. While a complete summary of function apps is outside the scope of this lab, you should know the function apps allow you to create custom functions using a variety of programming languages, and to trigger them using any number of events.

In the case of this lab step, you’ll set up a function app to interact with the Cognitive Services API using Node.js. You’ll then configure the function to be triggered by visiting its URL in a browser tab.

  1. On the left side of the blade, click on Functions and then + Add.

  2. Click the HTTP Trigger option:


Important: If you don’t see the HTTP trigger option, click More Templates and then Finish and view templates to see the correct option.

  1. Name the function language-detection and click Create:


  1. Click Code + Test in the left sidebar then look at the upper part of the console and switch to the function.json file:

  2. Replace the contents of the file with the following snippet and click Save:

      "bindings": [
       "authLevel": "function",
       "type": "httpTrigger",
       "direction": "in",
       "name": "req",
       "methods": [
       "type": "http",
       "direction": "out",
       "name": "$return"
      "disabled": false

The function.json file manages the behavior of your function app.

  1. Switch to the index.js file:

  2. Replace the contents of the file with the following code:

'use strict';
let https = require('https');
const subscription_key = "INSERT_YOUR_KEY_HERE";
const endpoint = "INSERT_YOUR_ENDPOINT_HERE";
const path = '/text/analytics/v2.1/languages';
module.exports = async function (context, req) {
  let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},
  let body = JSON.stringify(documents);
  let request_params = {
    method: 'POST',
    hostname: (new URL(endpoint)).hostname,
    path: path,
    headers: {
      'Ocp-Apim-Subscription-Key': subscription_key,
  let response = await makeRequest(request_params, body);
  context.res = {
    body: response
function makeRequest(options, data) {
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      let responseBody = '';

      res.on('data', (chunk) => {
        responseBody += chunk;

      res.on('end', () => {

    req.on('error', (err) => {

  1. Replace the contents of subscription_key and endpoint with the key and endpoint you retrieved in an earlier step, so that the variables look similar to this:

Copy code 1 2 const subscription_key = “4779caba69344c3d97bca9863d726af6”; const endpoint = “https://southcentralus.api.cognitive.microsoft.com/”;

  1. Click Save.

  2. While a deep understanding of code is outside the scope of this lab, you should take note of a couple of things:

The endpoint and subscription key that you set will allow the function app to communicate with your Cognitive Services API. The endpoint tells the app where to find the API, and the subscription key allows the app to authenticate. The path variable declares the exact path the app should request on the Cognitive Services API. This path system allows the API to offer many different services in one endpoint: Copy code 1 const path = ‘/text/analytics/v2.1/languages’;

The documents variable declares a list of documents that will be passed to the language detection API. The API will return the languages that these documents are most likely written in Currently there are three documents, and the goal will be to get the language used in each one:

let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},

The rest of the code simply manages the formatting of the data, requesting the language detection service from the API, and returning the result. Next, you’ll visit the function app’s URL to see it work.

  1. Look at the upper command bar and click Test/Run.

  2. Select GET as Http Method before clicking Run:


  1. View the output in the Output tab:


Notice that what’s returned is a JSON object with a “documents” object containing three results, one for each language you submitted to it. Each result has a “name” value with the predicted language and a “score” value with the likelihood that the decision is accurate. A score of 1 means that the Cognitive Services API was completely confident in its language detection. Here’s what the formatted JSON object looks like, for reference:

  "documents": [
      "id": "1",
      "detectedLanguages": [
          "name": "English",
          "iso6391Name": "en",
          "score": 1
      "id": "2",
      "detectedLanguages": [
          "name": "French",
          "iso6391Name": "fr",
          "score": 1
      "id": "3",
      "detectedLanguages": [
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1
  "errors": []
  1. Using the Azure Text Analytics API for Language Detection
  2. Using the Azure Text Analytics API for Sentiment Analysis

Sentiment Analysis

API Reference

The output is a number from 0 to 1, with 1 being the most positive language and zero being the most negative opinion expressed.

Key phrase extraction


Named Entity Recognition (NER) identifies entities in the text and group them into different entity categories, such as organization name, location, event, etc.

"documents": [
      "id": "1",
      "keyPhrases": [
        "input text"

Detect Language


vs. Content Moderator - Moderate Detect Language to Auto Correct, PII, listid, classify, language. Classification of Profanity returns JSON with several categories:

  • Category 1: sexually explicit or adult in certain situations.
  • Category 2: sexually suggestive or mature in certain situations.
  • Category 3: considered offensive in certain situations.

v2.1 API Testing Console

Input text is in a formatted JSON document files of up to 5,120 characters. Each file can contain up to 1,000 id’s, each associated with a text string.

As with GraphQL, the API returns the detected language and a numeric score between 0 and 1. Scores close to 1 indicate 100% certainty that the identified language is true. A total of 120 languages are supported.

   "id": "3",
      "detectedLanguages": [
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1

Only one language code is returned for each document submitted. Mixed language content within the same document returns the language with the largest representation in the content, but with a lower positive rating, reflecting the marginal strength of that assessment.

Named Entity Recognition


Sample Response:

  "documents": [{
    "id": "1",
    "entities": [{
      "name": "Seattle",
      "matches": [{
        "wikipediaScore": 0.15046201222847677,
        "entityTypeScore": 0.80624294281005859,
        "text": "Seattle",
        "offset": 26,
        "length": 7
      "wikipediaLanguage": "en",
      "wikipediaId": "Seattle",
      "wikipediaUrl": "https://en.wikipedia.org/wiki/Seattle",
      "bingId": "5fbba6b8-85e1-4d41-9444-d9055436e473",
      "type": "Location"
    }, {
      "name": "last week",
      "matches": [{
        "entityTypeScore": 0.8,
        "text": "last week",
        "offset": 34,
        "length": 9
      "type": "DateTime",
      "subType": "DateRange"
  "errors": []

Linked Entities from a well known knowledge base

API Reference

Speech Services

Speech-to-text (STT) has two different REST APIs:

  • Speech-to-text REST API v3.0 is used for Batch transcription and Custom Speech.

  • Speech-to-text REST API for short audio is used for online transcription as an alternative to the Speech SDK. Requests using this API can transmit only up to 60 seconds of audio per request.

Microsoft Cognitive Services Speech SDK Samples




Speech Translation (Speech-to-text aka STT)

Get the monthly subscription mobile app on iPhone, Android, or Amazon. It has a Phrasebook of common phrases.

DEMO: Speech Translation recognizes and synthesizes speech, and translates spoken languages. REMEMBER: The sequence of services involves two APIs:


Speech-to-Text API -> Speech Correction -> Machine Translation -> Text-to-Speech API

Azure Cognitive TTS Samples

“Speech Recognition” and Text Analysis are not involved in this use case.

Telephone voice menus use “Speech Synthesis”, defined by the Speech Synthesis Markup Language (SSML).




The speech-to-text service includes multiple pre-defined voices with support for multiple languages and regional pronunciations, with language detection. In addition to standard voices, neural voices leverage neural networks to overcome common limitations in speech synthesis with regard to intonation, resulting in a more natural sounding voice.

PROTIP: Neural voices are created from samples that use a 24 khz sample rate.

Speech recognition can use a acoustic model of phonemes (sounds) or a language model that matches phonemes with words.

Custom voices can be created with the text-to-speech API.

PROTIP: Since you have to use your own subscription to follow this tutorial from Microsoft, skip clicking “Launch VM mode” and follow the Python notebook on Speech on the regular Portal.

SAMPLE: https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-speech-tts/azure-cognitive-tts-samples/

  1. PROTIP: In a CLI window, run my Bash shell script to Create a Cognitive Services resource and get its two keys:

    cd ~/clouddrive/azure-quickly
    git pull

    To synthesize speech, the system typically tokenizes the text to break it down into individual words, and assigns phonetic sounds to each word. It then breaks the phonetic transcription into prosodic units (such as phrases, clauses, or sentences) to create phonemes that will be converted to audio format. These phonemes are then synthesized as audio by applying a voice, which will determine parameters such as pitch and timbre; and generating an audio wave form that can be output to a speaker or written to a file.


To specify that the speech input to be transcribed to text is in an audio file, use AudioConfig.

Change the voice used in speech synthesis by setting the SpeechSynthesisVoiceName property of the SpeechConfig object to the desired voice name.

### Translator Text (text-to-text aka TTT)

Microsoft’s Translator service can translate text between more than 90 languages and dialects (including Klingon in Star Trek), specified using ISO 639-1 two-letter language codes and 3166-1 cultural codes such as “en-US” for US English, “en-GB” for British English, “fr-CA” for Canadian French, etc.

Hands-on tool without a compute instance:

  1. translator.microsoft.com

  2. Click on “Start conversation”, log in and enter your name and language.

  3. Share the conversation code with other participants, who can join using the Micreosoft Translator app or website.

  4. Speak or type in your language to communicate with other participants in the conversation. Other participants will see your messages in their own language.

https://docs.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/overview What is Custom Translator?

Foe Parallel Data, equivalent documents in different languages:

  • https://docs.microsoft.com/en-us/azure/cognitive-services/
  • https://docs.microsoft.com/ja-jp/azure/cognitive-services/
  • https://docs.microsoft.com/ko-ko/azure/cognitive-services/


09 - Translation.ipynb

BLAH: You are asked to use your own Subscription anyway, so instead of the Exercise - Translate text and speech, use portal.azure.com directly.

A Python program can run from your laptop or mobile phone making API calls to the Translator endpoint at:



VIDEO; Raza Salehi’s 1 hr video course “Build a Translator system”.

VIDEO intro with sample code at https://github.com/microsoft/text-analytics-walkthrough

Python notebook to Translate

For response “script” : “Latn”, text was transliterated in English.

A custom translator is needed to train a model to recognize and translate domain-specific words and phrases in specific industries such as aerospace, automotive, chemistry, mechanical, etc.


Training is done by have pairs of documents (English and French, etc.).

10,000 aligned parallel sentences are neede to train a translator.

In addition to Microsoft Office formats, files with extension .ALIGN for parellel languages are perfectly aligned. Translation Memory systems can export parallel documents in XLF, XLIFF, TMX, suffix. Microsoft’s LocStudio files have .LCL suffix.

Translation runs can each take several hours. So batch processing is supported.

If you don’t have admin

Need admin approval
Mt Studio Web Prod
Mt Studio Web Prod needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.


  • Microsoft Azure Cognitive Services: Translator Text API” video course</a> by William Myers

Immersive Reader

Other sample code


Vision services

App for the blind: VIDEO: INTRO: SeeingAI.com. Permissions for the “See It All” app are for its internal name “Mt Studio Web Prod”.


HISTORY: In 2014, Microsoft showed off its facial recognition capabilities with how-old.net to guess how old someone is. At conferences they built a booth that takes a picture.

LEARN: https://docs.microsoft.com/en-us/learn/modules/read-text-computer-vision/

DEMO: Seeing AI app talking camera narrates the world around blind people.

  • Semantic segmentation is the ML technique which individual pixels in the image are classified according to the object to which they belong.
  • Image analysis

  • Face detection, analysis, and recognition
  • Optical character recognition (OCR) for small amounts of text
  • The Read API works asynchronously on images with a lot of text, to parse pages, lines, and words.
  • Video Indexer service analyzes the visual and audio channels of a video, and indexes its content.

Computer Vision

Computer Vision” analyzes images and video to extract descriptions, tags, objects, and text. API Reference, DOCS, INTRO:


  • Read the text in the image
  • Detects Objects
  • Identifies Landmarks
  • Categorize image

Flow API samples

https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?tabs=visual-studio&pivots=programming-language-csharp READ

Computer Vision demo

  1. Select images and review the information returned by the Azure Computer Vision web service:

    DEMO: https://aidemos.microsoft.com/computer-vision

    1. Click an image to see results of “Analyze and describe images”. Objects are returned with a bounding box to indicate their location within the image.

    2. Click “Try another image” for another selection.
    3. Click “Next step”.
    4. Read text in imagery.
    5. Read handwriting
    6. Recognize celebrities & landmarks - the service has a specialized domain model trained to identify thousands of well-known celebrities from the worlds of sports, entertainment, and business. The “Landmarks” model can identify famous landmarks, such as the Taj Mahal and the Statue of Liberty.

    Additionally, the Computer Vision service can:

    • Detect image types - for example, identifying clip art images or line drawings.
    • Detect image color schemes - specifically, identifying the dominant foreground, background, and overall colors in an image.
    • Generate thumbnails - creating small versions of images.
    • Moderate content - detecting images that contain adult content or depict violent, gory scenes.

    Computer Vision API shows all the features.


  1. Right-click “Launch VM mode” for the “AI-900” lab on a Window VM.
  2. X: Click the Edge browser icon.
  3. X: Click to remove pop-ups.
  4. Go to portal.azure.com
  5. Sign in using an email which you have an Azure subscription.
  6. Type the password. You can’t copy outside the VM and paste into it.
  7. X: Do not save your password.
  8. Open Visual Studio to see
  9. X: On another browser tab, view the repo (faster):


  10. Follow the instructions in the notebook to create a resource, etc.

    TODO: Incorporate the code and put it in a pipeline that minimizes manual actions.

  11. To take a quiz and get credit, click in the VM “here to complete the Learn module with a Knowledge Check.

Custom Vision

Azure Custom Vision trains custom models referencing custom (your own) images. Custom vision has two project types:

  • Image classification is a machine-learning based form of computer vision in which a model is trained to categorize images based on their (class or) primary subject matter they contain.

  • Object detection goes further than classification to identify the “class” of individual objects within the image, and to return the coordinates of a bounding box that indicates the object’s location.

LAB: Steps:

  1. Create state-of-the-art computer vision models.
  2. Upload and tag training images.
  3. Train classifiers for active learning.
  4. Perform image prediction to identify probable matches to a trained model.
  5. Perform object detection to locate elements within an image and return a bounding box.

  6. Open


    DOCS: az-ai-image-class-623x410


    aka.ms/learn-image-classification which redirects to

  8. Instructions are at

  9. Load the code from:


https://docs.microsoft.com/en-us/learn/modules/evaluate-requirements-for-custom-computer-vision-api/3-investigate-service-authorization Custom Vision APIs use two subscription keys, each control access to an API:

  • A training key to access API members which train the model.
  • A prediction key to access API members which classify images against a trained model.


References: CV API

Ink Recognizer


Ink converts handwriting to plain text, in 63+ core languages.

It was deprecated on 31 January 2021.

https://docs.microsoft.com/en-us/azure/cognitive-services/ink-recognizer/quickstarts/csharp Quickstart: Recognize digital ink with the Ink Recognizer REST API and C#

QUESTION: Does it integrate with a tablet?

Video Indexer

Media Services & Storage Account:

  1. In Portal, Media Services blade.

  2. Specify Resource Group, account name, storage account, System-managed identity.


    Video Indexer

  3. In a browser, go to the Video Indexer Portal URL:


    NOTE: Video Indexer is under Media Services rather than Cognitive Services.

  4. Click the provider to login: AAD (Entra) account, Personal Microsoft account, Google.

    PROTIP: Avoid using Google due to the permissions you’re asked to give:

  5. Say Yes to Video Indexer permission to: Access your email addresses & View your profile info and contact list, including your name, gender, display picture, contacts, and friends.

    NOTE: You’ll get an email with subject “Your subscription to the Video Indexer API”.

    On your mobile phone you’ll get a “Connected to new app” notice for Microsoft Authenticator.

  6. Click “Account settings”. PRICING: up to 10 hours (600 minutes) of free indexing to website users and up to 40 hours (2,400 minutes) of free indexing to API users. Media reserved units are pre-paid. See FAQ

  7. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in


  8. Switch back.
  9. In Account settings, click “Copy” to get the Account ID GUID in your Clipboard.
  10. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in


  11. Highlight the sample value and paste (Command+V).
  12. Switch back.

  13. Go to the “Azure Video Analyzer for Media Developer Portal”:DOCS


  14. Click “Sign In”. Click “Profile”

    NOTE: The UI has changed since publication of Microsoft’s tutorial, which says “Go to the Products tab, then select Authorization.”

  15. Click “Show” on the “Primary key” line. Double-click on the subscription key to copy to Clipboard (Command+C).
  16. Switch to the file which defines Azure environment variable VIDEO_INDEXER_API_KEY (in setmem.sh) as described in


  17. Highlight the sample value and paste (Command+V).
  18. Switch back.

    Upload video using Portal GUI


    Upload video using program


  19. DOCS: Select the Azure Video Indexer option for uploading videos: upload from URL (there is also send file as byte array by an API call, which has limits of 2 GB in size and a 30-minute timeout.


    az-video-upload.py in https://wilsonmar.github.io/azure-quickly

  20. Make an additional call to retrieve insights.

  21. Reference existing asset ID

    Search Media files

  22. In “Media files” at https://www.videoindexer.ai/media/library

  23. Click “Samples”, and click on a video file to Play to see the media’s people, topics (keywords).

    NOTE: Search results include exact start times where an insight exists, possibly multiple matches for the same video if multiple segments are matched.

  24. Click a tag to see where it was mentioned in the timeline.

    Alternately, use the API to search: ???

    Model customizations

    Each video consists of scenes grouping shots, which each contain keyframes.

    • A scene represents a single event within the video. It groups consecutive shots that are related. It will have a start time, end time, and thumbnail (first keyframe in the scene).

    • A shot represents a continuous segment of the video. Transitions within the video are detected which determine how it is split into shots. Shots have a start time, end time, and list of keyframes.

    • Keyframes are frames that represent the shot. Each one is for a specific point in time. There can be gaps in time between keyframes but together they are representative of the shot. Each keyframe can be downloaded as a high-resolution image.

  25. Click “Model customizations”

  26. Set as thumbnail another keyframe.



Azure Face


Azure “Face” is used to build face detection and facial recognition solutions in five categories:

  • Face Verification: Check the likelihood that two faces belong to the same person.
  • Face Detection: Detect human faces in an image.
  • Face Identification: Search and identify faces.
  • Face Similarity: Find similar faces.
  • Face Grouping: Organize unidentified faces into (face list) groups, based on their visual similarity.

NOTE: On June 11, 2020, Microsoft announced that it will not sell facial recognition technology to police departments in the United States until strong regulation, grounded in human rights, has been enacted. As such, customers may not use facial recognition features or functionality included in Azure Services, such as Face or Video Indexer, if a customer is, or is allowing use of such services by or for, a police department in the United States.

A face location is face coordinates – a rectangular pixel area in the image where a face has been identified.

The Face API can return up to 27 landmarks for each identified face that you can use for analysis. Azure allows a person to can have up to 248 faces. There is a 6 MB limit on the size of each file (jpeg, png, gif, bmp).

Face attributes are predefined properties of a face or a person represented by a face. The Face API can optionally identify and return the following types of attributes for a detected face:

  • Age
  • Gender
  • Smile intensity
  • Facial hair
  • Head pose (3D)
  • Emotion

Emotions detected in JSON response is a floating point number:

  • Neutral
  • Anger
  • Contempt
  • Disgust
  • Fear
  • Hppiness
  • Sadness
  • Surprise

PROTIP: “Happiness: 9.99983543,” is near certainty at 1.0. 2.80234E-08” indicates 8


DEMO: LAB: https://github.com/microsoft/hackwithazure/tree/master/workshops/web-ai-happy-sad-angry

Create a Face API subscription

Subscribe to the Face API:

  1. Sign in to the Azure portal.
  2. Go to
    • Create a resource > AI + Machine Learning > Face</a>
  3. Enter a unique name for your Face API subscription name in variable MY_FACE_ACCT

    Paste in setme.sh

    export MY_FACE_ACCT=faceme
  4. Choose “westus”, the Location nearest to you.
  5. Select F0 the free or lowest-cost Pricing tier.
  6. Check “By checking this box, I certify that use of this service or any service that is being created by this Subscription Id, is not by or for a police department in the United States.”
  7. Click “Create” to subscribe to the Face API.

  8. When provisioned, “Go to resource”.
  9. In “Keys and Endpoint”, copy Key1 and paste in setme.sh

    export MY_FACE_KEY1=subscription_key

The endpoint used to make REST calls is “$MY_FACE_ACCT.cognitiveservices.azure.com/”




https://docs.microsoft.com/en-us/azure/cognitive-services/Face/Overview What is the Azure Face service?


Face API



Azure Form Recognizer

“Form Recognizer” extracts information from images obtained from scanned forms and invoices.








  • Image classification - https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/01%20-%20Image%20Analysis%20with%20Computer%20Vision.ipynb

  • Object detection - https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/02%20-%20Image%20Classification.ipynb

Conversational AI

HISTORY: In 2015, Microsoft unleashed the Tay chat bot, then had to bring it down after hackers submitted enough racial slurs that they fooled the system into thinking that was normal and acceptable.

HISTORY: XiaoIce, a chatbot Microsoft launched in China, “has more than 200 million users, has engaged in 30 billion conversations, and has an average conversation length of 23 turns, which averages out to about half an hour, achieving human parity at translation from Chinese to English. Japan-based Rinna and the US-based Zo)


A Bot Framework enables the creation of Virtual Assistant

A LUIS app creates these types of entities:

  • Machine-learned
  • List
  • RegEx
  • Pattern.any


Bots are extended by Skills

QnA Maker

The cognitive service name “QnA Maker” (Question and Answer Maker)</a> is a cloud-based API service that lets you create a conversational question-and-answer layer over your existing data. The service enables the building of knowledge bases of questions and answers that form the basis of a dialog between a human and an AI agent.

Microsoft created the QnA Maker portal to make it easier than writing code to create and manage knowledge bases using the QnA Maker REST API or SDK.

The knowledge base gets smarter as it continually learns from user behavior.

The knowledge base can be built by extracting questions and answers from your semi-structured content, including FAQs, manuals, and documents.

QnA Maker limits control the size of Knowledge base.

### Create QnA Service

https://github.com/Microsoft/BotBuilder-CognitiveServices/tree/master/CSharp/Samples/QnAMaker is only for C#.

  1. View the DOCS:

    View the v2 (previous release)

  2. The Jupyter notebook:


  3. On an appropriate browser profile (personal or work email) associated with the subscription you wnat to use.
  4. Go to the QnA Maker portal at:


  5. Sign in.


  6. “Create a knowledge base” from the top menu.
  7. STEP 1: “Create a QnA service”, which opens another tab
  8. Basics: During testing, do NOT click the checkbox for “Managed”. In prod, telemetry and compute are included automatically with your QnA Maker resource. If you do not select managed, you will be prompted to create an App Insights and App Service resources for the required telemetry and compute that you will have to manage for your QnA Maker resource. Read more <a target=”_blank” href=”“https://aka.ms/qnamaker-createoptions-description”>here</a>.

  9. Subscription: Your Azure subscription
  10. Resource group: Select an existing resource group or create a new one
  11. Resource group location: “(US) West US”

  12. Name: A unique name for your QnA resource (App name)
  13. Pricing tier: both are “3 transactions per second, 100 transactions per minute”:
    • Free F0 (3 managed documents per month, 5…)
    • Standard S0 ($10 per month for unlimited documents …)

    App Service details - for runtime :

  14. Azure Search location: Any available location
  15. Azure Search pricing tier: F
    • Free F (50 MB in 3 indexes)
    • Basic B (2 GB in 15 indexes, 3 scale-out units per service on 1 partition with 3 replicas)
    • Standard S1 (25 GB in 50 indexes) - default
    • Standard S2 (100 GB in 200 indexes)
    • Standard S3 (200 GB in 200 indexes)
    • Standard S3 (1000 Index/Partition)
    • L1 (1 TB in 10 indexes, High Storage, 36 units)
    • L2 (2 TB in 10 indexes, High Storage, 36 units)

    Note: If you have already provisioned a free-tier QnA Maker or Azure Search resources, your quota may not allow you to create another one. In which case, select a tier other than F0 / F.

    App Service details - for runtime :

  16. App name: Same as Name (“.azurewebsites.net” will be appended automatically)
  17. Website location: Same as Azure Search location

    App insights details - for telemetry and chat logs :

  18. App insights: Disable, which will hide the “App insights location”, but appear in Review.

  19. Click “Review + create”. Create. NOTE: when these resources complete deployment:
    • Microsoft.Web/sites/config
    • Microsoft.CognitiveServices/accounts
    • Microsoft.Search/searchServices
    • microsoft.insights/components
    • microsoft.insights/components
    • Microsoft.Web/sites
    • Microsoft.Search/searchServices
    • Microsoft.Web/serverfarms

  20. While you wait for the dots to stop flashing “Deployment in progress”, return to the QnA Maker portal tab. You may have timed out.

  21. When “Your deployment is complete”, click “Go to resources” for “Congratulations! Your keys are ready.”

    Connect QnA service to KB

    STEP 2: Connect your QnA service to your KB.

  22. “Refresh” the list of available QnA service resources.
  23. Microsoft Azure Directory ID: The Azure directory ID for your subscription
  24. Azure subscription name: Your Azure subscription
  25. Azure QnA service: The QnA service resource you created in the previous step

    NOTE: In the Preview there is a checkbox “Enable language setting per knowledge base”.

  26. Language: English

    STEP 3: Name your KB.

  27. Type a name: For example: “Margie’s Travel KB”. Spaces are allowed?

    STEP 4: Populate your KB.

  28. ”+ Add URL”
  29. Copy and paste this example URL:


  30. Add file

  31. chit-chat: Adding “chit-chat” to your knowledge base (by selecting a personality) automatically adds questions and responses to your knowledge base, which enables your bot to answer small-talk questions in a voice (personality) of your choice:
    • None
    • Professional « select this.
    • Friendly
    • Witty
    • Caring
    • Enthusiastic

    QUESTION: What is the range of popularity?

    QUESTION: Extraction? I’m stuck here.

  32. Do NOT check “Enable multi-turn extraction from URLs, .pdf or .docx files.”

  33. Click “Create your KB”. Wait for a minute or so while your Knowledge base is created.

  34. Review the questions and answers that have been imported from the FAQ document and the professional chit-chat pre-defined responses.

    Test the knowledge base

    Make an API call


    https://go.microsoft.com/fwlink/?linkid=2100213 Coding</a>

Azure Bot Service

provides a platform for creating, publishing, and managing bots. Developers can use the Bot Framework to create a bot and manage it with Azure Bot Service - integrating back-end services like QnA Maker and LUIS, and connecting to channels for web chat, email, Microsoft Teams, and others.

Microsoft Bot Framework supports two approaches to integrate bots with agent engagement platforms such as Customer support service:

  • Bot as agent distributes calls to bots on the same level as live (human) agents. Handoff protocols regulate a bot’s disengagement and transfer to live person.
  • Bot as proxy presents a bot to filter interaction with live people.
  1. DEMO: See a healthcare bot built using the Azure Bot Service:

    <a target=”_blank”” href=” https://www.microsoft.com/research/project/health-bot/”> https://www.microsoft.com/research/project/health-bot</a>

    Select the option to Try a demo of an example end-user experience. Use the web chat interface to interact with the bot.

MS LEARN: Create a Bot with the Bot Framework Composer

  1. Run the Python Jupyter notebook

    https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/11%20-%20QnA%20Bot.ipynb Conversational AI LAB

  2. Sign in using the Microsoft account associated with your Azure subscription.

PROTIP: Use NVM to install Node

https://github.com/Microsoft/botbuilder-tools#install-cli-tools says to install Node.js version 10.14.1 or higher

https://github.com/microsoft/botframework-cli says to install Node.js version 12

Since the current version is now 16, we cannot use the command suggested in the doc:

    npm i -g @microsoft/botframework-cli

Bot Framework

  1. bf is the bot framework CLI command

    One-stop-shop CLI to manage your bot’s resources. BF CLI and AZ CLI together cover your end-to-end bot development workflow needs.
      @microsoft/botframework-cli/4.13.3 darwin-x64 node-v16.1.0
      $ bf [COMMAND]
      chatdown      Converts chat dialog files in <filename>.chat format into
                 transcript files. Writes corresponding <filename>.transcript
                 for each .chat file.
      config        Configure various settings within the cli.
      dialog        Dialog related commands for working with .schema and .dialog
      help          display help for bf
      lg            Parse, collate, expand and translate lg files.
      luis          Manages LUIS assets on service and/or locally.
      orchestrator  Display Orchestrator CLI available commands
      plugins       Install, uninstall and show installed plugins
      qnamaker      QnA Maker


Knowledge Mining Solution Accelerator

Search = Bing ?

  1. Get on the Bing Resource portal GUI.

  2. Define a Resource Group.

  3. Price Tier: Free

    PROTIP: Autosuggest requires the “S2” (Standard) pricing tier. Spell Check requires either S1 or S2.

  4. Click “Automation Options” link for a JSON Template.
  5. Click “Create” at the top.
  6. Create

    Notice the service name at the upper left is “Microsoft.BingSearch” and has a Global location. It’s Endpoint is: https://api.bing.microsoft.com/

Search for AI enrichment of docs

The Azure Cognitive Search service uses a Cognitive Search resource to support AI-powered search and knowledge mining solutions such as:

  • Index documents and data from a range of sources.
  • Use cognitive skills to enrich index data.
  • Store extracted insights in a knowledge store for analysis and integration.


“Document cracking” during indexing extracts text content from unstructured text or non-text content (such as images, scanned documents, or JPEG files). The indexer accesses an Azure data storage service.



Transcriptions can be done in real-time or in batch mode.

Batch mode is when audio recordings are stored on a file share, and a shared access signature (SAS) URI is used by a program to asynchronously receive transcription results.

  1. Take the introductory tutorial:

    Introduction to Machine Learning with Hands-On Labs


  2. Create a model.

  3. Prepare Data:

    As per this video using

    • Clean Missing Data - Clip Outliers
    • Edit Metadata
    • Feature Selection
    • Filter
    • Learning with Counts
    • Normalize Data
    • Partition and Sample
    • Principal Component Analysis
    • Quantize Data
    • SQLite Transformation
    • Synthetic Minority Oversampling Technique

  4. Train the model

    • Cross Validation
    • Retraining
    • Parameter Sweep

  5. Score and test the model.

  6. Make predictions with Elastic APIs

    • Request-Response Service (RRS) Predictive Experiment - Batch Execution Service (BES)
    • Retraining API


Anomaly Detector


Among Anomaly Detector API Samples

Anomaly Detection

Anomaly Detector identifies potential problems early on.


  1. Sensors in the car collect telemetry, such as engine revolutions, brake temperature, and so on.
  2. An anomaly detection model is trained to understand expected fluctuations in the telemetry measurements over time.
  3. If a measurement occurs outside of the normal expected range, the model reports an anomaly that can be used to alert the race engineer to call the driver in for a pit stop to fix the issue before it forces retirement from the race.

Data Science VM



Microsoft AI ML Community in Signapore

If you have an OReilly.com account:

On Udemy:

Notes to be inserted

Steps for data transformation:

  • Feature selection
  • Finding and removing data outliers
  • Impute missing values
  • Normalize numeric features


Microsoft’s History with AI

In April 2018 Microsoft reorganized into two divisions that offers AI:

  • The research division, headed by Harry Shum, put AI into Bing search, Cortana voice recognition and text-to-speech, ambient computing, and robotics. See Harry’s presentation in 2016.

  • Microsft’s “computing fabric” offerings, led by Scott Guthrie, makes AI services available for those building customizable machine learning with speech, language, vision, and knowledge services. Tools offered include Cognitive Services and Bot Framework, deep-learning tools like Azure Machine Learning, Visual Studio Code Tools for AI, and Cognitive Toolkit.

At Build 2018, Microsoft announced Project Brainwave to run Google’s Tensorflow AI code and Facebook’s Caffe2, plus Microsoft’s own “Cognitive Toolkit” (CNTK).

  • BrainScript uses a dynamically typed C-like syntax to express neural networks in a way that looks like math formulas. Brainscript has a Performance Profiler.

  • Hyper-parameters are a separate module (alongside Network and reader) to perform SGD (stochastic-gradient descent).

Microsoft has advanced hardware:

    [ This pdf white paper says the "high-performance, precision-adaptable FPGA soft processor is at the heart of the system, achieving up to 39.5 TFLOPs of effective performance at Batch 1 on a state-of-the-art Intel Stratix 10 FPGA." Microsoft's use of field programmable gate arrays (FPGA) calculates AI reportedly "five times faster than Google's TPU hardware". "Each FPGA operates in-line between the server’s network interface card (NIC) and the top-of-rack (TOR) switch, enabling in-situ processing of network packets and point-to-point connectivity between hundreds of thousands of FPGAs at low latency (two microseconds per switch hop, one-way)."

Microsoft Conversational AI Platform for Developers is a 2021 book published by Apress by Stephan Bisser of Siili Solutions in Finland. The book covers Microsoft’s Bot Framework, LUIS, QnA Maker, and Azure Cognitive Services. https://github.com/orgs/BotBuilderCommunity/dashboard



In July and August Microsoft Reactor ran a “Skills Challenge” to reward a badge for those who complete an AI tutorial.


  • AI Builder
  • Machine Learning
  • MLOps
  • Cognitive

https://www.youtube.com/watch?v=ss-kyogPRNo by Carlotta


This is one of a series on AI, Machine Learning, Deep Learning, Robotics, and Analytics:

  1. AI Ecosystem
  2. Machine Learning
  3. Testing AI

  4. Microsoft’s AI
  5. Microsoft’s Azure Machine Learning Algorithms
  6. Microsoft’s Azure Machine Learning tutorial
  7. Microsoft’s Azure Machine Learning certification

  8. Python installation
  9. Juypter notebooks processing Python for humans

  10. Image Processing
  11. Tessaract OCR using OpenCV
  12. Amazon Lex text to speech

  13. Code Generation

  14. Multiple Regression calculation and visualization using Excel and Machine Learning
  15. Tableau Data Visualization