Microsoft AI (Azure OpenAI Generative Services)

How to automate AI workflows in Microsoft’s Azure and Fabric, despite marketing rebrands, and passing AI-900 & AI-102 certification exams.

Here is a hands-on deep dive introduction of Microsoft’s Artificial Intelligence (AI) offerings running on their Azure cloud.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Join Communities

AI is moving fast, so everything breaks. Like walking across a desert, we don’t get very far on our own in a strange land.

1. Accept invite at https://discord.com/invite/6GUBsZfMBq for the “Azure AI Foundry” on Discord (app).

2. Add to your calendar events: https://www.youtube.com/@theaishowlive/streams for the weekly “AI Show sponsored by Azure AI Foundry”. PROTIP: Videos actually start several minutes in.

   AMA Office hours for the following series at https://aka.ms/PythonAI/series by Pamela Fox.

3. Sign up for Microsoft for Startups Founders Hub to receive free OpenAI credits and up to $150k towards Azure credits to access OpenAI models through Azure OpenAI Services.

4. Review issues in the GitHub at

   https://github.com/microsoft/generative-ai-for-beginners/issues

5. If you find something wrong in the content, add an issue in its GitHub.

6. https://stackoverflow.com/questions/tagged/azure+python

7. https://lu.ma/oss4ai is not specific to Microsoft, but is based in the Pacific Northwest.

Acronyms & Glossary

• AOAI = Azure OpenAI Service
• OAI = OpenAI
• MAF = Multi-Agent Framework

Deep Learning vs. Machine Learning vs. AI services vs. LLMs vs. Generative AI

Semantic Kernel vs. Foundry

Microsoft offers several technologies to work with Large Language Models to build AI agents and orchestrate complex AI workflows within their applications. Launched in 2024.

• Azure AI Foundry runs only within Microsoft’s Azure cloud as a cloud-native service that manages orchestration, state, monitoring, security, compliance, and multi-agent workflows. Use to for quicker enterprise security compliance readiness, scalability, and operational efficiency. Foundry’s Model Catalog offers a choice of 11,000+ LLMs from various providers. It supports multiple deployment options including serverless and managed compute.

  Semantic Kernel is used by Azure AI Foundry itself as the internal orchestration engine for its multi-agent workflows.

• Semantic Kernel is an open-source SDK imported within conventional programming language code running on machines operated locally local models through Ollama or ONNX runtimes. It’s cloud-agnostic because it runs in-process within the application environment to give developers full control over orchestration logic, execution flow, debugging, and data management. Its program code explicitly manages state and memory.

See https://cloudsummit.eu/blog/navigating-microsoft-ai-development-ecosystem

LLM (Models)

Azure AI Foundry uses deployments to make models available.

Model Evaluation Metrics

Each metric offer a unique insight into the performance of LLMs:

• Groundedness: Measures alignment of the LLM application’s output with the input source or database.
• Relevance: Assesses how pertinent the LLM application’s output is to the given input.
• Coherence: Evaluates the logical flow and readability of the LLM application’s text.
• Fluency: Assesses the grammatical and linguistic accuracy of the LLM application’s output.
• Similarity: Quantifies the contextual and semantic match between the LLM application’s output and the ground truth.
  
  

These are for quality assurance to ensure that interactions with LLM applications are accurate and effective.

Charges occur on inference performed on model deployments.

ai.Azure.com AI Foundry

The Azure AI Foundry Project menu at https://ai.azure.com/foundryProject/overview?…

1. Tasks:
2. Prerequisites of account, region,
3. Demo of each Task using pre-defined data
4. Playgrounds
5. Custom data
6. Templates
7. Deployments
8. Actions (Custom Functions, Azure Functions, Azure Logic Apps)
9. Agents
10. Connected Agents
11. Threads from runs
12. Knowledge from Sharepoint, Azure AI Search, Fabric, Grounding with Bing, Tripadvisor, Morningingstar
13. Knowledge to RAG database
14. OpenAI Vector stores, Data files
15. Model Catalog
16. Models + endpoints
17. Fine-tuning
18. Billing (Tokens)
19. Custom Python programming
20. Monitoring
21. Tracing
22. +Trends, Alerts
23. Guradrails + controls
24. Governance

LEARN: https://microsoftlearning.github.io/mslearn-ai-information-extraction/ Develop AI-powered information extraction solutions in Azure git@github.com:MicrosoftLearning/mslearn-ai-information-extraction.git mslearn-ai-information-extraction

• Extract information from multimodal content - Use Azure AI Content Understanding to extract insights from documents, images, audio recordings, and videos. get structured data from multimodal content in document, image, audio, and video files. GUI: Content Understanding with no code https://ai.azure.com/explore/aiservices/vision/contentunderstanding

https://microsoftlearning.github.io/mslearn-ai-information-extraction/Instructions/Labs/01-content-understanding.html “What region supports Azure AI Content understanding” Australia East, Sweden Central, West US https://ai.azure.com/managementCenter/allResources “# Create” new AI hub resource > Azure AI Foundry resource => aif-westus-251019a https://ai.azure.com/doc/azure/ai-foundry/how-to/develop/sdk-overview?tid=64f782d9-f764-4be8-88d4-a5ac8719aef3 Terminology:

• Person directory = A structured way to store face data for recognition tasks. You can add individual faces to the directory and later search for visually similar faces. You can also create person profiles, associate faces to them, and match new face images to known individuals. This setup supports both flexible face matching and identity recognition across images and videos.
• Processing Location = An API request parameter that defines the geographic region where Azure AI Services analyzes your data. You can choose from three options: geography, dataZone, and global to control where processing occurs. This setting helps meet data residency requirements and optimize performance or scalability based on your needs.
• Span = A reference indicating the location of an element (for example, field, word) within the extracted Markdown content. A character offset and length represent a span. Different programming languages use various character encodings, which can affect the exact offset and length values for Unicode text. To avoid confusion, spans are only returned if the desired encoding is explicitly specified in the request. Some elements can map to multiple spans if they aren’t contiguous in the markdown (for example, page). Description
• File Any type of data, including text, documents, images, videos, and audio.
• File type The MIME type of a file, such as text/plain, application/pdf, image/jpeg, audio/wav, and video/mp4. Generic categories like document refer to all corresponding MIME types supported by the service.
• Generation method The process of determining the extracted value of a specified field. Content Understanding supports: • Extract: Directly extract values from the input content, such as dates from receipts or item details from invoices. • Classify: Classify content into predefined categories, such as call sentiment or chart type. • Generate: Generate values from input data, such as summarizing an audio conversation or generating scene descriptions from videos.
• Grounding source The specific regions in content where a value was generated. It has different representations depending on the file type: • Image - A polygon in the image, often an axis-aligned rectangle (bounding box). • PDF/TIFF - A polygon on a specific page, often a quadrilateral. • Audio - A start and end time range. • Video - A start and end time range with an optional polygon in each frame, often a bounding box.

Specific limitations of video processing to keep in mind: 1. Frame sampling (~ 1 FPS): The analyzer inspects about one frame per second. Rapid motions or single-frame events may be missed. 2. Frame resolution (512 × 512 px): Sampled frames are resized to 512 pixels square. Small text or distant objects can be lost. 3. Speech: Only spoken words are transcribed. Music, sound effects, and ambient noise are ignored.

/Users/johndoe/github-wilsonmar/mslearn-ai-information-extraction/Labfiles/content-app Define a schema based on a sample invoice: AI Foundry > Content Understanding > Custom task (at the top) > + Create = Task name: cu-task-8089 Invoice analysis, Description: Extract data from an invoice “Single file analysis”, Advanced settings: Azure AI services connection: The Azure AI Services resource in your Azure AI Foundry hub Azure Blob Storage account: The default storage account in your Azure AI Foundry hub Make connect at https://ai.azure.com/managementCenter/resource/project/connections Connections are a way to authenticate and consume both Microsoft and non-Microsoft resources within your projects. For example, connections can be used for prompt flow, training data, and deployments. See https://ai.azure.com/doc/azure/ai-foundry/how-to/connections-add In “Add a connection to external assets”, Select “storage account” PROTIP: Make sure to use the same region as the Foundry resource to avoid extra charges. In portal.azure.como Create a storage account.

1. Search for “Storage account” and select the Storage Account option that is published by Microsoft.
2. ”+ Create” your storage account. 4.2. Select Region: PROTIP: Create the storage account in the same region you created your project. To find this, go back to Azure AI Foundry, select the project name in the top right corner of the screen (which is the same option you selected earlier to find the link to your resource group), and review the Location of the resource group. Note: it will not default to the same region.

4.1. Resource group: 4.1. Storage account name: “blogwestus251020a” (no dashes allowed) 4.3. Primary service: Azure Blob Storage or Azure Data Lake Storage Gen 2. 4.4. Performance: Standard. 4.5. Redundancy: Geo-redundant storage (GRS). 4.6. Select Review+Create. 4.7. Select Create.

key: https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/evaluations-storage-account

• Develop a Content Understanding client application - Use Azure AI Content Understanding REST API to develop a client app for an analyzer.

• Analyze forms with prebuilt Azure AI Document Intelligence models - Use prebuilt Azure AI Document Intelligence models to process text fields from documents.

• Analyze forms with custom Azure AI Document Intelligence models - Create a custom Document Intelligence model to extract specific data from documents.

• Create a knowledge mining solution - Use Azure AI Search to extract key information from documents and make it easier to search and analyze.

Microsoft’s Confusing AI Branding

Names for Microsoft’s AI offerings have gone through many (confusing) re-brandings over time:

Cortana => Bing => Cognitive Services => OpenAI => Generative AI => AI Language => AI Studio (Prompt Flow) => Azure AI Foundry => Multi-Agent Framework (MAF)

Azure AI Foundry took over domain ai.azure.com also used for AI Studio before 2025. Azure AI Foundry introduces deeper integration, management tools, SDKs, and support for multi-agent enterprise AI, representing Microsoft’s consolidated AI innovation hub under Azure.

“Cortana” was the brand-name for Microsoft’s AI. Cortana is the name of the fictional artificially intelligent character in the Halo video game series. Cortana was going to be Microsoft’s answer to Alexa, Siri, Hey Google, and other AI-powered personal assistants which respond to voice commands controlling skills that turn lights on and off, etc. However, since 2019, Cortana is considered a “skill” (app) that Amazon’s Alexa and Google Assistant can call, working across multiple platforms.

For Search, the Bing “Bing” brand, before OpenAI was separated out from “Cognitive Services” to its own at https://docs.microsoft.com/en-us/azure/search, although it’s used in “Conversational AI” using an “agent” (Azure Bot Service) to participate in (natural) conversations. BTW: in 2019 Cortana decoupled from Windows 10 search.

Since October 31st, 2020, Bing Search APIs transitioned from Azure Cognitive Services Platform to Azure Marketplace. The Bing Search v7 API subscription covers several Bing Search services (Bing Image Search, Bing News Search, Bing Video Search, Bing Visual Search, and Bing Web Search),

Azure IoT (Edge) Services are separate.

The 2023 rebranding for Microsoft’s AI services to mimic human intelligence is “AI Language”, which includes Cognitive Services and Bing.

Microsoft has three service “Providers”:

PROTIP: On Jan 8, 2024, https://aka.ms/language-studio had “coming soon” for Video and Learn, and “preview” for several services. Essentially Microsoft has two separate offerings by different groups:

• https://learn.microsoft.com/en-us/azure/ai-services/
  
  

The “AI Services” era: Microsoft has been the “sole provider” of servers to OpenAI as part of some agreement that counts as investment.

In 2025, Microsoft was not part of the “$500 billion investment” announced in the Trump White House among OpenAI, Oracle, Softbank.

A few weeks later, Microsoft announced “Microsoft AI” (MAI).

Microsoft’s History with AI

In April 2018 Microsoft reorganized into two divisions to offers AI:

• The research division, headed by Harry Shum, put AI into Bing search, Cortana voice recognition and text-to-speech, ambient computing, and robotics. See Harry’s presentation in 2016.

• Microsft’s “computing fabric” offerings, led by Scott Guthrie, makes AI services available for those building customizable machine learning with speech, language, vision, and knowledge services. Tools offered include Cognitive Services and Bot Framework, deep-learning tools like Azure Machine Learning, Visual Studio Code Tools for AI, and Cognitive Toolkit.

At Build 2018, Microsoft announced Project Brainwave to run Google’s Tensorflow AI code and Facebook’s Caffe2, plus Microsoft’s own “Cognitive Toolkit” (CNTK).

• BrainScript uses a dynamically typed C-like syntax to express neural networks in a way that looks like math formulas. Brainscript has a Performance Profiler.

• Hyper-parameters are a separate module (alongside Network and reader) to perform SGD (stochastic-gradient descent).
  
  

Microsoft has advanced hardware:

[ This pdf white paper says the "high-performance, precision-adaptable FPGA soft processor is at the heart of the system, achieving up to 39.5 TFLOPs of effective performance at Batch 1 on a state-of-the-art Intel Stratix 10 FPGA." Microsoft's use of field programmable gate arrays (FPGA) calculates AI reportedly "five times faster than Google's TPU hardware". "Each FPGA operates in-line between the server’s network interface card (NIC) and the top-of-rack (TOR) switch, enabling in-situ processing of network packets and point-to-point connectivity between hundreds of thousands of FPGAs at low latency (two microseconds per switch hop, one-way)."

Microsoft Conversational AI Platform for Developers is a 2021 book published by Apress by Stephan Bisser of Siili Solutions in Finland. The book covers Microsoft’s Bot Framework, LUIS, QnA Maker, and Azure Cognitive Services. https://github.com/orgs/BotBuilderCommunity/dashboard

Microsoft Mechanics YouTube channel is focused on Microsoft’s AI work.

VIDEO: What runs ChatGPT? Inside Microsoft’s AI supercomputer

• Microsoft’s own Megatron-Turing model has 530 billion parameters
• [3:05] Onnx runtime for model portability
• [3:08] Deepspeed to componentize models has become the defacto framework for distributed training
• 285,000 AMD Infiniband CPU Cores & 10,000 NVIDIA V100 Tensor Core GPUs Infiniband connected to Quantum-2 Infiniband via NVLink
• Bisectional bandwidth to 3.6 TBps per server
• [10:45] Low Rank Adaptive (LoRA) Fine Tuning to update only portions of a model
  
  

NOTE: Wikipedia has 6,781,394 articles containing 4.5 billion words. English Wiktionary contains 1,439,188 definitions. Webster’s Third New International Dictionary has 470,000 English words.

https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/

Microsoft Reactor ran a “Skills Challenge” to reward a badge for those who complete an AI tutorial.

https://learn.microsoft.com/en-us/training/challenges

• AI Builder
• Machine Learning
• MLOps
• Cognitive
  
  

Computer Vision by Carlotta

Microsoft competes for talent with Google, Amazon, IBM, China’s Tencent, and many start-ups.

Azure AI certifications

Among Microsoft’s Azure professional certifications illustrated by this pdf, there are three levels of AI:

1. AI-900 $99 Fundamentals is the entry-level exam. It’s a pre-requisite for:

2. AI-102 $165 Associate exam focuses on the use of pre-packaged cloud-based services for AI development. It has free re-cert after 1-year.

   https://www.linkedin.com/in/alison-felix/ notice that in exercises some of the items are duplicated. For example:

   • The 1st one goes to …Exercises/ai-foundry/01-azure-search.html
   • The 2nd one goes to …Exercises/01-azure-search.html

   Get 50% off the AI-102 if you finish Coursera’s Microsoft AI & ML Engineering Professional Certificate by Mark DiMauro at Univ. Pittsbergh. It consists of 5 courses:

   1. Foundations of AI and Machine Learning - 35 hours
   2. AI and Machine Learning Algorithms and Techniques - 45 hours
   3. Building Intelligent Troubleshooting Agents - 45 hours
   4. Microsoft Azure for AI and Machine Learning - 21 hours
   5. Advanced AI and Machine Learning Techniques and Capstone - 32 hours
   • Coursera LAB: Getting started with Jupyter Notebooks in Azure Machine Learning Studio

   There is also a Coursera course on “Developing AI Applications with Azure”.

MS Reactor Tutorials

1. Microsoft Developer Community Blog at https://techcommunity.microsoft.com/category/azure/blog/azuredevcommunityblog covers ALL the various tech Microsoft has (365, Bicep, etc. in Spanish, English, etc.)

2. https://devpost.com/software/avoid-maga-word-bans/joins/mG6Kzv8uN6hYCjW0jCt9lA

3. https://devpost.com/submit-to/23928-azure-ai-developer-hackathon/manage/submissions/630940-avoid-word-bans/project_details/edit DevPost Hackthon project described as “Avoid word bans”.

4. https://www.linkedin.com/pulse/program-find-words-banned-wilson-mar-msc-2ccbc/?trackingId=Ux0g5xgWdyTMf7EAC4hFow%3D%3D

5. Pamela Fox (MS Evangelist) (pamelafox.org) holds Office hours on Discord

6. She held in 2025 the <a target=”_blank” href=”>

7. Pamela held in 2025 Microsoft Reactor livestream events in London, etc. on Intelligent Applications https://developer.microsoft.com/en-us/reactor/series/s-1491/

From aka.ms/PythonAI/series to https://github.com/pamelafox/python-openai-demos/discussions/21

• LLMs Recording Slides on GumRoad
• Vector embeddings Recording slides on GumRoad

• 18 March RAG Recording Slides on GumRoad https://github.com/microsoft/RAG_Hack?tab=readme-ov-file#stream-schedule

• 19 March, 2025

  7:00 AM (UTC-06:00) Dynamics 365 CE - 2025 Release Topic: Intelligent Applications

• 20 March, 2025

  4:30 PM (UTC-06:00) Python + AI: Vision models

• 25 March, 2025 | 4:30 PM (UTC-06:00) Python + AI: Function calling & structured outputs
• 27 March, 2025 | 4:30 PM (UTC-06:00) Python + AI: Quality & Safety

1. https://github.com/pamelafox/python-openai-demos
2. “Code” button to create GitHub Codespace.

3. AI Toolkit accesses FREE GitHub Models (these have limited tokens) at https://github.com/marketplace/models

4. https://gateway.on24.com/wcc/eh/4304051/category/142480/python-ai Python & AI resources

5. https://github.com/marketplace?type=models

6. https://github.com/Azure-Samples/llama-index-python This sample shows how to quickly get started with LlamaIndex.ai on Azure🚀

• https://github.com/Azure-Samples/raft-distillation-recipe A recipe that will walk you through using either Meta Llama 3.1 405B or GPT-4o deployed on Azure AI to generate a synthetic dataset using UC Berkeley’s Gorilla project RAFT method.

• https://github.com/Azure-Samples/contoso-chat This sample has the full End2End process of creating RAG application with Prompty and Azure AI Foundry. It includes GPT 3.5 Turbo LLM application code, evaluations, deployment automation with AZD CLI, GitHub actions for evaluation and deployment and intent mapping for multiple LLM task mapping.

History of AI exams:

• Previous exam 774 was retired. It was based on VIDEO: Azure Machine Learning Studio (classic) web services, which reflected “All Microsoft all the time” using proprietary “pickle” (pkl) model files. Classes referencing it are now obsolete.

• The MS LEARN site refers to files in https://github.com/MicrosoftLearning/mslearn-ai900

• AI-100 on June 30, 2021 with a shift from infrastructure (KeyVault, AKS, Stream Analytics) to programming C#, Python, and curl commands. (Free re-cert after 2-years).

• DP-090 and this LAB goes into implementing a Machine Learning Solution with Databricks, which has its own AI certification path.

• DP-100 covers development of custom models using Azure Machine Learning.

• DP-203 Data Engineering on Microsoft Azure goes into how to use machine learning within Azure Synapse Analytics. It was retired on March 31, 2025.

George Chen’s Journey to Microsoft Certified: AI Engineer Associate. Videos with PPT file!

AI-900

PROTIP: http://aka.ms/AIFunPath which expands to https://github.com/MicrosoftLearning/mslearn-ai-fundamentals

Exam definitions are at Microsoft’s LEARN</a> includes a free text-based tutorial called “Learning Paths” to learn skills:

• Describe AI workloads and considerations (15-20%)
• Describe fundamental principles of machine learning on Azure (30-35%)
• Describe features of computer vision workloads on Azure (15-20%)
• Describe features of Natural Language Processing (NLP) workloads on Azure (15-20%)

• Describe features of conversational AI workloads on Azure (15-20%)
  
  

• Ravi Kirans’ Study Guide contains links to MS Docs.

Tim Warner has created several video courses on AI-900 and AI-100:

• CloudSkills.io Microsoft Azure AI Fundamentals course references
  https://github.com/timothywarner/ai100cs

• On OReilly.com, his “Crash Course” Excel spreadsheet of exam objectives.

• OReilly.com references
  https://github.com/timothywarner/ai100

CloudAcademy’s 4h AI-900 video course includes lab time (1-2 hours at a time).

• AI-900 Study Guide - YouTube

• AI-900 Sample Practice Exam Questions

• https://www.udemy.com/course/microsoft-ai-900/

• https://www.itexams.com/info/AI-900

Emilio Melo on Linkedin Learning

Practice tests:

• https://www.whizlabs.com/learn/course/microsoft-azure-ai-900/
• https://www.examtopics.com/exams/microsoft/ai-900/

• https://www.whizlabs.com/learn/course/designing-and-implementing-an-azure-ai-solution/
  
  

• 2-hour AGuideToCloud video class by Susanth Sutheesh

AI-102

AI-102 is intended for software developers wanting to build AI-infused applications.

https://learn.microsoft.com/en-us/credentials/certifications/azure-ai-engineer

First setup development environments:

1. Read https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy
2. Get an organizational account (not gmail).
3. Provide that business address at aka.ms/oaiapply request access to Azure OpenAI service.

• Install Visual Studio Code and add-ons for C# and Python,
• Install NodeJs for Bot Framework Composer and Bot Framework Emulator
  
  

AI-102 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

• VIDEOM: Plan and manage an Azure AI Solution Cognitive Services (15-20%)
• VIDEO: Implement decision support solutions (10–15%)
• VIDEO: Implement computer vision solutions (15–20%)
• VIDEO: Implement natural language processing solutions (30–35%)
• VIDEO: Implement knowledge mining and document intelligence solutions (10–15%)
• VIDEO: Implement generative AI solutions (10–15%)
  
  

C# and Python programs at https://github.com/MicrosoftLearning/AI-102-AIEngineer (by Graeme Malcolm) was archived on Dec 23, 2023.

PROTIP: hands-on exercises at https://microsoftlearning.github.io/AI-102-AIEngineer/ in Microsoft’s 5-day live course AI-102T00: Designing and Implementing a Microsoft Azure AI Solution (with cloud time).

1. Lab Environment Setup -
2. Enable Resource Providers https://github.com/wilsonmar/azure-quickly/blob/main/az-utils.py register_subscription_providers()
3. Get Started with Azure AI Services https://github.com/MicrosoftLearning/AI-102-AIEngineer is archived/split among:
   • https://github.com/MicrosoftLearning/mslearn-ai-services
   • https://github.com/MicrosoftLearning/mslearn-ai-language
   • https://github.com/MicrosoftLearning/mslearn-ai-vision
   • https://github.com/MicrosoftLearning/mslearn-ai-document-intelligence
   • https://github.com/MicrosoftLearning/mslearn-knowledge-mining
   • https://github.com/MicrosoftLearning/mslearn-openai
4. Manage Azure AI Services Security
   • https://github.com/MicrosoftLearning/mslearn-ai-services
5. Monitor Azure AI Services

6. Use an Azure AI Services Container

   • https://github.com/MicrosoftLearning/mslearn-ai-language
7. Analyze Text
8. Translate Text
9. Recognize and Synthesize Speech
10. Translate Speech
11. Create a language understanding model with the Azure AI Language service
12. Create a Conversational Language Understanding Client Application

13. Create a Question Answering Solution

14. Create a Bot with the Bot Framework SDK - retired in favor of Agents Playground by end of December of 2025. https://github.com/Microsoft/BotFramework-Emulator/blob/master/README.md

15. Create a Bot with Bot Framework Composer https://learn.microsoft.com/en-us/previous-versions/composer/install-composer?tabs=macos downloads BotFramework-Composer-2.1.2-mac.zip to “Bot Framework Composer.app” to $HOME/Applications (Feb 3, 2022) alias=bfc

16. Analyze Images with Azure AI Vision
    • https://github.com/MicrosoftLearning/mslearn-ai-vision
17. Analyze Video with Video Analyzer
18. Classify Images with Azure AI Custom Vision
19. Detect Objects in Images with Custom Vision
20. Detect and Analyze Faces
21. Read Text in Images

22. Extract Data from Forms

    • https://github.com/MicrosoftLearning/mslearn-knowledge-mining
23. Create an Azure AI Search solution
24. Create a Custom Skill for Azure AI Search

25. Create a Knowledge Store with Azure AI Search

    • https://github.com/MicrosoftLearning/mslearn-openai

LEARN” “Develop generative AI apps in Azure” https://microsoftlearning.github.io/mslearn-ai-services/
https://github.com/MicrosoftLearning/mslearn-ai-services

• 01-use-azure-ai-services.md
• 02-ai-services-security.md
• 03-monitor-ai-services.md
• 04-use-a-container.md
• 05-personalizer-exercise.md
  
  

1. AI Vision Tutorials
2. AI Language Tutorials
3. AI Language
4. AI Decision Making
5. Other: OpenAI Tutorials

VIDEO Andrew Brown’s Azure AI Engineer Associate Certification (AI-102) – Full Course to PASS the Exam on FreeCodeCamp’s YouTube channel

https://learn.microsoft.com/en-us/training/courses/ai-102t00 modules:

• Prepare to develop AI solutions on Azure
• Create and consume Azure AI services
• Secure Azure AI services
• Monitor Azure AI services
• Deploy Azure AI services in containers
  
  

On Coursera: Coursera video course: Developing AI Applications on Azure by Ronald J. Daskevich at LearnQuest is structured for 5 weeks. Coursera’s videos shows the text at each point of its videos. NOTE: It still sends people to https://notebooks.azure.com and covers Microsoft’s TDSP (Team Data Science Process) VIDEO:

1. Business understanding (Charter, Objectives, Data sources, Data dictionaries)
2. Data acquisition and understanding (Clean dataset, pipeline)
3. Modeling
4. Deployment model for use (testing) Flow: entry script to accept requests and score them, Deployment configs
5. Customer Acceptance
   
   

Resources:

• https://www.whizlabs.com/microsoft-azure-certification-ai-102/ $49.90
• Preview 45 min. Exam: Designing and Implementing an Azure AI Solution (AI-102)
• https://medium.com/version-1/mastering-azure-ai-certifications-passing-the-ai-100-and-ai-102-exams-1st-time-c8a8fe223fcc
  
  

AI-100 (RETIRED)

On June 30, 2021 Microsft retired the AI-100 exam in favor of AI-102 exam (avilable in $99 beta since Feb 2021). AI-100 exam, as defined at Microsoft’s LEARN has free written tutorials on each of the exam’s domains:

• Analyze solution requirements (25-30%)
• Design AI solutions (40-45%)
• Implement and monitor AI solutions (25-30%)
  
  

https://github.com/MicrosoftLearning/AI-100-Design-Implement-Azure-AISol

https://github.com/MicrosoftLearning/Principles-of-Machine-Learning-Python

• Plan and manage an Azure Cognitive Services solution (15-20%)
  • MS: Evaluate text with Azure Cognitive Language Services
• Implement Computer Vision solutions (20-25%)
  • MS: Process and classify images with the Azure cognitive vision services
• Implement natural language processing solutions (20-25%)

  • MS: Process and Translate Speech with Azure Cognitive Speech Services

  • MS: Process natural language with Azure Cognitive Language Services

• Implement knowledge mining solutions (15-20%)
  • MS: Implement knowledge mining with Azure Cognitive Search
• Implement conversational AI solutions (15-20%) - chatbots
  • MS: Create conversational AI solutions

Resources:

• Raza Salehi (@zaalion) created on OReilly.com an AI-100 exam prep “crash course” which references his <a target=”_blank” href=”https://github.com/zaalion/oreilly-ai-100 and https://github.com/zaalion/uy-cognitve-services-crash-course

Guy Hummel’s CloudAcademy.com 7hr AI-100 video course.

McK Udemy by Anand Rao Nednur

Raza Salehi created on Pluralsight.com a series for Microsoft Azure AI Engineer (AI-100)

Practice tests:

• https://www.whizlabs.com/learn/course/microsoft-azure-ai-100/
• Ravi’s links still refer to AI-100

Prerequisites to ai.azure.com (AI Foundry)

Many give up entirely because they did not get their working environment properly setup before diving into the “For Beginners” courses.

This is covered in LEARN modules certification AZ-104:

Using Command Line:

1. Install fancy Terminal program Warp.

2. Install package manager Homebrew (if you’re on macOS).

3. Install Git.
4. Install Visual Studio Code and fancy IDEs Windsurf, Cursor, Claude, etc.
5. Install Python interpreter.
6. Install Conda.
7. Install JupyterLab.
8. Install Azure CLI.
9. Install Microsoft Edge browser.

10. Install Google’s Chrome browser.

    On Microsoft's Edge browser:

11. Create an email (on protonmail.com or gmail.com)
12. Save the password in a password manager.
13. Credit card. If you have a Capital One card, generate a unique card number.
14. Put the expiration date as an event in your calendar.

15. Phone. If you can, generate a unique phone “burner” number.

16. Get onboarded to a Microsoft Azure subscriptions and learn Portal GUI menu keyboard shortcuts on the Azure portal at

    https://portal.azure.com/?quickstart=True#view/Microsoft_Azure_Resources/QuickstartCenterBlade

    • Effortless create SQL database using new vCore (not old DTU) purchasing model always-on

    https://portal.azure.com (Azure Cloud Shell)

17. Setup a CLI scripting environment in https://shell.azure.com like I describe in my mac-setup page

    WARNING: A storage account is required for the CLI to work AND it costs money per month.

18. Open your Command Terminal. On macOS, use Terminal. On Windows, use cmd, Bash (Windows Terminal) or PowerShell.

19. Use CLI to Create a Cognitive Service to get keys to call the first REST API from among sample calls to many REST APIs: the Translator Text API.

20. On Windows 11, install Edge browser according to

    https://microsoftlearning.github.io/mslearn-ai-services/Instructions/setup.html

    The setup is for this LAB pop-up:

    https://learn.microsoft.com/en-us/training/modules/create-manage-ai-services/5a-exercise-ai-services

21. Setup PowerShell scripts

22. Create Azure master account using email, credit card, and phone.
23. Create Subscription with credit card.
24. Azure AI Foundry Account
25. Create Azure Project (with Hub)

26. View Hub in the Azure AI Foundry Management center

27. Define IAM Roles and accounts for Azure AI Foundary
28. Define Quota

29. View Azure Cost analysis

30. Create a GitHub account using your email address.

31. Create a GitHub Personal Access Token to use Github Models Marketplace free access to LLMs used to create AI Agents.

32. Open Command Prompt or Terminal.
33. Create a folder for your GitHub account.

34. Navigate to the folder for your GitHub account.

35. Go to your Personal Access Tokens settings in your GitHub Account.
36. Select the Fine-grained tokens option on the left side of your screen.
37. Select Generate new token.
38. Copy your new token that you have just created.
39. Pick a region.

40. Add token & region to the .env file for this course.

41. In the https://portal.azure.com

42. In the https://ai.azure.com/ “Azure AI Foundry Portal”

    The Azure AI Foundry SDK provides the AI Projects client library, which you can use to access connections in a project or hub.

43. In the https://ml.azure.com/ “Azure Machine Learning Portal”

PROTIP: Within “Develop generative AI solutions in Azure” exercises explained:

• Excercise: Prepare for an AI development project - how to create and manage hubs and projects, add connected resources, and explore Azure AI Services and Azure OpenAI models in the Azure AI Foundry portal. Learn how to organize cloud resources in Azure AI Foundry projects so that developers are set up for success when building AI solutions.

• Choose and deploy a language model - Generative AI applications are built on one or more language models. Learn how to find and select appropriate models for your generative AI project.

• Create a generative AI chat app - Learn how to use the Azure AI Foundry SDK to build an app that connects to your project and chats with a language model.

• Use a prompt flow to manage conversation in a chat app - Learn how to use prompt flows to manage conversational dialogs and ensure that prompts are constructed and orchestrated for best results.

• Create a generative AI app that uses your own data - Learn how to use the Retrieval Augmented Generation (RAG) model to build a chat app that grounds prompts using your own data.

• Fine-tune a language model - Learn how to use your own training data to fine-tune a model and customize its behavior.

• Apply content filters to prevent the output of harmful content - Learn how to apply content filters that mitigate potentially offensive or harmful output in your generative AI app.

• Evaluate generative AI model performance - Learn how to evaluate models and prompts to optimize the performance of your chat app and its ability to respond appropriately.

Beginners Series from Microsoft Cloud Advocates

DO THIS: Click “Star” and “Watch” for “All Activity” in each repo below:

• github.com/microsoft/generative-ai-for-beginners (aka.ms/genai-beginners) “Learn the fundamentals of building Generative AI applications with our 21-lesson comprehensive course by Microsoft Cloud Advocates. 21 Lessons teaching everything you need to know to start building Generative AI applications”. (Version 3) Maintained by @bethanyjep. VIDEO series

  • Generative AI Code Samples - a collection of code samples as extra learning and materials from the “Generative AI for Beginners” course.

  • github.com/Azure-Samples/rag-data-openai-python-promptflow - A copilot sample that uses python to ground the copilot responses in company data.

  • deprecated: Quickstart sample for using the Azure AI Studio with the SDK or CLI options - and the LangChain framework.

  • github.com/Azure-Samples/rag-data-openai-python-promptflow “ChatGPT + Enterprise data with Azure OpenAI and AI Search - Python”

  • AI-in-a-Box - leverages the expertise of Microsoft across the globe to develop and provide AI and ML solutions to the technical community. Our intent is to present a curated collection of solution accelerators that can help engineers establish their AI/ML environments and solutions rapidly and with minimal friction.

  • Lesson 15: End to End LLM App development with Azure AI Studio and Prompt Flow - Python
  • Lesson 16: RAG chat: Local development of chat app

• github.com/microsoft/AI-For-Beginners (aka.ms/ai-beginners) says it’s a “12-week, 24-lesson curriculum! It includes practical lessons, quizzes, and labs. The curriculum is beginner-friendly and covers tools like TensorFlow and PyTorch, as well as ethics in AI.” It’s an update of the:

• github.com/microsoft/AI-For-Beginners (aka.ms/ml-beginners) “12 weeks, 26 lessons, 52 quizzes, classic Machine Learning for all.” using primarily Scikit-learn.

• github.com/microsoft/ai-agents-for-beginners (aka.ms/ai-agents-beginners) “10 Lessons to Get Started Building AI Agents”

How to Make Money from this?

End-user Productivity of Human-Computer Interaction (HCI)

Microsoft “democratizes” Machine Learning and AI by providing a front-end GUI that hides some of the complexities, enabling them to be run possibly without programming.

Alas, Microsoft's Azure AI gallery of samples and users' contributions is no longer at https://gallery.azure.ai/browse

1. DEMO: Hands-on with AI/Guidelines for Human-AI Interaction: Click each card to see examples of each guideline

   https://aka.ms/hci-demo which redirects you to
   https://aidemos.microsoft.com/guidelines-for-human-ai-interaction/demo

   • Initially - make clear what the system can do & how well the system can do what it can do.
   • During interaction - Time services based on context; show contexually revelvant info; Match revelvant social norms; Migrate social biases.
   • When Wrong - support efficient invocation, dismissal, correction; Scope services when in doubt.
   • Over Time - remember recent interactions; learn from user behavior; update and adapt cautiously; encourage granular feedback
     
     

   PROTIP: Some fonts are real small. Zoom in to read it.

   Hybrid workflow Pipelines

   PROTIP: Although most of Microsoft’s product documents focus on one technology at a time, actual production work enjoyed by real end-users usually involves a pipeline consisting of several services. For example: ingesting (stream processing) a newsfeed NLP (Natural Langague Processing):

   

   That and other flows are in the Azure Architecture Center.

2. For more about ALGORITHMS USED, see my explanations at https://bomonike.github.io/machine-learning-algorithms, which lists them by alphabetical order and grouped by.

Case studies of how people are already making use of AI/ML to save time and money:

• Calculate a positive/negative sentiment score for each post in your social media account and send you an alert based on a trigger (such as negative sentiment less than 5).
• Use text-to-speech to generate a voice message sent to international phones as voice message (via Twillo)

• Create a recommendation engine (such as what Netflix) from the Internet Movie Database (imdb.com)

• Predictive Maintenance data science webinar
• modsy.com 3D view
• Customer and Partner Success Stories for “bot”
  
  

Microsoft’s AI products portfolio

Click picture for full-page view.

In the diagram above, Microsoft makes a distinction between “Business Users & Citizen Developers” who use their Applications and “Power Platform” and geeky “Developers & Data Scientists” who use “Azure AI” in the Azure cloud.

In the diagram above, Microsoft categorize Azure’s AI services these groups (all of which have GUI, CLI, and API interfaces):

• Applied AI Services are part of automated workflows, but a service can be included among
• Cognitive Services of foundational utilities used to build custom apps
• ML (Machine Learning) Platform, which I cover in a separate article
  
  

PROTIP: Several services are NOT shown in the diagram above. The list in a Microsoft LEARN module show a different order:

DEFINITION: There are what are called “narrow” or “weak” AI.

Azure Cloud Services

1. Click (Azure cloud) SERVICES USED:

   • Azure Machine Learning
   • PowerBI
   • Azure Blob Storage
   • Azure HDInsight
   • Azure Data Factory
   • Azure SQL DataWarehouse
   • Azure App Service
   • Azure Sql
   • Azure Event Hubs
   • Azure Stream Analytics
   • Azure Cognitive Service - LUIS
   • Azure Data Lake Analytics
   • Azure Data Lake Store
   • Azure Virtual Machine
   • Azure Batch
     
     

Availability of AI Products

Every product Microsoft offers is listed, by region in this table showing availability (regardless of whether you have permissions to each specific product).

https://learn.microsoft.com/en-us/azure/ai-foundry/reference/region-support Services of the Azure AI Foundary

• Azure OpenAI - Note that some models might not be available within the Azure AI Foundry model catalog. Azure OpenAI quotas and limits
• Speech capabilities - Azure AI Speech capabilities including custom neural voice vary in regional availability due to underlying hardware availability. Speech service supported regions
• Serverless API deployments - Some models in the model catalog can be deployed as a serverless API with pay-as-you-go billing. Region availability for models in Serverless API endpoints
• Azure AI Content Safety - To use the Content Safety APIs, you must create your Azure AI Content Safety resource in a supported region.
• Azure AI Agent Service - Azure AI Agent Service supports the same models as the chat completions API in Azure OpenAI. Azure AI Agent Service region availability

in Product Availability by Region

• AI builder -
• Azure AI Search - AI Enrichment, Semantic Ranker
• Azure AI Services -
  • Azure AI Language
  • Azure AI Language Understanding
  • Azure AI Vision Face API
  • Azure AI Vision
  • Health Insights
  • Azure AI Speech
  • Azure AI Anomaly Detector
  • Azure AI Content Safety
  • Azure AI Document Intelligence
  • Azure AI Immersive Reader
  • Azure AI Metrics Advisor
  • Personalizer
  • Immersive Reader
  • Bing Speech
  • Azure AI Custom Vision
  • Azure AI Translator
• Azure Applied AI Services
  • Azure AI Document Intelligence
  • Azure Video Indexer
  • Azure AI Immersive Reader
  • Azure AI Metrics Advisor
• Azure Health AI Deidentification service
• Azure OpenAI Service

https://learn.microsoft.com/en-us/azure/ai-studio/how-to/costs-plan-manage

Azure AI Foundry

Azure AI Foundry provides tools and functionality within a project that developers can use to build AI solutions efficiently, including:

• Visual Studio Code containers that define a hosted development environment in which you can write, test, and deploy code.

• Management with hubs of project assets, including models and endpoints, data and indexes, and deployed web apps. https://learn.microsoft.com/en-us/training/modules/prepare-azure-ai-development/4-azure-ai-foundry

• Playgrounds in which you can test prompts with generative AI models.

• A model catalog in which you can find and deploy machine learning models from multiple sources, including Azure OpenAI and the Hugging Face model library.

• Access to Azure AI services, including visual interfaces to experiment with and configure services as well as endpoints and keys that you can use to connect to them from client applications.

• Fine-tuning functionality for generative AI models that you need to customize based on custom training prompts and responses.

• Prompt Flow, a prompt orchestration tool that you can use to define the logic for a generative AI application’s interaction with a model.

• Tools to assess, evaluate, and improve AI applications, including tracing, evaluations, and content safety and security management.

Cognitive Services List

The method shown in this section gives you a list of all “kinds” of services Azure currently provides for your Azure subscription.

After your Command Terminal environment on your laptop has the Installed prerequisites of obtaining an Azure account with subscription and installed the “az” CLI modules needed to run “az” commands on your Terminal, run this command:

az cognitiveservices account list-kinds

  Response:

[
  "AIServices",
  "AnomalyDetector",
  "CognitiveServices",
  "ComputerVision",
  "ContentModerator",
  "ContentSafety",
  "ConversationalLanguageUnderstanding",
  "CustomVision.Prediction",
  "CustomVision.Training",
  "Face",
  "FormRecognizer",
  "HealthInsights",
  "ImmersiveReader",
  "Internal.AllInOne",
  "LUIS.Authoring",
  "LanguageAuthoring",
  "MetricsAdvisor",
  "Personalizer",
  "QnAMaker.v2",
  "SpeechServices",
  "TextAnalytics",
  "TextTranslation"
]
   

PROTIP: “ContentModerator” has been deprecated.

Cognitive Services Grouping

PROTIP: The services above are listed in random order. But the table below groups services to help you quickly get to links about features, tutorials, and SDK/API references quickly.

As of this writing, in various marketing and certification training DOCS, Azure Cognitive Services are grouped into these (which is the basis this article is arranged. Click on the underlined and bolded category name to jump to the list of services associated with it, in this order (like on the AI Products Portfolio diagram:

1. AI Vision (Visual Perception) provides the ability to use computer vision capabilities to accept, interpret, and process input from images, video streams, and live cameras. Interpret the world visually through cameras, videos, images

2. AI Speech - Text-to-Speech and Speech-to-Text to interpret written or spoken language, and respond in kind. This provides the ability to recognize speech as input and synthesize spoken output. The combination of speech capabilities together with the ability to apply NLP analysis of text enables a form of human-compute interaction that’s become known as conversational AI, in which users can interact with AI agents (usually referred to as bots) in much the same way they would with another human.

3. AI Language - aka Natural language Processing (NLP) to translate text (Text Analysis), etc.

4. AI Decision (Making) provides the ability to use past experience and learned correlations to assess situations and take appropriate actions. For example, recognizing anomalies in sensor readings and taking automated action to prevent failure or system damage. supervised and unsupervised machine learning

5. Other: OpenAI (to power your apps with large-scale AI models) is a recent add to this confusing category because of so many branding changes (Cortana, Bing, Cognitive, OpenAI, etc.)

Text analysis and conversion provides the ability to use natural language processing (NLP) to not only “read”, but also generate realistic responses and extract semantic meaning from text.

Pricing

PROTIP: The table below groups each kind of cognitive AI service along with how many FREE transactions Microsoft provides for each service on its Cognitive Services pricing page.

IMPORTANT PROTIP: Microsoft allows its free “F0” tier to be applied to only a single Cognitive Service at a time. To remain free, you would need to rebuild a new Cognitive Service with a different “Kind” between steps.

Pricing of AI Services Kinds

Within each grouping, each service is listed in the sequence within that group’s LEARN module.

Demo	Kind	FREE	API
global	"CognitiveServices"	See below	-
?	"AIServices"	?
?	"Internal.AllInOne"	?
AI Vision
MSR	"ComputerVision" => AI Vision	2 TPC, 10K/mo.	API
MSR	"CustomVision.Prediction"	10K predictions/mo.	API
Features	"CustomVision.Training"	2 TPS, 1 hour/mo.	API
MSR	"Face" => AI Vision	20 TPM, 30K/mo.	API
Features	Video Indexer (Applied AI)	600 minutes in portal, 2400 in API	API
-	Ink Recognizer" (DELISTED)	-	-
AI Speech
MSR	"SpeechServices" => AI Speech (Speech-To-Text", Text-to-Speech")	1 concurrent, 5 hours/mo.*	API
Features	"SpeakerRecognition" (DELISTED?)	20/min, 5K/mo.	API
Features	Speech Translation (DELISTED?)	20/min, 5K/mo.	API
MSR	AI Language
MSR	"TextTranslation" => AI Translator	2M chars/mo.	API
Features	"LUIS" (Language Understanding, UNLISTED)	5 TPS, 10K trans/mo.	API
Features	"LUIS.Authoring"	5 TPS	API
Features	"QnAMaker" (UNLISTED)	3 TPS, 50K trans/mo, 3 docs/mo.	API
Features	"QnAMaker.v2"	3 TPS, 5K/mo.	API
Features	"TextAnalytics" (Detect Language, Sentiment, Named Entities, Key Phrases)	5K trans/mo.	API
Features	"ImmersiveReader" (Applied AI)	3M chars/mo.	API
?	"LanguageAuthoring"	?
AI Decision (Making)
Features	"AnomalyDetector"	2,000 trans/mo.	API
Features	"ContentModerator" (deprecated)	1 TPS	API
Features	"ContentSafety"	?
Features	"MetricsAdvisor" (Applied AI)	preview	API
Features	"Personalizer"	50K trans/mo.	API
Search
?	KnowledgeMining (UNLISTED)	?
?	Microsoft Bing (Applications)	?
Features	"Bing.CustomSearch" (UNLISTED)	50K trans/mo.	-
Features	"Bing.Search.v7" (UNLISTED)	3 TPS, 1000/mo	API
?	AI Search	Cognitive Search (Applied AI)	?
MSR	Document Intelligence = FormRecognizer (Applied AI)	500 pages/mo.	API
Other
?	"HealthInsights"	?	?
$	Bot Service (Applied AI)	?	?
Features	OpenAI Service	?	SDK

Links are provided for each service to its Features and API/SDK pages.

MSR CognitiveAllInOne GUI Responsible AI

“MSR” in the table above identifies a Multi-Service Resource accessed using a single key and endpoint to consolidate billing.

PROTIP: HANDS-ON: Create the CognitiveAllInOne resource using the GUI so that you can acknowledge Microsoft’s terms for Responsible AI use:

PROTIP: Apply of Terraform to create AI service resources will error out unless that is checked.

AI Capabilities

https://learn.microsoft.com/en-us/training/modules/prepare-azure-ai-development/

most medium to large-scale development scenarios it’s better to provision Azure AI services resources as part of an Azure Foundry hub - enabling you to centralize access control and cost management, and making it easier to manage shared resource usage based on AI development projects. Hubs provide a top-level container for managing shared resources, data, connections and security configuration for AI application development. A hub can support multiple projects, in which developers collaborate on building a specific solution. A hub provides a centrally managed collection of shared resources and management configuration for AI solution development. You need at least one hub to use all of the solution development features and capabilities of AI Foundry.

https://learn.microsoft.com/en-us/training/modules/prepare-azure-ai-development/4-azure-ai-foundry

Learning Sequence

This document covers:

1. Automatically shut down Resource Groups of a Subscription by creating a Logic App.

2. Run an API connecting to an established endpoint (SaaS) you don’t need to setup: Bing Search.

3. Create Functions

4. Create a Workspace resource to run …

5. Create a Workspace resource and
   Create Compute instance to run
   Automated ML of regression of bike-rentals.
6. Create Compute instance to run a iPython notebook
7. Create ML Workspace in Portal, then ml.azure.com

8. Us cognitivevision.com to Create Custom Vision for …

9. QnA Maker Conversational AI
10. Train a Machine Learning model using iPython notebook
11. IoT - “Hey Google, ask Azure to shut down all my compute instances”.

Automation necessary for PaaS

IMPORTANT PROTIP: As of this writing, Microsoft Azure does NOT have a full SaaS offering for every AI/ML service. You are required to create your own computer instances, and thus manage machine sizes (which is a hassle). Resources you create continue to cost money until you shut them down.

So after learning to set up the first compute service, we need to cover automation to shut them all down while you sleep.

So that you’re not tediously recreating everything everyday, this tutorial focuses on automation scripts (CLI Bash and PowerShell scripts) to create compute instances, publish results, then shut itself down. Each report run overwrites files from the previous run so you’re not constantly piling up storage costs.

Another reason for being able to rebuild is that you if you find that the pricing tier chosen is no longer suitable for your solution, you must create a new Azure Cognitive Search resource and recreate all indexes and objects.

When you use my Automation scripts at https://github.com/wilsonmar/azure-quickly/ to create resources the way you like, using “Infrastructure as Code”, so you can throw away any Subscription and begin anew quickly.

My scripts also makes use of a more secure way to store secrets than inserting them in code that can be checked back into GitHub.

Effective deletion hygiene is also good to see how your instances behave when it takes advantage of cheaper spot instances which can disappear at any time. This can also be used for “chaos engineering” efforts.

To verify resource status and to discuss with others, you still need skill at clicking through the Portal.azure.com, ML.azure.com, etc.

References:

• VIDEO: To release IP address, don’t stop machines, but delete the resource.

• VIDEO: shut down automatically all your existing VMs (using a PowerShell script called by a scheduled Logic App), by Frank Boucher at github.com/FBoucher

• VIDEO

• Auto-shutdown by Resource Manager on a schedule is only for VMs in DevOps

• https://www.c-sharpcorner.com/article/deploy-a-google-action-on-azure/

• start/stop by an Automation Account Runbook for specific tags attached to different Resource Groups: Assert: “AutoshutdownSchedule: Tuesday” run every hour. Google Translate

https://github.com/Azure/azure-sdk-for-python

Challenges

Get 50% off by completing just one of these challenges:

• Azure AI Fundamentals = AI-900
• Generative AI with Azure OpenAI
• Azure AI Document Intelligence
• Azure AI Language
• Azure AI Vision
• Azure Optimization = AZ-500

Document Intelligence and Knowledge Mining

“Document Intelligence” is a rebrand of “Cognitive Services Form Recognizer”.

1. View the marketing page at:

   https://documentintelligence.ai.azure.com/studio

   Azure AI Document Intelligence model overview

2. Prebuilt models expect a common type of form or document:

   • Business cards
   • Receipts

   • US Bank Statements

   • Invoices
   • W-2 US tax declaration
   • ID Documents
   • Health insurance cards

   NOT:

   • Resumes
3. CODING:

   poller = document_analysis_client.begin_analyze_document(
         "prebuilt-layout", AnalyzeDocumentRequest(url_source=docUrl
      ))
   result: AnalyzeResult = poller.result()
   

4. LAB: Exercise - Analyze a document using Azure AI Document Intelligence

5. You are using the prebuilt layout model to analyze a document with many checkboxes. You want to find out whether each box is checked or empty. What object should you use in the returned JSON code? Selection marks record checkboxes and radio buttons and include whether they’re selected or not. NOT Bounding boxes. NOT Confidence indicators.

   PROTIP: Although Word documents are also from Microsoft, DOCX format files are NOT supported by Azure AI Document Intelligence. But PDF documents are supported. Azure AI Document Intelligence is designed to analyze scanned and photographed paper documents, not documents that are already in a digital format so you should consider using another technology to extract the data in Word documents.

6. Get into the Azure Marketplace to look for Azure AI Document Intelligence – an Azure service that you can use to analyze forms completed by your customers, partners, employers, or others and extract the data that they contain.

7. https://learn.microsoft.com/en-us/training/modules/plan-form-recognizer-solution/2-understand

• https://microsoftlearning.github.io/mslearn-ai-document-intelligence/
  • https://github.com/MicrosoftLearning/mslearn-ai-document-intelligence
  • Use prebuilt Document Intelligence models
  • Extract Data from Forms
  • Create a composed Document Intelligence model
    
    

References:

• What is Azure AI Document Intelligence?
• Guidance for integration and responsible use with Azure AI Document Intelligence
• Azure AI Document Intelligence models
• Azure AI Document Intelligence Studio (preview)
• Create an Azure AI Document Intelligence resource
• Get started: Azure AI Document Intelligence C# SDK (beta)
• Azure AI Document Intelligence custom models

• Composed custom models

• Language support for Azure AI Document Intelligence
• Azure AI Document Intelligence read model
• Azure AI Document Intelligence general document model
• Azure AI Document Intelligence layout model

AI Landing Zones (ALZ)

In production usage, several Azure subscriptions are needed.

The above image replaced the previous version) is the Azure Landing Zones OpenAI Reference Architecture defining how to envelope OpenAI with utilities to ensure a defensive security posture. It maps how resources are integrated in a structured, consistent manner, plus ensuring governance, compliance, and security.

The diagram above is an adaptation of Microsoft’s enterprise-scale Azure Landing Zone, a part of Microsoft’s Cloud Adoption Framework (CAF).

Each Design Area has:

A. Enterprise enrollment [TF]
B. Identity and accessment [TF]
C. Management group and subscription organization
D. Management subscription
E. Connectivity subscription
F. AI Services (Landing Zone) subscription
G. Monitoring?
H. Sandbox subscription
I. Platform DevOps Team [TF]</br />

Design Areas F (the Landing Zone for AI):

• Managed Identities
• UDR (User-Defined Routes) for each type of outbound connect from the VNet, defined by Azure service tags or IP addresses.
• Azure WebApp provides a GUI front-end
• DatabaseSubnet houses a CosmosDB to persist data to the WebApp
• API Management Services tracks usage by the API key assigned each user
• Application Gateway
• The Azure FrontDoor for DDOS attacks
• ServicesSubnet houses an Azure KeyVault on a Private Endpoint
• AISubnet houses Private Endpoints for:
  • Storage Account
  • Azure OpenAI Services
  • Azure AI Services
    
    

To create the resources in the diagram:
If you prefer using Bicep:

1. https://github.com/Azure/ALZ-Bicep/wiki/DeploymentFlow
2. https://github.com/Azure/ALZ-Bicep/wiki/ConsumerGuide
3. https://learn.microsoft.com/en-us/azure/architecture/landing-zones/bicep/landing-zone-bicep
4. https://github.com/Azure/ALZ-Bicep/wiki/Accelerator
   
   

If you prefer using Terraform:

• https://registry.terraform.io/modules/Azure/caf-enterprise-scale/azurerm/latest
• https://github.com/Azure/terraform-azurerm-caf-enterprise-scale/wiki/Examples
• https://github.com/Azure/terraform-azurerm-caf-enterprise-scale?tab=readme-ov-file#readme
  
  

Included are Private Endpoints, Network Security Groups and Web Application Firewalls.

Sample ML Code

PROTIP: AI-102 is heavy on questions about coding.

Samples (unlike examples) are a more complete, best-practices solution for each of the snippets.

PROTIP: github.com/Azure-Samples from Microsoft offers samples code to use Cognitive Services REST API by each language:

• .NET C#
• Java
• NodeJs
• Python
• Go
  
  

https://docs.microsoft.com/en-us/samples/azure-samples/azure-sdk-for-go-samples/azure-sdk-for-go-samples/

A complete sample app is Microsoft’ Northwinds Traders consumer ecommerce store: install

IMPORTANT: Cognitive Services SDK Samples for:

• (Node)
• Python at https://github.com/Azure-Samples/cognitive-services-python-sdk-samples.git
• Java

Tim Warner’s https://github.com/timothywarner/ai100 includes Powershell scripts:

• keyvault-soft-delete-purge.ps1
• keyvault-storage-account.ps1
• python-keyvault.py
• ssh-to-aks.md - SSH into AKS cluster nodes
• xiot-edge-windows.ps1
• autoprice.py
  
  

Among Azure Machine Learning examples is a CLI at https://github.com/Azure/azureml-examples/tree/main/cli

PROTIP: CAUTION: Each service has a different maturity level in its documentation at azure.microsoft.com/en-us/downloads, such as SDK for Python open-sourced at github.com/azure/azure-sdk-for-python, described at docs.microsoft.com/en-us/azure/developer/python.

AI Services

The hands-on steps below enables you to operate offline on a macOS laptop to run the Azure AI SDK in a Docker container using VS Code Dev Containers.

https://learn.microsoft.com/en-us/azure/ai-services/

runs from VSCode to run Python 3.10 with Docker containers within Azure AI Studio at https://ai.azure.com

Following How to start:

Work with Azure AI projects in VS Code

1. Install Docker Desktop, https://code.visualstudio.com/docs/devcontainers/tutorial

2. Install Visual Studio Code (VSCode). Within VSCode, install Dev Containers extension to use VS Code Dev Container

3. run the command

   Dev Containers: Try a Dev Container Sample…

4. Install Git and configure it.

5. Create a folder “ai_env” from GitHub for the “Sample quickstart repo for getting started building an enterprise chat copilot in Azure AI Studio”:

   FOLDER="ai_env"
   git clone https://github.com/azure/aistudio-copilot-sample "${FOLDER}" --depth 1
   cd "${FOLDER}"
   

   Response:

   Cloning into 'ai_env'...
   remote: Enumerating objects: 122, done.
   remote: Counting objects: 100% (122/122), done.
   remote: Compressing objects: 100% (107/107), done.
   remote: Total 122 (delta 11), reused 88 (delta 8), pack-reused 0
   Receiving objects: 100% (122/122), 227.58 KiB | 951.00 KiB/s, done.
   Resolving deltas: 100% (11/11), done.
   

6. Define provenance:

   git remote add upstream https://github.com/azure/aistudio-copilot-sample
   git remote -v
   

7. Open with VSCode to an error:

   code ai_env

8. Select the “Reopen in Dev Containers” button. If it doesn’t appear, open the command palette (Ctrl+Shift+P on Windows and Linux, Cmd+Shift+P on Mac) and run the Dev Containers: Reopen in Container command.

9. Install Homebrew

10. Install the Azure CLI using Homebrew (requires python@3.11?):

    brew install azure-cli

11. Install the .NET (dotnet) CLI commands:

    DOTNET_ROOT="/opt/homebrew/Cellar/dotnet/7.0.100/libexec"

12. Add $HOME/.dotnet/tools folder to .zshrc file run at bootup.

13. Intall miniconda3.

14. From any folder, create a Conda environment containing the Azure AI SDK:

    conda create --name ai_env python=3.10 pip
    yes | conda activate ai_env
    

    Sample response:

    Collecting package metadata (current_repodata.json): done
    Solving environment: done
     
    ==> WARNING: A newer version of conda exists. <==
      current version: 23.5.0
      latest version: 24.1.1
     
    Please update conda by running
     
     $ conda update -n base -c conda-forge conda
     
    Or to minimize the number of packages updated during conda update use
     
      conda install conda=24.1.1
     
    ## Package Plan ##
     
      environment location: /Users/wilsonmar/miniconda3/envs/ai_env
     
      added / updated specs:
     - python=3.10
     
    The following packages will be downloaded:
     
     package                    |            build
     ---------------------------|-----------------
     python-3.10.13             |h00d2728_1_cpython        12.4 MB  conda-forge
     setuptools-69.1.0          |     pyhd8ed1ab_0         460 KB  conda-forge
     ------------------------------------------------------------
                                            Total:        12.9 MB
     
    The following NEW packages will be INSTALLED:
     
      bzip2              conda-forge/osx-64::bzip2-1.0.8-h10d778d_5
      ca-certificates    conda-forge/osx-64::ca-certificates-2024.2.2-h8857fd0_0
      libffi             conda-forge/osx-64::libffi-3.4.2-h0d85af4_5
      libsqlite          conda-forge/osx-64::libsqlite-3.45.1-h92b6c6a_0
      libzlib            conda-forge/osx-64::libzlib-1.2.13-h8a1eda9_5
      ncurses            conda-forge/osx-64::ncurses-6.4-h93d8f39_2
      openssl            conda-forge/osx-64::openssl-3.2.1-hd75f5a5_0
      pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
      python             conda-forge/osx-64::python-3.10.13-h00d2728_1_cpython
      readline           conda-forge/osx-64::readline-8.2-h9e318b2_1
      setuptools         conda-forge/noarch::setuptools-69.1.0-pyhd8ed1ab_0
      tk                 conda-forge/osx-64::tk-8.6.13-h1abcd95_1
      tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
      wheel              conda-forge/noarch::wheel-0.42.0-pyhd8ed1ab_0
      xz                 conda-forge/osx-64::xz-5.2.6-h775f41a_0
     
    Proceed ([y]/n)? _
    

15. Install the Azure AI CLI using the .NET (dotnet) CLI command:

    dotnet tool install --global Azure.AI.CLI  --prerelease
    

    Response:

    You can invoke the tool using the following command: ai
    Tool 'azure.ai.cli' (version '1.0.0-preview-20240216.1') was successfully installed.
    

16. Obtain Python dependency libraries defined in requirements.txt:

    conda install pip
    pip install -r requirements.txt
    

    Alternately, if there were instead an environment.yml.

17. Get the ai cli:

    ai
    

    Results with the prerelease version:

    AI - Azure AI CLI, Version 1.0.0-preview-20240216.1
    Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.
     
    This PUBLIC PREVIEW version may change at any time.
    See: https://aka.ms/azure-ai-cli-public-preview
     
    ___ ____  ___  _____
      / _ /_  / / _ |/_  _/
     / __ |/ /_/ __ |_/ /_
    /_/ |_/___/_/ |_/____/
     
    USAGE: ai  [...]
     
    HELP
     
      ai help
      ai help init
     
    COMMANDS
     
      ai init [...]             (see: ai help init)
      ai config [...]           (see: ai help config)
     
      ai dev [...]              (see: ai help dev)
     
      ai chat [...]             (see: ai help chat)
      ai flow [...]             (see: ai help flow)
     
      ai search [...]           (see: ai help search)
      ai speech [...]           (see: ai help speech)
     
      ai service [...]          (see: ai help service)
     
    EXAMPLES
     
      ai init
      ai chat --interactive --system @prompt.txt
     
      ai search index update --name MyIndex --files *.md
      ai chat --interactive --system @prompt.txt --index-name MyIndex
     
    SEE ALSO
     
      ai help examples
     
      ai help find "prompt"
      ai help find "prompt" --expand
     
      ai help find topics "examples"
      ai help list topics
     
      ai help documentation
    

18. Navigate to the folder

19. Initialize the ai project:

    ai init
    

20. Select “Initialize: Existing AI project”.
21. Select “LAUNCH: ai login (interactive device code)”

You have signed in to the Microsoft Azure Cross-platform Command Line Interface application on your device. You may now close this window.
   

1. Highlight and copy the code.

2. Open an internet browser (Safari) to the page

   https://microsoft.com/devicelogin

3. Click and paste the code from your laptop Clipboard (to authenticate).
4. Select your user email.
5. Click Continue to “Are you trying to sign in to Microsoft Azure CLI? Only continue if you downloaded the app from a store or website that you trust.”

6. Close the browser tab when you see

   You have signed in to the Microsoft Azure Cross-platform Command Line Interface application on your device. You may now close this window.

7. Confirm that the subscription shown is the one you want to use.

8. cd to the folder

   Notice that file src/copilot_aisdk/requirements.txt contains:

   openai
   azure-identity
   azure-search-documents==11.4.0b6
   jinja2
   

9. Select “(Create w/ standalone Open AI resource)” rather than integrated
10. Select RESOURCE GROUP, CREATE RESOURCE GROUP, CREATE AZURE OPENAI RESOURCE
11. Select a name for “** CREATING **”

12. For “AZURE OPENAI DEPLOYMENT (CHAT)” select “(Create new)”. FIXME:

    CREATE DEPLOYMENT (CHAT)
    Model: *** No deployable models found ***
    CANCELED: No deployment selected
    

13. Select one or skip:
    • ada-embedding-002
    • gpt-35-turbo-16k for chat,
    • gpt-35-turbo-16k
    • gpt4-32k evaluation
      
      

14. Azure AI search resource

    This generates a config.json file in the root of the repo for the SDK to use when authenticating to Azure AI services.

Alternating:

https://github.com/azure-samples/azureai-samples

1. Select the Node sample from the list.

   File devcontainer.json is a config file that determines how your dev container gets built and started.

Vision services

Vision Tutorial

https://microsoftlearning.github.io/mslearn-ai-vision/ EXERCISES based on
https://github.com/MicrosoftLearning/mslearn-ai-vision

• Analyze Images with Azure AI Vision
• Classify images with an Azure AI Vision custom model
• Detect Objects in Images with Custom Vision
• Detect and Analyze Faces
• Read Text in Images
• Analyze Video with Video Analyzer (Indexer)
• Classify Images with Azure AIx Custom Vision
  
  

LLMOps: LLMOps Microsoft Developer videos

AI Foundry GUI

PROTIP: New cloud resources are usually available initially on a small number of locations. For example, at time of writing, <a target=”_blank” href=”https://ai.azure.com/resource/contentunderstanding/overview>”Content Understanding”</a> was only available in australiaeast, swedencentral, westus. So create all resources in these regions to avoid cross-region network fees.

Observe and optimize:

To help debug complex apps or optimize performance, a detailed view of the execution flow within AI apps (including the inputs and outputs of each node within the application) is obtained from Tracing (on ai.azure.com/tracing) connect to “Application Insights” resources created on the portal.

LEARN: To assess operational and evaluation metrics (analytics) for apps, Microsoft offers its “Azure AI Foundry Observability” service (activated at https://ai.azure.com/observability/applicationAnalytics) to integrate with “Azure Monitor Application Insights” that continuously monitor and evaluate deployed AI applications. The services ensure that apps are performant, safe, and produce high-quality results in production. The “Foundry Observability dashboard” provides visibility of critical quality and safety metrics.

Monitoring by connecting to an Application Insights resource.

Conversation history from AI chat completions sessions using all Azure OpenAI models can be stored as “stored completions” metadata to use as datasets for evaluations and fine-tuning.

Batch jobs enable large-scale deployments on models.

Data + indexes include Datasets (flat files), Azure OpenAI Files, Indexes, Vector stores.

Web apps: Deploying an LLM makes it available for use in a website, an application, in production environments.

Cognitive Vision GUI

aka.ms/cognitivevision resolves to YouTube channel “Microsoft Mechanics” rebranded to include “agents”, such as:

• VIDEO “Introducing Azure AI Foundry - Everything you need for AI development”

1. In a web browser, navigate to Vision Studio:

   https://portal.vision.cognitive.azure.com/gallery/featured

2. Sign in.

3. Select a Subscription.

   VIDEO: The menu of AI Vision service categories uses Microsoft’s “Florence foundational model” (new in 2023) models trained with an “open world” of billions of images combined with a large language model. That enables the identification of objects and their location in a frame such as this:

   

4. Click each, then “Try it out”:

   Optical character recognition (OCR):

   • Extract text from images - Extract printed and handwritten style text from images and documents for supported languages.
     
     Spatial analysis - Video Retrieval and Summary:
   • Video Retrieval and Summary - Generate a brief summary of the main points shown in video. Locate specific keywords and jump to the relevant section.
   • Count people in an area - Analyze real-time or recorded video to count the number of people in a designated zone in a camera’s field of view.
   • Detect when people cross a line - Analyze real-time streaming video to detect when a person crosses a line in the camera’s field of view.
   • Detect when people enter/exit a zone - Analyze real-time streaming video to detect when a person enters or exits a zone in the camera’s field of view.
   • Monitor social distancing - Analyze real-time streaming video that tracks when people violate a distance rule in the camera’s field of view.
     
     
   • Face - Detect faces in an image
     
     Image analysis - click “Try it out” on each of these:
   • Recognize products on shelves - Identify products on shelves, gaps in product availability, and compliance for planograms.
   • Customize models with images - Create custom image classification and object detection models with images using Vision Studio and Azure ML.
   • Search photos with image retrieval - Retrieve specific moments within your photo album. For example, you can search for: a wedding you attended last summer, your pet, or your favorite city.
   • Add dense captions to images - Generate human-readable captions for all important objects detected in your image.
   • Remove backgrounds from images - Easily remove the background and preserve foreground elements in your image.
   • Add captions to images - Generate a human-readable sentence that describes the content of an image.
   • Detect common objects in images - Recognize the location of objects of interest in an image and assign them a label.
   • Extract common tags from images - Use an AI model to automatically assign one or more labels to an image.
   • Detect sensitive content in images - Detect sensitive content in images so you can moderate their usage in your applications.
   • Create smart-cropped images - Create cropped image thumbnails based on the key areas of a larger image.
     
     
5. Click “I acknowledge that this service will incur usage to my Azure account.”

6. Select a Region (East US, West Europe, West US, or West US 2)

   The “cog-ms-learn-vision” resource is created for you automatically.

7. Click “View all resources”
8. Check “cog-ms-learn-vision” created for you automatically.
9. Click “Select as default resource”.
10. Close.

Use REST API

https://github.com/Azure-Samples/cognitive-service-vision-model-customization-python-samples

https://microsoftlearning.github.io/mslearn-ai-fundamentals/Instructions/Labs/05-ocr.html

1. On the Getting started with Vision landing page, select Optical character recognition, and then the Extract text from images tile.

2. Under the Try It Out subheading, acknowledge the resource usage policy by reading and checking the box.

3. download ocr-images.zip by selecting

   https://aka.ms/mslearn-ocr-images

4. Open the folder.

5. On the portal, select Browse for a file and navigate to the folder on your computer where you downloaded ocr-images.zip. Select advert.jpg and select Open.

6. review what is returned:

   In Detected attributes, any text found in the image is organized into a hierarchical structure of regions, lines, and words.

   On the image, the location of text is indicated by a bounding box, as shown here:

App for the blind: VIDEO: INTRO: SeeingAI.com. Permissions for the “See It All” app are for its internal name “Mt Studio Web Prod”.

https://docs.microsoft.com/en-us/learn/paths/explore-computer-vision-microsoft-azure

HISTORY: In 2014, Microsoft showed off its facial recognition capabilities with a website (how-old.net which now is owned by others) to guess how old someone is. At conferences they built a booth that takes a picture.

LEARN: https://docs.microsoft.com/en-us/learn/modules/read-text-computer-vision/

DEMO: Seeing AI app talking camera narrates the world around blind people.

• Semantic segmentation is the ML technique which individual pixels in the image are classified according to the object to which they belong.

• Image analysis

• Face detection, analysis, and recognition
• Optical character recognition (OCR) for small amounts of text
• The Read API works asynchronously on images with a lot of text, to parse pages, lines, and words.
• Video Indexer service analyzes the visual and audio channels of a video, and indexes its content.
  
  

Computer Vision

Computer Vision” analyzes images and video to extract descriptions, tags, objects, and text. API Reference, DOCS, INTRO:

• Read the text in the image
• Detects Objects
• Identifies Landmarks
• Categorize image
  
  

Flow API samples

https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?tabs=visual-studio&pivots=programming-language-csharp READ

Computer Vision demo

1. Select images and review the information returned by the Azure Computer Vision web service:

   DEMO: https://aidemos.microsoft.com/computer-vision

   1. Click an image to see results of “Analyze and describe images”. Objects are returned with a bounding box to indicate their location within the image.

   2. Click “Try another image” for another selection.
   3. Click “Next step”.
   4. Read text in imagery.
   5. Read handwriting
   6. Recognize celebrities & landmarks - the service has a specialized domain model trained to identify thousands of well-known celebrities from the worlds of sports, entertainment, and business. The “Landmarks” model can identify famous landmarks, such as the Taj Mahal and the Statue of Liberty.
      
      

   Additionally, the Computer Vision service can:

   • Detect image types - for example, identifying clip art images or line drawings.
   • Detect image color schemes - specifically, identifying the dominant foreground, background, and overall colors in an image.
   • Generate thumbnails - creating small versions of images.
   • Moderate content - detecting images that contain adult content or depict violent, gory scenes.
     
     

   Computer Vision API shows all the features.

HANDS-ON LAB:

1. Right-click “Launch VM mode” for the “AI-900” lab on a Window VM.
2. X: Click the Edge browser icon.
3. X: Click to remove pop-ups.
4. Go to portal.azure.com
5. Sign in using an email which you have an Azure subscription.
6. Type the password. You can’t copy outside the VM and paste into it.
7. X: Do not save your password.
8. Open Visual Studio to see

9. X: On another browser tab, view the repo (faster):

   https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/01%20-%20Image%20Analysis%20with%20Computer%20Vision.ipynb

10. Follow the instructions in the notebook to create a resource, etc.

    TODO: Incorporate the code and put it in a pipeline that minimizes manual actions.

11. To take a quiz and get credit, click in the VM “here to complete the Learn module with a Knowledge Check.

Custom Vision

Azure Custom Vision trains custom models referencing custom (your own) images. Custom vision has two project types:

• Image classification is a machine-learning based form of computer vision in which a model is trained to categorize images based on their (class or) primary subject matter they contain.

• Object detection goes further than classification to identify the “class” of individual objects within the image, and to return the coordinates of a bounding box that indicates the object’s location.
  
  

HANDS-ON: LEARN hands-on lab

• Labfiles for C-Sharp and Python.

LAB: Steps:

1. Create state-of-the-art computer vision models.
2. Upload and tag training images.
3. Train classifiers for active learning.
4. Perform image prediction to identify probable matches to a trained model.

5. Perform object detection to locate elements within an image and return a bounding box.
   
   

6. Open

   https://www.customvision.ai

   DOCS:

7. MS LEARN HANDS-ON LAB:

   aka.ms/learn-image-classification which redirects to
   docs.microsoft.com/en-us/learn/modules/classify-images-custom-vision

8. Instructions are at

9. Load the code from:

   https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/03%20-%20Object%20Detection.ipynb

https://docs.microsoft.com/en-us/learn/modules/evaluate-requirements-for-custom-computer-vision-api/3-investigate-service-authorization Custom Vision APIs use two subscription keys, each control access to an API:

• A training key to access API members which train the model.
• A prediction key to access API members which classify images against a trained model.
  
  

https://docs.microsoft.com/en-us/learn/modules/evaluate-requirements-for-custom-computer-vision-api/4-examine-the-custom-vision-prediction-api

References: CV API

Face => AI

VIDEO: API DEMO:

Azure “Face” is used to build face detection and facial recognition solutions in five categories:

• Face Verification: Check the likelihood that two faces belong to the same person.
• Face Detection: Detect human faces in an image.
• Face Identification: Search and identify faces.
• Face Similarity: Find similar faces.
• Face Grouping: Organize unidentified faces into (face list) groups, based on their visual similarity.
  
  

NOTE: On June 11, 2020, Microsoft announced that it will not sell facial recognition technology to police departments in the United States until strong regulation, grounded in human rights, has been enacted. As such, customers may not use facial recognition features or functionality included in Azure Services, such as Face or Video Indexer, if a customer is, or is allowing use of such services by or for, a police department in the United States.

FaceTutorials

HANDS-ON: LEARN tutorial using these Lab files.

A face location is face coordinates – a rectangular pixel area in the image where a face has been identified.

The Face API can return up to 27 landmarks for each identified face that you can use for analysis. Azure allows a person to can have up to 248 faces. There is a 6 MB limit on the size of each file (jpeg, png, gif, bmp).

Face attributes are predefined properties of a face or a person represented by a face. The Face API can optionally identify and return the following types of attributes for a detected face:

• Age
• Gender
• Smile intensity
• Facial hair
• Head pose (3D)
• Emotion
  
  

Emotions detected in JSON response is a floating point number:

• Neutral
• Anger
• Contempt
• Disgust
• Fear
• Hppiness
• Sadness
• Surprise
  
  

PROTIP: “Happiness: 9.99983543,” is near certainty at 1.0. 2.80234E-08” indicates 8

https://github.com/Azure-Samples/cognitive-services-FaceAPIEnrollmentSample

DEMO: LAB: https://github.com/microsoft/hackwithazure/tree/master/workshops/web-ai-happy-sad-angry

https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/quickstarts-sdk/identity-client-library?tabs=windows%2Cvisual-studio&pivots=programming-language-csharp

Create a Face API subscription

Subscribe to the Face API:

1. Sign in to the Azure portal.
2. Go to
   • Create a resource > AI + Machine Learning > Face</a>

3. Enter a unique name for your Face API subscription name in variable MY_FACE_ACCT

   Paste in setme.sh

   export MY_FACE_ACCT=faceme
   

4. Choose “westus”, the Location nearest to you.
5. Select F0 the free or lowest-cost Pricing tier.
6. Check “By checking this box, I certify that use of this service or any service that is being created by this Subscription Id, is not by or for a police department in the United States.”

7. Click “Create” to subscribe to the Face API.

8. When provisioned, “Go to resource”.

9. In “Keys and Endpoint”, copy Key1 and paste in setme.sh

   export MY_FACE_KEY1=subscription_key
   

The endpoint used to make REST calls is “$MY_FACE_ACCT.cognitiveservices.azure.com/”

https://github.com/wilsonmar/azure-quickly/blob/main/az-face-init.sh

https://docs.microsoft.com/en-us/learn/modules/identify-faces-with-computer-vision/8-test-face-detection?pivots=csharp

References:

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/Overview What is the Azure Face service?

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/

Face API

https://docs.microsoft.com/en-us/azure/cognitive-services/Face/quickstarts/client-libraries?tabs=visual-studio&pivots=programming-language-csharp

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/04%20-%20Face%20Analysis.ipynb

CustomVision

Form Recognizer => Document Intelligence

“Azure Cognitive Services Form Recognizer is now Azure AI Document Intelligence.”

Azure AI Document Intelligence / Form Recognizer is a cloud-based Azure AI service that uses machine learning to extract key-value pairs, text, tables, and key data from your documents.

The number of free pages is limited to 500 per month.

https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/documentintelligence/azure-ai-documentintelligence/samples

https://azuresdkdocs.z19.web.core.windows.net/python/azure-ai-documentintelligence/latest/index.html Azure AI Document Intelligence client library for Python

https://azure.github.io/azure-sdk-for-python/index.html Azure SDK for Python

Video Indexer

Media Services & Storage Account:

1. In Portal, Media Services blade.

2. Specify Resource Group, account name, storage account, System-managed identity.

   

3. In a browser, go to the Video Indexer Portal URL:

   https://www.videoindexer.ai

   NOTE: Video Indexer is under Media Services rather than Cognitive Services.

4. Click the provider to login: AAD (Entra) account, Personal Microsoft account, Google.

   PROTIP: Avoid using Google due to the permissions you’re asked to give:

5. Say Yes to Video Indexer permission to: Access your email addresses & View your profile info and contact list, including your name, gender, display picture, contacts, and friends.

   NOTE: You’ll get an email with subject “Your subscription to the Video Indexer API”.

   On your mobile phone you’ll get a “Connected to new app” notice for Microsoft Authenticator.

6. Click “Account settings”. PRICING: up to 10 hours (600 minutes) of free indexing to website users and up to 40 hours (2,400 minutes) of free indexing to API users. Media reserved units are pre-paid. See FAQ

7. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in

   https://bomonike.github.io/azure-quickly

8. Switch back.
9. In Account settings, click “Copy” to get the Account ID GUID in your Clipboard.

10. Switch to the file which defines Azure environment variable VIDEO_INDEXER_ACCOUNT (in setmem.sh) as described in

    https://bomonike.github.io/azure-quickly

11. Highlight the sample value and paste (Command+V).

12. Switch back.

13. Go to the “Azure Video Analyzer for Media Developer Portal”:DOCS

    api-portal.videoindexer.ai

14. Click “Sign In”. Click “Profile”

    NOTE: The UI has changed since publication of Microsoft’s tutorial, which says “Go to the Products tab, then select Authorization.”

15. Click “Show” on the “Primary key” line. Double-click on the subscription key to copy to Clipboard (Command+C).

16. Switch to the file which defines Azure environment variable VIDEO_INDEXER_API_KEY (in setmem.sh) as described in

    https://bomonike.github.io/azure-quickly

17. Highlight the sample value and paste (Command+V).

18. Switch back.

    Upload video using Portal GUI

    DOCS

    Upload video using program

    https://api-portal.videoindexer.ai/api-details#api=Operations&operation=Get-Account-Access-Token

19. DOCS: Select the Azure Video Indexer option for uploading videos: upload from URL (there is also send file as byte array by an API call, which has limits of 2 GB in size and a 30-minute timeout.

    Program

    az-video-upload.py in https://bomonike.github.io/azure-quickly

20. Make an additional call to retrieve insights.

21. Reference existing asset ID

    Search Media files

22. In “Media files” at https://www.videoindexer.ai/media/library

23. Click “Samples”, and click on a video file to Play to see the media’s people, topics (keywords).

    NOTE: Search results include exact start times where an insight exists, possibly multiple matches for the same video if multiple segments are matched.

24. Click a tag to see where it was mentioned in the timeline.

    Alternately, use the API to search: ???

    Model customizations

    Each video consists of scenes grouping shots, which each contain keyframes.

    • A scene represents a single event within the video. It groups consecutive shots that are related. It will have a start time, end time, and thumbnail (first keyframe in the scene).

    • A shot represents a continuous segment of the video. Transitions within the video are detected which determine how it is split into shots. Shots have a start time, end time, and list of keyframes.

    • Keyframes are frames that represent the shot. Each one is for a specific point in time. There can be gaps in time between keyframes but together they are representative of the shot. Each keyframe can be downloaded as a high-resolution image.

25. Click “Model customizations”

26. Set as thumbnail another keyframe.

https://github.com/Azure-Samples/media-services-video-indexer

https://dev.to/adbertram/getting-started-with-azure-video-indexer-and-powershell-3i32

Azure Form Recognizer

“Form Recognizer” extracts information from images obtained from scanned forms and invoices.

Resources:

• Raza Salehi’s Recognizer tutorial on Pluralsight

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/06%20-%20Receipts%20with%20Form%20Recognizer.ipynb

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/tables-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-java/formrecognizer-java-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-net/azure-form-recognizer-client-sdk-samples/

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/formrecognizer-samples/

OCR

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/05%20-%20Optical%20Character%20Recognition.ipynb

• Image classification - https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/01%20-%20Image%20Analysis%20with%20Computer%20Vision.ipynb

• Object detection - https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/02%20-%20Image%20Classification.ipynb

Ink Recognizer

VIDEO:

Ink converts handwriting to plain text, in 63+ core languages.

It was deprecated on 31 January 2021.

https://docs.microsoft.com/en-us/azure/cognitive-services/ink-recognizer/quickstarts/csharp Quickstart: Recognize digital ink with the Ink Recognizer REST API and C#

QUESTION: Does it integrate with a tablet?

Create Cognitive Services

Azure Machine Learning 2.0 CLI (preview) examples

https://github.com/Azure-Samples/Cognitive-Services-Vision-Solution-Templates

BTW https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-services-quickstart-code/cognitive-services-quickstart-code/ https://github.com/Azure-Samples/cognitive-services-sample-data-files

My script does the same as these manual steps:

1. In Portal.azure.com
2. G+\ Cognitive Services
3. Click the Name you created.
4. Click “Keys and Endpoint” in the left menu.

5. Click the blue icon to the right of KEY 1 heading to copy it to your invisible Clipboard.

6. Endpoint: https://tot.cognitiveservices.azure.com/

   TODO: DOCS: Automate above steps to create compute and server startpup script.

   PROTIP: These instructions are not in Microsoft LEARN’s tutorial.

   “Your document is currently not connected to a compute. Switch to a running compute or create a new compute to run a cell.”

7. Click the Run triangle for “Your document is currently not connected to a compute.”
8. “Create compute”
   • Virtual machine type: CPU or GPU
   • Virtual machine size: Select from all options (64 of them) QUESTION: What is the basis for “recommended”?
   • The cheapest is “Standard_F2s_v2” with “2 cores, 4GB RAM, 16GB storage” for Compute optimized at “$0.11/hr”. See Microsoft’s description of virtual machine types here.
   • Next
   • Compute name: PROTIP: Use 3-characters only, such as “wow” or “eat”.
   • Enable SSH access: leave unchecked
   • Create
   • Wait (5 minutes) for box to go from “Creating” to “Running”.
     
     

Kinds of Cognitive Services CLI

You would save money if you don’t leave servers running, racking up charges.

You can confidently delete Resource Groups and all resources attached if you have automation in CLI scripts that enable you to easily create them later.

Instead of the manual steps defined in this LAB, run my Bash script in CLI, as defined by this DOC:

1. G+\ Cognitive Services.

2. Click the ＋Create a resource button, search for Cognitive Services, and create a Cognitive Services resource with the following settings:
3. Subscription: Your Azure subscription.
4. Resource group: Select or create a resource group with a unique name.
5. Region: Choose any available region:
6. Name: Enter a unique name.
7. Pricing tier: S0
8. I confirm I have read and understood the notices: Selected.

TODO: Instead of putting plain text of cog_key in code, reference Azure Vault. Have the code in GitHub.

Azure has a cognitiveservices CLI subcommand.

https://docs.audd.io/?ref=public-apis

AI Language services

Formerly called “NLP” (Natural Language Processing), Intro: Tutorial: https://docs.microsoft.com/en-us/learn/paths/explore-natural-language-processing

NLP enables the creation of software that can:

• Analyze and interpret text in documents, email messages, and other sources.
• Interpret spoken language, and synthesize speech responses.
• Automatically translate spoken or written phrases between languages.
• Interpret commands and determine appropriate actions.
• Recognize Speaker based on audio.
  
  

Within Microoft, NLP consists of these Azure services (described below):

• LUIS (Language Understanding Intelligent Service)
• Text Analytics
• Speech
• Translator Text
  
  

AI Language Tutorials

https://microsoftlearning.github.io/mslearn-ai-language/

• https://learn.microsoft.com/en-us/training/paths/develop-language-solutions-azure-ai/
• https://github.com/MicrosoftLearning/mslearn-ai-language
• Analyze text
• Create a Question Answering Solution
• Create a language understanding model with the Azure AI Language service
• Custom text classification
• Extract custom entities
• Translate Text
• Recognize and Synthesize Speech
• Translate Speech
  
  

Speaker Recognition

Speaker Recognition for authentication.

In contrast, Speaker Diarization groups segments of audio by speaker in a batch operation.

“In cloudinary”

LUIS

Think of “LUIS” as Amazon Alexa’s frienemy.

https://www.luis.ai, provides examples of how to use LUIS (Language Understanding Intelligent Service) thus:

A machine learning-based service to build natural language into apps, bots, and IoT devices. Quickly create enterprise-ready, custom models that continuously improve.

Bot Framework Emulator Follow the instructions at https://github.com/Microsoft/BotFramework-Emulator/blob/master/README.md to download and install the latest stable version of the Bot Framework Emulator for your operating system.

Bot Framework Composer Install from https://docs.microsoft.com/en-us/composer/install-composer.

Utterances are input from the user that your app needs to interpret.

https://github.com/Azure-Samples/cognitive-services-language-understanding

CHALLENGE: Add natural language capabilities to a picture-management bot.

• Create a LUIS Service Resource (in Python or C#)
• Add Intents
• Add utterances
• Work with Entities
• Use Prebuilt Domains and Entities
• Train and Publish the LUIS Model
  
  

https://www.slideshare.net/goelles/sharepoint-saturday-belgium-2019-unite-your-modern-workplace-with-microsofsts-ai-ecosystem

LUIS Regional websites

Perhaps for less latency, create the LUIS app in the same geographic location where you created the service:

• “(US) West US” in North America: www.luis.ai
  
• “(Europe) West Europe” and “(Europe) Switzerland North”: eu.luis.ai
  
• “(Asia Pacific) Australia East”: au.luis.ai
  
  

Create LUIS resource

NOTE: LUIS is not a service like “Cognitive Services”, but a Marketplace item:

1. In the Azure portal, select + Create a resource.
2. In the “Search services and Marketplace” box, type LUIS, and press Enter.
3. In search results, select “Language Understanding from Microsoft”.
4. Select Create.
5. Leave the Create options set to Both (Authoring and Prediction).
6. Choose a subscription.
7. Create a new resource group named “LearnRG”.

8. Enter a unique name for your LUIS service. This is a public sub-domain to a LUIS Regional websites (above)

   PROTIP: Different localities have different costs.

9. For Authoring Location, choose the one nearest you.
10. For Authoring pricing tier, select F0
11. Set your Prediction location to the same region you chose for Authoring location
12. For Prediction pricing tier, select F0. If F0 is not available, select S0 or other free/low cost tier.
13. Select Review + Create.
14. If the validation succeeds, select Create.
15. After “Your deployment is complete” appears, click “Go to resource” (within Cognitive Services).
16. Within the Resource Management section, select the “Keys and Endpoints” section.
17. Click the blue button to the right of KEY 1’s value. That copies it to your Clipboard.

18. If you are making use of my framework, open the environment variables file:

    code ../setmem.sh

19. Locate the variables and replace the sample values:

    export MY_LUIS_AUTHORING_KEY=”abcdef1234567896b87f122281e9187e” export MY_LUIS_ENDPOINT=””

    SECURITY PROTIP: Putting key values in a global variables definition file separate from the code.

1.

authoring_key = env:MY_LUIS_AUTHORING_KEY
   authoring_endpoint = env:MY_LUIS_ENDPOINT
   

1. Select a LUIS Regional websites (above)
2. Login.
3. Choose an authoring resource (Subscription). If there are several listed, select the top one.

4. Create a new authoring service (Azure service).

   Create a LUIS app in the LUIS portal

5. Now at https://www.luis.ai/applications, notice at the upper-right “PictureBotLUIS (westus, F0)” where the Directory name usually appears in the Portal.

   The pages displayed will be different if you have already created a LUIS app or have no apps created at all. Select either Go to apps, or the apps option that is available on your initial LUIS page.

6. Select “+ New app” for conversation Apps for the “Create new app” pop-up dialog.

   The LUIS user interface is updated on a regular basis and the actual options may change, in terms of the text used. The basic workflow is the same but you may need to adapt to the UI changes for the text on some elements or instructions given here.

   Take note of the other options such as the ability to import JSON or LU file that contains LUIS configuration options.

7. Give your LUIS app a name, for example, PictureBotLUIS.

8. For Culture, type “en-us” then select the appropriate choice for you language.

9. Optionally, give your LUIS app a description so it’s clear what the app’s purpose is.
10. Optionally, if you already created a Prediction Resource, you can select that in the drop-down for Prediction resource.

11. Select Done.

12. Dismiss the guidance dialog that may display.

    

    DEFINITION: Each bot action has an intent invoked by an utterance, which gets processed by all models. The top scoring model LUIS selects as the prediction.

    An utterance that don’t map to existing intents is called the catchall intent “None”.

    Intents with significantly more positive examples (“data imbalance” toward that intent) are more likely to receive positive predictions.

    Create a LUIS app with code

13. On your local machine (laptop), install Visual Studio Code for your operating system.

14. If you will be completing your coding with Python, ensure you have a Python environment installed locally. Once you have Python installed, you will need to install the extension for VS Code.

    Alternately, to use C# as your code language, start by installing the latest .NET Core package for your platform. You can choose Windows, Linux, or macOS from the drop-down on this page. Once you have .NET Core installed, you will need to add the C# Extension to VS Code. Select the Extensions option in the left nav pane, or press CTRL+SHIFT+X and enter C# in the search dialog.

15. Create a folder within your local drive project folder to store the project files. Name the folder “LU_Python”. Open Visual Studio Code and Open the folder you just created. Create a Python file called “create_luis.py”.

    cd
    cd projects
    mkdir LU_Python
    cd LU_Python
    touch create_luis.py
    code .

16. Install the LUIS package to gain access to the SDK.

    sudo pip install azure-cognitiveservices-language-luis

17. Open an editor (Visual Studio Code)

    Create entities in the LUIS portal

18. Be signed in to your LUIS app and on the Build tab.
19. In the left column, select Entities, and then select + Create.
20. Name the entity facet (to represent one way to identify an image).

21. Select Machine learned for Type. Then select Create.

    Create a prediction key

    https://docs.microsoft.com/en-us/learn/modules/manage-language-understanding-intelligent-service-apps/2-manage-authoring-runtime-keys

22. Sign in to your LUIS portal.
23. Select the LUIS app that you want to create the prediction key for.
24. Select the Manage option in the top toolbar.

25. Select Azure Resources in the left tool bar.

    Unless you have already created a prediction key, your screen should look similar to this. The key information is obscured on purpose.

    A Starter Key provides 1000 prediction endpoint requests per month for free.

    Examples of utterances are on the Review endpoint utterances page on the Build tab.

    Install and run LUIS containers

    docker pull mcr.microsoft.com/azure-cognitive-services/luis:latest

26. Specify values for Billing LUIS authoring endpoint URI and ApiKey:

    docker run --rm -it -p 5000:5000 --memory 4g --cpus 2 --mount type=bind,src=c:\input,target=/input
    --mount type=bind,src=c:\output\,target=/output
    mcr.microsoft.com/azure-cognitive-services/luis
    Eula=accept
    Billing={ENDPOINT_URI}
    ApiKey={API_KEY}

    ”–mount type=bind,src=c:\output\,target=/output” indicates where the LUIS app saves log files to the output directory. The log files contain the phrases entered when users hit the endpoint with queries.

27. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -G \
    -d verbose=false \
    -d log=true \
    --data-urlencode "query=Can I get a 5x7 of this image?" \
    "http://localhost:5000/luis/v3.0/apps/{APP_ID}/slots/production/predict"
    

28. Get the AppID from your LUIS portal and paste it in the placeholder in the command.

    curl -X GET \
    "http://localhost:5000/luis/v2.0/apps/{APP_ID}?q=can%20I%20get%20an%20a%205x7%20of%20this%20image&staging=false&timezoneOffset=0&verbose=false&log=true" \
    -H "accept: application/json"
    

DEMO JSON responses

1. DEMO: voice control lighting in a virtual home.

   https://aidemos.microsoft.com/luis/demo

2. Select suggested utterances to see the JSON response:

   • Book me a flight to Cairo
   • Order me 2 pizza
   • Remind me to call my dad tomorrow
   • Where is the nearest club?
     
     

   Type instructions, use the microphone button to speak commands.

   LUIS identifies from your utterance your intents and entities.

   Entity types

   • List - fixed, closed set of related words (small, tiny, smallest). Case-sensitve
   • RegEx - (credit card numbers)
   • Prebuilt
   • Pattern.Any
   • Machine Learned
     
     

LUIS CLI

Run my az-luis-cli.sh.

DOCS:

https://github.com/cloudacademy/using-the-azure-machine-learning-sdk

1. Create a Resource referenced when LUIS Authoring is defined. The resource name should be lower case as it is used for the endpoint URL, such as:

   https://luis-resource-name.cognitiveservices.azure.com/

2. The Bot Framework CLI requires Node.js.

   npm i -g npm

3. Install Bot Framework Emulator per instructions at:
   

   https://github.com/Microsoft/BotFramework-Emulator/blob/master/README.md

   Download to downloads folder at:
   https://aka.ms/bf-composer-download-mac

4. Install Bot Framework Composer per instructions at:
   

   https://docs.microsoft.com/en-us/composer/install-composer

   On a Mac, download the .dmg
   https://aka.ms/bf-composer-download-mac

5. Use Node.js to install the latest version of Bot Framework CLI from the command line:

   npm i -g @microsoft/botframework-cli

   https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-bf-cli-deploy-luis?view=azure-bot-service-4.0

   LUIS can generate from models a TypeScript or C# typw (program code).

   Luis.ai

6. DEMO: https://www.luis.ai

7. Sign in and create an authoring resource refercening the Resource Group.

   PROTIP: In the list of Cognitive services kinds, resources and subscription keys created for LUIS authoring are separate than ones for prediction runs so that utilization for the two can be tracked separately.

   https://aka.ms/AI900/Lab4 which redirects to
   Create a language model with Language Understanding which trains a (LUIS) language model that can understand spoken or text-based commands. He’s Alexa’s boyfriend, ha ha.

8. Process Natural Lanaguage using Azure Cognitive Language Services

   https://github.com/MicrosoftLearning/AI-102-LUIS contains image files for reference by https://github.com/MicrosoftLearning/AI-102-Code-Repos https://github.com/MicrosoftLearning/AI-102-Process-Speech

   PROTIP: LUIS does not perform text summarization. That’s done by another service in the pipeline.

Terminology:

• Utterance is the user’s input that a model needs to interpret, such as “turn the lights on”.
• Entity is the word (or phrase) that is the focus of the utterance, such as “light” in our example.
• Intent is the action or task that the user wants to execute. It reflects in utterance as a goal or purpose. For example, “TurnOn”.

References:

• VIDEO: Starship commander enabled in-game voice commands using Azure.

• Adding Language Understanding to Chatbots With LUIS by Emilio Meira

• github.com/chomado/GoogleHomeHack in slides by Madoka Chiyoda

Text Analytics API Programming

Techniques DOCS:

NOTE: The previous version references interactive Python Notebooks such as MS LEARN HANDS-ON LAB referencing “07 - Text Analytics.ipynb”.

1. Look at the DEMO GUI at:

   aidemos.microsoft.com/text-analytics

2. Click “Next Step” through the various processing on a sentence:

   • Sentiment analysis (how positive or negative a document is)

   • Key phrase extraction

   • Entity Linking to show link in Wikipedia.

   • Bing Entity Search

   • Text to Speech services:
   • Language Detection (is it English, German, etc.)
   • Translator Text
     
     

API:

Text Analytics API Reference

Some Text Analytics API services are synchronous and asynchronous

Cloud Academy lab “Using Text Analytics in the Azure Cognitive Services API”

docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/textanalytics-samples

Azure Text Analytics client library samples for JavaScript has step-by-step instructions:

1. Logging into the Microsoft Azure Portal

   <a target=”_blank” href=”https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.CognitiveServices%2Faccounts”“>Cognitive Services</a>

2. Select the service already defined for you.

   Retrieving Azure Cognitive Services API Credentials

3. At the left menu click “Keys and Endpoint”. The Endpoint URL contains:

   https://southcentralus.api.cognitive.microsoft.com/

   The Endpoint is the location you’ll be able to make requests to in order to interact with the Cognitive Services API. The Key1 value is the key that will allow you to authenticate with the API. Without the Key1 value, you will receive unauthenticated errors.

4. In the Azure Portal, type Function App into the search bar and click Function App:

alt

1. Click the only option:

alt

You’ll be brought to the Function App blade. While a complete summary of function apps is outside the scope of this lab, you should know the function apps allow you to create custom functions using a variety of programming languages, and to trigger them using any number of events.

In the case of this lab step, you’ll set up a function app to interact with the Cognitive Services API using Node.js. You’ll then configure the function to be triggered by visiting its URL in a browser tab.

1. On the left side of the blade, click on Functions and then + Add.

2. Click the HTTP Trigger option:

alt

Important: If you don’t see the HTTP trigger option, click More Templates and then Finish and view templates to see the correct option.

1. Name the function language-detection and click Create:

alt

1. Click Code + Test in the left sidebar then look at the upper part of the console and switch to the function.json file:

2. Replace the contents of the file with the following snippet and click Save:

   {
     "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "get",
        "post"
      ]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
     ],
     "disabled": false
   }

The function.json file manages the behavior of your function app.

1. Switch to the index.js file:

2. Replace the contents of the file with the following code:

'use strict';
 
let https = require('https');
const subscription_key = "INSERT_YOUR_KEY_HERE";
const endpoint = "INSERT_YOUR_ENDPOINT_HERE";
const path = '/text/analytics/v2.1/languages';
 
module.exports = async function (context, req) {
  let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},
    ]
  };
 
  let body = JSON.stringify(documents);
 
  let request_params = {
    method: 'POST',
    hostname: (new URL(endpoint)).hostname,
    path: path,
    headers: {
      'Ocp-Apim-Subscription-Key': subscription_key,
    }
  };
 
  let response = await makeRequest(request_params, body);
 
  context.res = {
    body: response
  }
 
};

function makeRequest(options, data) {
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      res.setEncoding('utf8');
      let responseBody = '';

      res.on('data', (chunk) => {
        responseBody += chunk;
      });

      res.on('end', () => {
        resolve(JSON.parse(responseBody));
      });
    });

    req.on('error', (err) => {
      reject(err);
    });

    req.write(data)
    req.end();
  });
}

1. Replace the contents of subscription_key and endpoint with the key and endpoint you retrieved in an earlier step, so that the variables look similar to this:

Copy code 1 2 const subscription_key = “4779caba69344c3d97bca9863d726af6”; const endpoint = “https://southcentralus.api.cognitive.microsoft.com/”;

1. Click Save.

2. While a deep understanding of code is outside the scope of this lab, you should take note of a couple of things:

The endpoint and subscription key that you set will allow the function app to communicate with your Cognitive Services API. The endpoint tells the app where to find the API, and the subscription key allows the app to authenticate. The path variable declares the exact path the app should request on the Cognitive Services API. This path system allows the API to offer many different services in one endpoint: Copy code 1 const path = ‘/text/analytics/v2.1/languages’;

The documents variable declares a list of documents that will be passed to the language detection API. The API will return the languages that these documents are most likely written in Currently there are three documents, and the goal will be to get the language used in each one:

let documents = {
    'documents': [
      {'id': '1', 'text': 'This is a document written in English.'},
      {'id': '2', 'text': 'Je suis une phrase écrite en français.'},
      {'id': '3', 'text': 'Este es un documento escrito en español.'},
    ]
  };

The rest of the code simply manages the formatting of the data, requesting the language detection service from the API, and returning the result. Next, you’ll visit the function app’s URL to see it work.

1. Look at the upper command bar and click Test/Run.

2. Select GET as Http Method before clicking Run:

3. View the output in the Output tab:

alt

Notice that what’s returned is a JSON object with a “documents” object containing three results, one for each language you submitted to it. Each result has a “name” value with the predicted language and a “score” value with the likelihood that the decision is accurate. A score of 1 means that the Cognitive Services API was completely confident in its language detection. Here’s what the formatted JSON object looks like, for reference:

{
  "documents": [
    {
      "id": "1",
      "detectedLanguages": [
        {
          "name": "English",
          "iso6391Name": "en",
          "score": 1
        }
      ]
    },
    {
      "id": "2",
      "detectedLanguages": [
        {
          "name": "French",
          "iso6391Name": "fr",
          "score": 1
        }
      ]
    },
    {
      "id": "3",
      "detectedLanguages": [
        {
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1
        }
      ]
    }
  ],
  "errors": []
}

1. Using the Azure Text Analytics API for Language Detection
2. Using the Azure Text Analytics API for Sentiment Analysis

Sentiment Analysis

API Reference

The output is a number from 0 to 1, with 1 being the most positive language and zero being the most negative opinion expressed.

Key phrase extraction

CHALLENGE: API Reference

Named Entity Recognition (NER) identifies entities in the text and group them into different entity categories, such as organization name, location, event, etc.

"documents": [
    {
      "id": "1",
      "keyPhrases": [
        "world",
        "input text"
      ]
    },

Detect Language

CHALLENGE: API Reference

vs. Content Moderator - Moderate Detect Language to Auto Correct, PII, listid, classify, language. Classification of Profanity returns JSON with several categories:

• Category 1: sexually explicit or adult in certain situations.
• Category 2: sexually suggestive or mature in certain situations.
• Category 3: considered offensive in certain situations.
  
  

https://learn.microsoft.com/en-us/training/modules/analyze-text-ai-language/3-detect-language

v2.1 API Testing Console

Input text is in a formatted JSON document files of up to 5,120 characters. Each file can contain up to 1,000 id’s, each associated with a text string.

As with GraphQL, the API returns the detected language and a numeric score between 0 and 1. Scores close to 1 indicate 100% certainty that the identified language is true. A total of 120 languages are supported.

   "id": "3",
      "detectedLanguages": [
        {
          "name": "Spanish",
          "iso6391Name": "es",
          "score": 1
        }
   

Only one language code is returned for each document submitted. Mixed language content within the same document returns the language with the largest representation in the content, but with a lower positive rating, reflecting the marginal strength of that assessment.

Named Entity Recognition

CHALLENGE: API Reference

Sample Response:

{
  "documents": [{
    "id": "1",
    "entities": [{
      "name": "Seattle",
      "matches": [{
        "wikipediaScore": 0.15046201222847677,
        "entityTypeScore": 0.80624294281005859,
        "text": "Seattle",
        "offset": 26,
        "length": 7
      }],
      "wikipediaLanguage": "en",
      "wikipediaId": "Seattle",
      "wikipediaUrl": "https://en.wikipedia.org/wiki/Seattle",
      "bingId": "5fbba6b8-85e1-4d41-9444-d9055436e473",
      "type": "Location"
    }, {
      "name": "last week",
      "matches": [{
        "entityTypeScore": 0.8,
        "text": "last week",
        "offset": 34,
        "length": 9
      }],
      "type": "DateTime",
      "subType": "DateRange"
    }]
  }],
  "errors": []
}

Linked Entities from a well known knowledge base

API Reference

Speech Services

Microsoft has incorporated Immersive Reader throughout their products. Here is a good deep dive about what features are currently available on each product and device type:

Speech-to-text (STT) has two different REST APIs:

• Speech-to-text REST API v3.0 is used for Batch transcription and Custom Speech.

• Speech-to-text REST API for short audio is used for online transcription as an alternative to the Speech SDK. Requests using this API can transmit only up to 60 seconds of audio per request.
  
  

Microsoft Cognitive Services Speech SDK Samples

LAB:

https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/

https://docs.microsoft.com/en-us/learn/modules/transcribe-speech-input-text/?WT.mc_id=cloudskillschallenge_efc530c5-7105-4c12-8eb3-bc20ae3bee78

Speech-to-text

Transcriptions can be done in real-time or in batch mode.

Batch mode is when audio recordings are stored on a file share, and a shared access signature (SAS) URI is used by a program to asynchronously receive transcription results.

1. Take the introductory tutorial:

   Introduction to Machine Learning with Hands-On Labs

   https://azure.microsoft.com/en-us/documentation/articles/machine-learning-studio-overview-diagram

2. Create a model.

3. Prepare Data:

   As per this video using

   • Clean Missing Data - Clip Outliers
   • Edit Metadata
   • Feature Selection
   • Filter
   • Learning with Counts
   • Normalize Data
   • Partition and Sample
   • Principal Component Analysis
   • Quantize Data
   • SQLite Transformation
   • Synthetic Minority Oversampling Technique
     
     

4. Train the model

   • Cross Validation
   • Retraining
   • Parameter Sweep
     
     

5. Score and test the model.

6. Make predictions with Elastic APIs

   • Request-Response Service (RRS) Predictive Experiment - Batch Execution Service (BES)
   • Retraining API
     
     

https://github.com/timothywarner/ai100/tree/master/Speech-to-Text

Speech Translation (Speech-to-text aka STT)

Get the monthly subscription mobile app on iPhone, Android, or Amazon. It has a Phrasebook of common phrases.

DEMO: Speech Translation recognizes and synthesizes speech, and translates spoken languages. REMEMBER: The sequence of services involves two APIs:

Speech-to-Text API -> Speech Correction -> Machine Translation -> Text-to-Speech API

Azure Cognitive TTS Samples

“Speech Recognition” and Text Analysis are not involved in this use case.

Telephone voice menus use “Speech Synthesis”, defined by the Speech Synthesis Markup Language (SSML).

https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/08%20-%20Speech.ipynb

https://github.com/timothywarner/ai100/tree/master/Speech-to-Text

https://github.com/MicrosoftLearning/AI-SpeechToText

The speech-to-text service includes multiple pre-defined voices with support for multiple languages and regional pronunciations, with language detection. In addition to standard voices, neural voices leverage neural networks to overcome common limitations in speech synthesis with regard to intonation, resulting in a more natural sounding voice.

PROTIP: Neural voices are created from samples that use a 24 khz sample rate.

Speech recognition can use a acoustic model of phonemes (sounds) or a language model that matches phonemes with words.

Custom voices can be created with the text-to-speech API.

PROTIP: Since you have to use your own subscription to follow this tutorial from Microsoft, skip clicking “Launch VM mode” and follow the Python notebook on Speech on the regular Portal.

SAMPLE: https://docs.microsoft.com/en-us/samples/azure-samples/cognitive-speech-tts/azure-cognitive-tts-samples/

1. PROTIP: In a CLI window, run my Bash shell script to Create a Cognitive Services resource and get its two keys:

   cd ~/clouddrive/azure-quickly
   git pull
   ./az-cog-cli.sh
   

   To synthesize speech, the system typically tokenizes the text to break it down into individual words, and assigns phonetic sounds to each word. It then breaks the phonetic transcription into prosodic units (such as phrases, clauses, or sentences) to create phonemes that will be converted to audio format. These phonemes are then synthesized as audio by applying a voice, which will determine parameters such as pitch and timbre; and generating an audio wave form that can be output to a speaker or written to a file.

   Text-to-speech

To specify that the speech input to be transcribed to text is in an audio file, use AudioConfig.

Change the voice used in speech synthesis by setting the SpeechSynthesisVoiceName property of the SpeechConfig object to the desired voice name.

Speech Translation

Text (text-to-text aka TTT)

Microsoft’s Translator service can translate text between more than 90 languages and dialects (including Klingon in Star Trek), specified using ISO 639-1 two-letter language codes and 3166-1 cultural codes such as “en-US” for US English, “en-GB” for British English, “fr-CA” for Canadian French, etc.

• Raza Salehi’s Translator

Hands-on tool without a compute instance:

1. translator.microsoft.com

2. Click on “Start conversation”, log in and enter your name and language.

3. Share the conversation code with other participants, who can join using the Micreosoft Translator app or website.

4. Speak or type in your language to communicate with other participants in the conversation. Other participants will see your messages in their own language.

https://docs.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/overview What is Custom Translator?

Foe Parallel Data, equivalent documents in different languages:

• https://docs.microsoft.com/en-us/azure/cognitive-services/
• https://docs.microsoft.com/ja-jp/azure/cognitive-services/
• https://docs.microsoft.com/ko-ko/azure/cognitive-services/

https://docs.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/how-to-upload-document

09 - Translation.ipynb

LEARN: Translate text with Azure AI Translator service

BLAH: You are asked to use your own Subscription anyway, so instead of the Exercise - Translate text and speech, use portal.azure.com directly.

A Python program can run from your laptop or mobile phone making API calls to the Translator endpoint at:

https://api.cognitive.microsofttranslator.com/translate?api-version=3.0

DOCS, API DOCS

VIDEO; Raza Salehi’s 1 hr video course “Build a Translator system”.

VIDEO intro with sample code at https://github.com/microsoft/text-analytics-walkthrough

Python notebook to Translate

For response “script” : “Latn”, text was transliterated in English.

A custom translator is needed to train a model to recognize and translate domain-specific words and phrases in specific industries such as aerospace, automotive, chemistry, mechanical, etc.

portal.customtranslator.azure.ai

Training is done by have pairs of documents (English and French, etc.).

10,000 aligned parallel sentences are neede to train a translator.

In addition to Microsoft Office formats, files with extension .ALIGN for parellel languages are perfectly aligned. Translation Memory systems can export parallel documents in XLF, XLIFF, TMX, suffix. Microsoft’s LocStudio files have .LCL suffix.

Translation runs can each take several hours. So batch processing is supported.

If you don’t have admin

Need admin approval
Mt Studio Web Prod
Mt Studio Web Prod needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.
   

References:

• Microsoft Azure Cognitive Services: Translator Text API” video course</a> by William Myers

Immersive Reader

• Raza Salehi’s Immersive Reader

Other sample code

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/documenttranslation-samples/

Conversational AI

HISTORY: In 2015, Microsoft unleashed the Tay chat bot, then had to bring it down after hackers submitted enough racial slurs that they fooled the system into thinking that was normal and acceptable.

HISTORY: XiaoIce, a chatbot Microsoft launched in China, “has more than 200 million users, has engaged in 30 billion conversations, and has an average conversation length of 23 turns, which averages out to about half an hour, achieving human parity at translation from Chinese to English. Japan-based Rinna and the US-based Zo)

Explore-conversational-ai

A Bot Framework enables the creation of Virtual Assistant

A LUIS app creates these types of entities:

• Machine-learned
• List
• RegEx
• Pattern.any
  
  

https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/concepts/plan?tabs=v1

Bots are extended by Skills

QnA Maker

The cognitive service name “QnA Maker” (Question and Answer Maker)</a> is a cloud-based API service that lets you create a conversational question-and-answer layer over your existing data. The service enables the building of knowledge bases of questions and answers that form the basis of a dialog between a human and an AI agent.

Microsoft created the QnA Maker portal to make it easier than writing code to create and manage knowledge bases using the QnA Maker REST API or SDK.

The knowledge base gets smarter as it continually learns from user behavior.

The knowledge base can be built by extracting questions and answers from your semi-structured content, including FAQs, manuals, and documents.

QnA Maker limits control the size of Knowledge base.

### Create QnA Service

https://github.com/Microsoft/BotBuilder-CognitiveServices/tree/master/CSharp/Samples/QnAMaker is only for C#.

1. View the DOCS:

   View the v2 (previous release)

2. The Jupyter notebook:

   https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/11%20-%20QnA%20Bot.ipynb

3. On an appropriate browser profile (personal or work email) associated with the subscription you wnat to use.

4. Go to the QnA Maker portal at:

   https://qnamaker.ai

5. Sign in.

   https://docs.microsoft.com/en-us/learn/paths/explore-conversational-ai/

6. “Create a knowledge base” from the top menu.
7. STEP 1: “Create a QnA service”, which opens another tab

8. Basics: During testing, do NOT click the checkbox for “Managed”. In prod, telemetry and compute are included automatically with your QnA Maker resource. If you do not select managed, you will be prompted to create an App Insights and App Service resources for the required telemetry and compute that you will have to manage for your QnA Maker resource. Read more <a target=”_blank” href=”“https://aka.ms/qnamaker-createoptions-description”>here</a>.

9. Subscription: Your Azure subscription
10. Resource group: Select an existing resource group or create a new one

11. Resource group location: “(US) West US”

12. Name: A unique name for your QnA resource (App name)
13. Pricing tier: both are “3 transactions per second, 100 transactions per minute”:
    • Free F0 (3 managed documents per month, 5…)
    • Standard S0 ($10 per month for unlimited documents …)
      
      

    App Service details - for runtime :

14. Azure Search location: Any available location
15. Azure Search pricing tier: F
    • Free F (50 MB in 3 indexes)
    • Basic B (2 GB in 15 indexes, 3 scale-out units per service on 1 partition with 3 replicas)
    • Standard S1 (25 GB in 50 indexes) - default
    • Standard S2 (100 GB in 200 indexes)
    • Standard S3 (200 GB in 200 indexes)
    • Standard S3 (1000 Index/Partition)
    • L1 (1 TB in 10 indexes, High Storage, 36 units)
    • L2 (2 TB in 10 indexes, High Storage, 36 units)
      
      

    Note: If you have already provisioned a free-tier QnA Maker or Azure Search resources, your quota may not allow you to create another one. In which case, select a tier other than F0 / F.

    App Service details - for runtime :

16. App name: Same as Name (“.azurewebsites.net” will be appended automatically)

17. Website location: Same as Azure Search location

    App insights details - for telemetry and chat logs :

18. App insights: Disable, which will hide the “App insights location”, but appear in Review.

19. Click “Review + create”. Create. NOTE: when these resources complete deployment:
    • Microsoft.Web/sites/config
    • Microsoft.CognitiveServices/accounts
    • Microsoft.Search/searchServices
    • microsoft.insights/components
    • microsoft.insights/components
    • Microsoft.Web/sites
    • Microsoft.Search/searchServices
    • Microsoft.Web/serverfarms
      
      

20. While you wait for the dots to stop flashing “Deployment in progress”, return to the QnA Maker portal tab. You may have timed out.

21. When “Your deployment is complete”, click “Go to resources” for “Congratulations! Your keys are ready.”

    Connect QnA service to KB

    STEP 2: Connect your QnA service to your KB.

22. “Refresh” the list of available QnA service resources.
23. Microsoft Azure Directory ID: The Azure directory ID for your subscription
24. Azure subscription name: Your Azure subscription

25. Azure QnA service: The QnA service resource you created in the previous step

    NOTE: In the Preview there is a checkbox “Enable language setting per knowledge base”.

26. Language: English

    STEP 3: Name your KB.

27. Type a name: For example: “Margie’s Travel KB”. Spaces are allowed?

    STEP 4: Populate your KB.

28. ”+ Add URL”

29. Copy and paste this example URL:

    https://github.com/MicrosoftDocs/ai-fundamentals/raw/master/data/qna_bot/margies_faq.docx

30. Add file

31. chit-chat: Adding “chit-chat” to your knowledge base (by selecting a personality) automatically adds questions and responses to your knowledge base, which enables your bot to answer small-talk questions in a voice (personality) of your choice:
    • None
    • Professional « select this.
    • Friendly
    • Witty
    • Caring
    • Enthusiastic
      
      

    QUESTION: What is the range of popularity?

    QUESTION: Extraction? I’m stuck here.

32. Do NOT check “Enable multi-turn extraction from URLs, .pdf or .docx files.”

33. Click “Create your KB”. Wait for a minute or so while your Knowledge base is created.

34. Review the questions and answers that have been imported from the FAQ document and the professional chit-chat pre-defined responses.

    Test the knowledge base

    Make an API call

    https://go.microsoft.com/fwlink/?linkid=2100125

    https://go.microsoft.com/fwlink/?linkid=2100213 Coding</a>

Decision (Making)

Anomaly Detector

Anomaly Detector will be retired on 10/1/2026.

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-net/azure-anomaly-detector-client-sdk-samples/

Among Anomaly Detector API Samples

Anomaly Detector identifies potential problems early on.

https://docs.microsoft.com/en-us/learn/modules/get-started-ai-fundamentals/3-understand-anomaly-detection

1. Sensors in the car collect telemetry, such as engine revolutions, brake temperature, and so on.
2. An anomaly detection model is trained to understand expected fluctuations in the telemetry measurements over time.
3. If a measurement occurs outside of the normal expected range, the model reports an anomaly that can be used to alert the race engineer to call the driver in for a pit stop to fix the issue before it forces retirement from the race.
   
   

Resources:

• Raza Salehi’s Anomaly Detector

Content Moderator

Content Moderator services detect potentially offensive or unwanted content.

This has been deprecated.

Evaluate, Find Faces, Match, OCR.

https://www.youtube.com/watch?v=gVFiA6ZQNAw

CHALLENGE: DOCS:

https://docs.microsoft.com/azure/cognitive-services/Content-Moderator/overview?WT.mc_id=Portal-Microsoft_Azure_Support#data-privacy-and-security

https://docs.microsoft.com/en-us/azure/cognitive-services/content-moderator/client-libraries?tabs=visual-studio&pivots=programming-language-csharp

Response from the Text Moderation API include:

• What type of potentially unwanted words were found (Profanity)
• A list of potentially unwanted words found in the text.
• Possible personally identifiable information (PII) found in the text.
  
  

Content Moderation (Evaluate, Find Faces, Match, OCR)

1. G+/ to Content Moderator”

2. “Create Custom Moderator”

   Oops!

   Could not create the marketplace item

   This marketplace item is not available.

   When working:

3. Select Create.

4. Enter a unique name for your resource, select a subscription, and select a location close to you.

5. Select the pricing tier for this resource, and then select F0.

6. Create a new resource group

7. Select Create.

Metrics Advisor

Metrics Advisor monitors metrics and diagnoses issues.

https://docs.microsoft.com/en-us/samples/azure/azure-sdk-for-python/metricsadvisor-samples/

Personalizer

Personalizer creates rich, personalized experiences for every user.

• Raza Salehi’s Personalizer

Search

knowledge-mining Tutorials

Knowledge Mining Solution Accelerator

HANDS-ON: Tutorial references these labfiles

• Create an Azure AI Search solution
• Create a Custom Skill for Azure AI Search
• Create a Knowledge Store with Azure AI Search
• Enrich an AI search index with custom classes
• Implement enhancements to search results
• Enrich a search index using Azure Machine Learning model
• Add to an index using the push API
• Debug search issues
• Set up semantic search
• Use the REST API to run vector search queries
  
  

Bing Search

1. Get on the Bing Resource portal GUI.

2. Define a Resource Group.

3. Price Tier: Free

   PROTIP: Autosuggest requires the “S2” (Standard) pricing tier. Spell Check requires either S1 or S2.

4. Click “Automation Options” link for a JSON Template.
5. Click “Create” at the top.

6. Create

   Notice the service name at the upper left is “Microsoft.BingSearch” and has a Global location. It’s Endpoint is: https://api.bing.microsoft.com/

Search for AI enrichment of docs

The Azure Cognitive Search service uses a Cognitive Search resource to support AI-powered search and knowledge mining solutions such as:

• Index documents and data from a range of sources.
• Use cognitive skills to enrich index data.
• Store extracted insights in a knowledge store for analysis and integration.
  
  

https://blog.api.rakuten.net/top-10-best-search-apis/

“Document cracking” during indexing extracts text content from unstructured text or non-text content (such as images, scanned documents, or JPEG files). The indexer accesses an Azure data storage service.

https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

Other services

HealthInsights

Bot Service

provides a platform for creating, publishing, and managing bots. Developers can use the Bot Framework to create a bot and manage it with Azure Bot Service - integrating back-end services like QnA Maker and LUIS, and connecting to channels for web chat, email, Microsoft Teams, and others.

Microsoft Bot Framework supports two approaches to integrate bots with agent engagement platforms such as Customer support service:

• Bot as agent distributes calls to bots on the same level as live (human) agents. Handoff protocols regulate a bot’s disengagement and transfer to live person.
• Bot as proxy presents a bot to filter interaction with live people.

1. DEMO: See a healthcare bot built using the Azure Bot Service:

   <a target=”_blank”” href=” https://www.microsoft.com/research/project/health-bot/”> https://www.microsoft.com/research/project/health-bot</a>

   Select the option to Try a demo of an example end-user experience. Use the web chat interface to interact with the bot.

MS LEARN: Create a Bot with the Bot Framework Composer

1. Run the Python Jupyter notebook

   https://github.com/MicrosoftLearning/mslearn-ai900/blob/main/11%20-%20QnA%20Bot.ipynb Conversational AI LAB

2. Sign in using the Microsoft account associated with your Azure subscription.

PROTIP: Use NVM to install Node

https://github.com/Microsoft/botbuilder-tools#install-cli-tools says to install Node.js version 10.14.1 or higher

https://github.com/microsoft/botframework-cli says to install Node.js version 12

Since the current version is now 16, we cannot use the command suggested in the doc:

npm i -g @microsoft/botframework-cli

Bot Framework

1. bf is the bot framework CLI command

   One-stop-shop CLI to manage your bot’s resources. BF CLI and AZ CLI together cover your end-to-end bot development workflow needs.
    
   VERSION
     @microsoft/botframework-cli/4.13.3 darwin-x64 node-v16.1.0
    
   USAGE
     $ bf [COMMAND]
    
   COMMANDS
     chatdown      Converts chat dialog files in <filename>.chat format into
                transcript files. Writes corresponding <filename>.transcript
                for each .chat file.
     config        Configure various settings within the cli.
     dialog        Dialog related commands for working with .schema and .dialog
                files.
     help          display help for bf
     lg            Parse, collate, expand and translate lg files.
     luis          Manages LUIS assets on service and/or locally.
     orchestrator  Display Orchestrator CLI available commands
     plugins       Install, uninstall and show installed plugins
     qnamaker      QnA Maker
   

References:

• Responsible Bots:
• Design the user experience
• Intro to Composer

Azure OpenAI

References:

• https://learn.microsoft.com/en-us/azure/ai-services/openai/
• https://azure.microsoft.com/en-us/services/openai/
• https://learn.microsoft.com/en-us/training/paths/get-started-with-artificial-intelligence-on-azure/
• https://bomonike.github.io/genai for definition of terms and tutorials about Generative AI, which produce new content based on what is described in the input. The OpenAI models are a collection of generative AI models that can produce language, code, and images.
• https://azure.microsoft.com/pricing/details/cognitive-services/openai-service
  
  

OpenAI is a San Francisco-based artificial intelligence research laboratory. OpenAI was founded by Elon Musk, Sam Altman, Greg Brockman, and Ilya Sutskever in December 2015 (to compete with Google’s DeepMind acquisition).

In 2019 Microsoft invested $1 billion in the company as time on Azure cloud and to develop a large-scale AI supercomputer built exclusively for OpenAI’s research in Azure. Azure powers all of OpenAI’s workloads.

In 2023, OpenAI exclusively licensed (closed-source) GPT-3 to Microsoft for their products and services.

In 2022, OpenAI made available their GPT-3.5 foundation model for free trial, offering several categories of capabilities

• ChatGPT: Generating natural language: summarizing complex text for different reading levels, suggesting alternative wording for sentences, etc.
• Generating code: translating code from one programming language into another, identifying and troubleshooting bugs in code, etc.
• Generating images from text prompts, based on OpenAI’s DALL-E model.

Source: https://openai.com/blog/openai-microsoft/

OpenAI’s avowed mission is to create Artificial General Intelligence (AGI) (to rival human ability).

OpenAI is a “separate” service from Azure Cognitive Services because now “traditional” Azure Cognitive Services focuses on making predictions based textual and discrete data where OpenAI added to ML “attention” algorithms working on binary data (voice, images, and video) to enable it to perform “Generative AI” which produces new content based on what is described in the input.

1. Assuming that you have a Azure account and subscription at the Azure portal https://portal.azure.com

2. Consider pricing at https://azure.microsoft.com/pricing/details/cognitive-services/openai-service

   OpenAI put a paywall behind its more advanced GPT-4 models,

   Billing is based on 1,000 “tokens” increments, with the first 100,000 tokens per month free. Beyond that:

   “Standard” use of the older/more limited gpt-3.5-turbo model is then $0.002 per 1,000 tokens.

   Charges for the GPT-4 model has two dimensions: process stage and the size of the foundational model used :

   • The   “8K context” has   13 billion parameters.

   • The “32K context” has 175 billion parameters.

   • Prompt
   • Completions
     
     

   Images generated using DALL-E are $2 per 100 images.

   Available for free for the first 100,000 tokens per month, then $0.004 per 1,000 tokens.

3. Apply for access to Asure OpenAI for your Region and Currency:

   https://aka.ms/oaiapply

OpenAI Tutorials

• https://microsoftlearning.github.io/mslearn-openai
  • https://learn.microsoft.com/en-us/training/browse/?terms=OpenAI
  • https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/
  • https://github.com/MicrosoftLearning/mslearn-openai
  • Get started with Azure OpenAI
  • Integrate Azure OpenAI into your app
  • Utilize prompt engineering in your app
  • Generate and improve code with Azure OpenAI Service
  • Generate images with a DALL-E model
  • Use your own data with Azure OpenAI
    
    

OpenAI service features and use cases

• AOAI GPT-3.5, GPT-3.5 Turbo, GPT-4, GPT-4 Turbo, and/or Embeddings Models (Conversational AI, Search, Summarization, Writing Assistance or content generation, Code-based scenarios, Reason over Structured and Unstructured data) (a Limited Access Cognitive Service):

  • Chat and conversation interaction: Users can interact with a conversational agent that responds with responses drawn from trusted documents such as internal company documentation or tech support documentation; conversations must be limited to answering scoped questions. Available to internal, authenticated external users, and unauthenticated external users.

  • Chat and conversation creation: Users can create a conversational agent that responds with responses drawn from trusted documents such as internal company documentation or tech support documentation; conversations must be limited to answering scoped questions. Limited to internal users only.

  • Code generation or transformation scenarios: For example, converting one programming language to another, generating docstrings for functions, converting natural language to SQL. Limited to internal and authenticated external users.

  • Journalistic content: For use to create new journalistic content or to rewrite journalistic content submitted by the user as a writing aid for pre-defined topics. Users cannot use the application as a general content creation tool for all topics. May not be used to generate content for political campaigns. Limited to internal users.

  • Question-answering: Users can ask questions and receive answers from trusted source documents such as internal company documentation. The application does not generate answers ungrounded in trusted source documentation. Available to internal, authenticated external users, and unauthenticated external users.

  • Reason over structured and unstructured data: Users can analyze inputs using classification, sentiment analysis of text, or entity extraction. Examples include analyzing product feedback sentiment, analyzing support calls and transcripts, and refining text-based search with embeddings. Limited to internal and authenticated external users.

  • Search: Users can search trusted source documents such as internal company documentation. The application does not generate results ungrounded in trusted source documentation. Available to internal, authenticated external users, and unauthenticated external users.

  • Summarization: Users can submit content to be summarized for pre-defined topics built into the application and cannot use the application as an open-ended summarizer. Examples include summarization of internal company documentation, call center transcripts, technical reports, and product reviews. Limited to internal, authenticated external users, and unauthenticated external users.

  • Writing assistance on specific topics: Users can create new content or rewrite content submitted by the user as a writing aid for business content or pre-defined topics. Users can only rewrite or create content for specific business purposes or pre-defined topics and cannot use the application as a general content creation tool for all topics. Examples of business content include proposals and reports. May not be selected to generate journalistic content (for journalistic use, select the above Journalistic content use case). Limited to internal users and authenticated external users.

  • Data generation for fine-tuning: Users can use a model in Azure OpenAI to generate data which is used solely to fine-tune (i) another Azure OpenAI model, using the fine-tuning capabilities of Azure OpenAI, and/or (ii) another Azure AI custom model, using the fine-tuning capabilities of the Azure AI service. Generating data and fine-tuning models is limited to internal users only; the fine-tuned model may only be used for inferencing in the applicable Azure AI service and, for Azure OpenAI service, only for customer’s permitted use case(s) under this form.

• DALL-E 2 and/or DALL-E 3 models (text to image)

  • Accessibility Features: For use to generate imagery for visual description systems. Limited to internal users and authenticated external users.

  • Art and Design: For use to generate imagery for artistic purposes only for designs, artistic inspiration, mood boards, or design layouts. Limited to internal and authenticated external users.

  • Communication: For use to create imagery for business-related communication, documentation, essays, bulletins, blog posts, social media, or memos. This use case may not be selected to generate images for political campaigns or journalistic content (for journalistic use, see the Journalistic content use case below). Limited to internal and authenticated external users.

  • Education: For use to create imagery for enhanced or interactive learning materials, either for use in educational institutions or for professional training. Limited to internal users and authenticated external users.

  • Entertainment: For use to create imagery to enhance entertainment content such as video games, movies, TV, videos, recorded music, podcasts, audio books, or augmented or virtual reality. This use case may not be selected to generate images for political campaigns or journalistic content (for journalistic use, see the below Journalistic content use case). Limited to internal and authenticated external users.

  • Journalistic content: For use to create imagery to enhance journalistic content. May not be used to generate images for political campaigns. Limited to internal users.

  • Marketing: For use to create marketing materials for product or service media, product introductions, business promotion, or advertisements. May not be used to create personalized or targeted advertisements to individuals. This use case may not be selected to generate images for political campaigns or journalistic content (for journalistic use, see the above Journalistic content use case). Limited to internal and authenticated external users.

  Most Valuable Professional (MVP) or Regional Director (RD) Demo Use: Azure OpenAI Service DALL·E capability (in accordance with a use case listed in this Question [X]). No production use, sale, or other disposition of an application is permitted under this use case; if an MVP, RD, or their employer wants to use an Azure OpenAI Service application in production, a separate form must be submitted, the appropriate use case must be selected, and a separate eligibility determination will be made.

  • Prototyping and conceptual development: For use to generate imagery for ideation or visualization of products or services. This use case may not be selected to prototype use cases for DALL·E 2 or DALL·E 3 that do not appear in this Question [X]. Limited to internal and authenticated external users. NOTE: Consult the Azure OpenAI Service Transparency Note for image generation best practices.

• OpenAI Whisper model (Speech-to-Text)

• GPT-4 Turbo with Vision

  • Chat and conversation interaction: Users can interact with a conversational agent that responds with information drawn from trusted documentation such as internal company documentation or tech support documentation. Conversations must be limited to answering scoped questions. Available to internal, authenticated external users, and unauthenticated external users.

  • Chatbot and conversational agent creation: Users can create conversational agents that respond with information drawn from trusted documents such as internal company documentation or tech support documents. For instance, diagrams, charts, and other relevant images from technical documentation can enhance comprehension and provide more accurate responses. Conversations must be limited to answering scoped questions. Limited to internal users only.

  • Code generation or transformation scenarios: Converting one programming language to another or enabling users to generate code using natural language or visual input. For example, users can take a photo of handwritten pseudocode or diagrams illustrating a coding concept and use the application to generate code based on that. Limited to internal and authenticated external users.

  • Reason over structured and unstructured data: Users can analyze inputs using classification, sentiment analysis of text, or entity extraction. Users can provide an image alongside a text query for analysis. Limited to internal and authenticated external users.

  • Summarization: Users can submit content to be summarized for pre-defined topics built into the application and cannot use the application as an open-ended summarizer. Examples include summarization of internal company documentation, call center transcripts, technical reports, and product reviews. Limited to internal, authenticated external users, and unauthenticated external users.

  • Writing assistance on specific topics: Users can create new content or rewrite content submitted by the user as a writing aid for business content or pre-defined topics. Users can only rewrite or create content for specific business purposes or pre-defined topics and cannot use the application as a general content creation tool for all topics. Examples of business content include proposals and reports. May not be selected to generate journalistic content (for journalistic use, select the above Journalistic content use case). Limited to internal users and authenticated external users.

  • Search: Users can search for content in trusted source documents and files such as internal company documentation. The application does not generate results ungrounded in trusted source documentation. Limited to internal users only.

  • Image and Video Tagging: Users can identify and tag visual elements, including objects, living beings, scenery, and actions within an image or recorded video. Users may not attempt to use the service to identify individuals. Limited to internal users and authenticated external users.

  • Image and Video Captioning: Users can generate descriptive natural language captions for visuals. Beyond simple descriptions, the application can identify and provide textual insights about specific subjects or landmarks within images and recorded video. If shown an image of the Eiffel Tower, the system might offer a concise description or highlight intriguing facts about the monument. Generated descriptions of people may not be used to identify individuals. Limited to internal users and authenticated external users.

  • Object Detection: For use to identify the positions of individual or multiple objects in an image by providing their specific coordinates. For instance, in an image that has scattered apples, the application can identify and indicate the location of each apple. Through this application, users can obtain spatial insights regarding objects captured in images. This use case is not yet available for videos. Limited to internal users and authenticated external users.

  • Visual Question Answering: Users can ask questions about an image or video and receive contextually relevant responses. For instance, when shown a picture of a bird, one might ask, “What type of bird is this?” and receive a response like, “It’s a European robin.” The application can identify and interpret context within images and videos to answer queries. For example, if presented with an image of a crowded marketplace, users can ask, “How many people are wearing hats?” or “What fruit is the vendor selling?” and the application can provide the answers. The system may not be used to answer identifying questions about people. Limited to internal users and authenticated external users.

  • Brand and Landmark recognition: The application can be used to identify commercial brands and popular landmarks in images or videos from a preset database of thousands of global logos and landmarks. Limited to internal users and authenticated external users.

Data Science VM

https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/overview#whats-included-in-the-data-science-vm

Resources

• Azure AI product page
• Azure case studies
• Microsoft Azure Blog
• Channel 9 AI Show
• AI for Good
• Seeing AI application
• The Future Computed book

Microsoft AI ML Community in Signapore

If you have an OReilly.com account:

• Azure Cognitive Services for Developers

• Implementing Azure Cognitive Services with QnA Maker

• Implementing Azure Cognitive Services for Language

• Implementing Azure Cognitive Services for Vision

• Programming the Microsoft Bot Framework

• Machine Learning In The Cloud With Azure Machine Learning

• Azure Search for Developers

On Udemy:

• Microsoft Azure Cognitive Services Crash Course

Notes to be inserted

Steps for data transformation:

• Feature selection
• Finding and removing data outliers
• Impute missing values
• Normalize numeric features
  
  

https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/limits-and-quotas

https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/the-ai-study-guide-azure-machine-learning-edition/ba-p/4063656

https://azure.github.io/Cloud-Native/ https://github.com/azure/cloud-native is a showcase on Azure Cloud Native, the products, events and how to get started or go deep with cloud native technologies, including Serverless on Azure.

Responsible AI

LEARN:

Using Microsoft’s API algorithms and data (such as celebrity faces, landmarks, etc.) means there has be some vetting by Microsoft’s FATE (Fairness, Accountability, Transparency, and Ethics) research group in NYC: Microsoft’s ethical principles guiding the development and use of artificial intelligence with people:

• Fairness: AI systems should treat all people fairly, regardless of race, belief, gender, sexuality, or other factors.
• Reliability & Safety: AI systems should perform reliably and safely, give reliable answers with quantifiable confidence levels.
• Privacy & Security: AI systems should be secure and respect privacy - protect sensitive data and operate within applicable data protection laws.
• Inclusiveness: AI systems should empower everyone and engage people - be available to all users, regardless of their abilities.
• Transparency: AI systems should be understandable - operate understandably and openly.
• Accountability: AI systems should have algorithmic accountability - run by people who are accountable for the actions of those systems.
  
  

Resources:

• https://www.microsoft.com/ai/responsible-ai-resources
• VIDEO: Ethics an Artificial Intelligence at the DotNet Day Zurich May 28th, 2019, by Laurent Bugnion.

Mehrnoosh Sameki has a https://cs-people.bu.edu/sameki/ResponsibleAI.html course in Responsible AI

“AI Alignment” refers to unintended consequences.

VIDEO: Another OpenAI Scientist QUITS —Says AGI Is a ‘TICKING TIME BOMB’

For Azure AI Foundry Models you pay for:

• MaaP = Modela as a P? means You pay for the compute sized chosen. Microsoft offers GPU Model tuning where you can provide MaaP models to dedicated GPU Hardware such as A100 and then complete fine tuning see https://aka.ms/ignite/pre016 for examples for fine tuning with Azure.

• MaaS = Models as a Service for Tokens consumed for pay-as-you-go inference APIs and hosted fine-tuning for Llama 2 family models. Currently, there’s no extra charge for Azure AI Foundry outside of typical AI services and other Azure resource charges.

• For GPU acceleration, it’s hard to get cost effectively on Azure AI foundry. So look at the cost of Hardware for Azure GPU VM Compute costs see the Azure pricing calc for estimates https://azure.microsoft.com/en-gb/pricing/calculator/

Lee Stott wrote in Oct 2024 “The Future of AI” says:

To fine tune your own model, follow https://aka.ms/ignite/pre016 which shows you how to fine tune models using Azure AI Foundry 1 click fine tuning or Microsoft Olive Pipeline fine tuning.

but you can also use the tool to fine tune model from hugging face.

GitHub Models are available at https://github.com/marketplace/models and each model card displays the quota/capacity limit

Prompt Flow

Prompt Flow no longer exists on the ai.azure.com website when it was called “AI Studio” before 2025.

is a visual GUI tool for the (Orchestration Layer. It acts as the “glue” between services to design, test, and deploy prompt-based workflows : multi-step AI applications like chatbots, agents, or decision systems using LLMs and other services. It enables manage complex interactions between models, APIs, and user inputs.

“LLM-infused” applications are designed to understand and generate human-like text based on the input they receive. They comprise of prompts that need engineering cadence and rigour.

• https://medium.com/@mukulkumargupta/azure-ai-studio-demystified-what-to-use-and-when-to-use-it-a-beginners-guide-6ee7077adf61

AI Studio

https://learn.microsoft.com/en-us/training/modules/get-started-prompt-flow-ai-studio/?sharingId=5C4A885CF4095357&WT.mc_id=cloudskillschallenge_77j7c8trq1r462&source=learn

https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-end-to-end-llmops-with-prompt-flow?view=azureml-api-2#create-a-genaiops-prompt-flow

Microsoft calls “Prompt flow” the template and process guide used in the complete end-to-end lifecycle to build, test, optimize, and deploy (operationalize) LLM-infused applications, which Mircrosoft calls flows. Its features include Centralized Code Hosting, Lifecycle Management, Variant and Hyperparameter Experimentation, A/B Deployment, reporting for all runs and experiments, etc.

1. PROTIP: Identify the region that supports the Azure AI Foundry Prompt Flow: ???

   westus
   

2. In the Azure AI Foundry, “+ Create new” Azure AI Foundry Resource (not AI hub resource). Next.
3. Specify your aifp (Azure AI Foundry Project) name such as:

   aif-promptflow-westus-251021a
   

   https://www.youtube.com/shorts/i7TW2iDJCsg

4. Define a folder to receive the cloned repo.
5. For local execution of LLMOps flows for experimentation and evaluation locally:

   git clone https://github.com/microsoft/llmops-promptflow-template.git --depth 1
   

   Alternately, to centrally host code for multiple prompt flows: Generative Artificial Intelligence Operations (GenAIOps):

   git clone https://github.com/microsoft/genaiops-promptflow-template.git --depth 1
   

6. Select a template to support either the older Azure Machine Learning or Azure AI Studio.

   Templates support multiple flows, each with its objectives, requirements, and complexities:

   • Local invocation and execution of flows for experimentation and evaluation locally.
   • Local invocation but execution of flows for experimentation and evaluation on Azure.
   • Pipeline based invocation but execution of flows for experimentation and evaluation on Azure.
   • Pipeline based invocation and execution of flows for experimentation and evaluation on Build Server.
   • Deployment of all types of flows to Kubernetes, Azure Web Apps and AML/AI Studio Managed compute.
     
     
7. Create an .env file at the top folder level. Add lines for each connection, updating the values for the placeholders. The examples in the example repo use the AzureOpenAI connection named aoai and API version 2024-02-01. ``` aoai={ “api_key”: “","api_base": "","api_type": "azure","api_version": "2024-02-01"}