Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Azure Monitoring

How to get logs and metrics ingested and displayed with actionable alerts

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Several Azure services are related to all the monitoring happening within Azure:

Changes in terminology

Dynatrace (for additional price) provides comprehensive monitoring support for Azure services, by integration with both OneAgent and Azure Monitor.

References:

PRICING for monitoring

VIDEO: “What to use for monitoring your applications in Azure”</a> by Azure Barry (Luijbregts)

Issues to monitor for:

  • poor response times,
  • changing usage rates,
  • exceptions,
  • security risks,
  • capacity limits,
  • suspicious activity.

Service Health

  1. In Portal GUI Service Health
  2. Health alerts, “+ Add Service health alert”

  3. “Add Action Groups”, Subscription, “+ Create action group”

    Action Types:

    • Automation Runbook
    • Azure Function
    • ITSM
    • Logic App
    • Secure Webhook
    • Webhook

Azure Monitor

PRICING

  1. The Azure Monitor service home screen is at https://docs.microsoft.com/en-us/azure/azure-monitor/overview

  2. Get to Monitor service from the Home menu, Search, or Favorites.

    You can return to this page by clicking “Overview” in the Monitor menu.

  3. Click the “What’s new” tab.

    PROTIP: These can be viewed at https://aka.ms/AzMonUpdates - Azure Monitoring Updates

    Between new product announcements occur in March each year at Microsoft’s Ignite conference, Kayode Prince videos about Azure Monitor.

    Theoretical arrangement

    VIDEO What is Azure Monitor? from Microsoft: az-mon-hybrid-1534x787

    DEFINITION: Azure Monitor collects two fundamental types of data:

    Insights are elicited from:

    • Applications
    • Virtual Machines (IaaS) which uses the Azure Diagnostics Extension
    • Storage accounts
    • Containers

    • Networks

    • Azure Cosmos DB
    • Key Vaults
    • Azure Cache for Redis
    • Insights Hub

    Developers can code .NET Framework, Node.js, or Python to call the Data Collector API to send data to Azure Monitor from custom sources such as a web app, an Azure function, or a mobile app.

  4. To Analyze:

    • Metrics Explorer analyzes collected metrics on a chart and compare metrics from different resources.
    • Log Analytics
    • Export: route Metrics to Logs to analyze data in Azure Monitor Metrics together with data in Azure Monitor Logs and to store metric values for longer than 93 days.

  5. To Visualize:

    • pin a chart from metrics explorer to an Azure Dashboard. Export the results of a query to Grafana to leverage its dashboarding and combined with other data sources. VIDEO

  6. To Respond:
    • in Alerts, configure metric alert rules to send notifications or takes automated action when some metric value crosses a threshold.
    • use Autoscale to increase or decrease resources based on a metric value crossing a threshold. Automate!
    • Action Groups.

  7. To Integrate:
    • Stream Metrics to an Event Hub to route them to external systems.
    • Logic apps
    • Ingest & Export APIs

Metric values can be accessed:

  • from a command line using PowerShell cmdlets
  • from custom application using REST API.
  • from a command line using CLI.

Also, archive performance / health history of resources for compliance, auditing, or offline reporting.

??? The agent for Linux and Windows isn’t only for connecting to Azure Monitor, it also supports Azure Automation hosts the Hybrid Runbook worker role and other services such as Change Tracking, Update Management, and Azure Security Center.

Logs

Live Log Streaming

  1. Start the log streaming service to show a duplicate of what is saved to log files for a single app instance. So it’s only good for initial debugging, to quick feedback on server issues.

    az webapp log tail --name $app_name --resource-group $resource_group_name

  2. To stop viewing live logs, press Ctrl+C.

HANDS-ON Exercise in Sandbox - Create basic Azure Monitor log queries to extract information from log data

  1. https://portal.azure.com/#blade/Microsoft_Azure_Monitoring_Logs/DemoLogsBlade?azure-portal=true

  2. Type a KQL to take the last 10 records from SecurityEvent logs. Click Run:

    SecurityEvent
 | take 10

    Logs are organized into records. Each type of record have different sets of properties. Logs typically contain text data with detailed descriptions that contain numeric values.

    • Time Generated is for the UTC time zone
    • Account
    • Account Type is Machine or User
    • Computer
    • Event Source Name
    • Channel: “Security”,”Microsoft-Windows-AppLocker/…”
    • Task
    • Level: “4”, “8”, “16”
    • EventData
    • EventID
    • Activity
    • AuthenticationPackageName
    • CommandLine
    • ElevatedToken
    • FileHash
    • FilePath
    • Fqbn (Fully qualified b name)
    • ImpersonationLevel “%%1833” is for English
    • IpAddress
    • IpPort “58710”

    Logs differ from metrics in that their record structure can vary and are often not collected at regular intervals. Applications can create custom logs by using the structure each needs.

  3. Look at events collected sporadically.

    Event
| where EventLog == "Application"
| where TimeGenerated > ago(24h)

    Events created by an application or service typically include enough information to provide complete context on their own. For example, an event can indicate that a particular resource was created or modified, a new host started in response to increased traffic, or an error was detected in an application.

    Heartbeats

  4. Display heartbeat events generated by computers, rendered as a bar chart each week (for the last three weeks):

    Heartbeat
 | where TimeGenerated >= startofweek(ago(21d))
 | summarize dcount(Computer) by endofweek(TimeGenerated) | render barchart kind=default

    Telemetry such as events and traces are stored as logs in addition to performance data so that thet can all be combined for analysis.

  5. From the Azure Portal menu, select “Monitor”.

  6. PROTIP: Click the “«” icon above the Portal menu to make room.

  7. Click “Logs” in the Log Queries menu.

  8. PROTIP: Click the “«” icon above the Monitor menu to make room.

    Observe that specific log queries (to run) on the right are grouped according to the left menu: Access -> ACCESS, Activity log -> ACTIVITY LOG, etc.

  9. Click to open the Topics menu, which groups specific log queries differently (in 4 dimensions):
    • Category (of architecture)
    • Resource Type
    • Solution
    • Topic

  10. Click “Community Git repo” on the upper-right for a new browser tab showing:

    https://github.com/microsoft/AzureMonitorCommunity

    Azure Workbooks

    Workbooks provide a flexible canvas for data analysis. It combines multiple sources/sets of data in an interactive report (visualization).

    • Logs
    • Metrics
    • Azure Resource Graph
    • Alerts
    • Workload health
    • Azure Resource Health
    • Azure Data Explorer

    See the Gallery:

    Workbooks support visualization types: Text, Charts, Grids, Tiles, Trees, Graphs

    Start with a template.

    Azure App Insights

    When an app is connected

    installs an instrumentation package in your app

    instrumentation key

    Application Insights Dashboard

    SDK analyzes your app for a typology, to identify connection to resources.

    Tracks availability of app

    KQL folders

    Folders in the repo have different items than in groupings Portal GUI:

    SameIn Portal GUIIn repo
    * Solution(s) - see comparion table below * Category (of architecture)
    * Resource Type
    * Solution
    * Topic     		* Azure Services
    * Scenarios (How to's)
    * Solutions

    Solution(s):

    SameIn Portal GUIIn repo
    * Change Tracking
    * ContainerInsights
    * LogManagement
    		 * Active Directory health check
    * Azure Monitor for VMs
    * Network Performance Monitor
    * Security and Audit
    * SecurityCenterFree
    * ServiceMap
    * Update Management
    * Custom Logs
    * Functions
    		 * ADAAssessment
    * DNSAnalytics
    * LogicAppB2B
    * SAP-SCP-Monitoring
    * SQLAssessment
    * SecurityInsights
    * ServiceDesk
    * SurfaceHub
    * WaaSUpdateInsights
    * WireData2

  11. In the repo README file, click “Log Analytics Demo environment” to open a new Portal tab:

    https://portal.loganalytics.io/demo

  12. Click “Query explorer” tab at the right.

  13. Expand “Solution Queries”:

    • Active Directory Health Check
    • Alert Management
    • Antimalware Assessment
    • Change Tracking
    • Security and Audit
    • SQL Health Check
    • Update Management
    • Wire data

  14. Expand “Favorites”:

    • All Syslog records with errors
    • Available memory (MB) per hour for computers that are named “Contoso*”
    • Find out which computers haven’t sent any data in the past 12 hours
    • Security events count by Computer during the last 12 hours
    • See the stream of data collected in the last 24 hour in intervals of one hour
    • Usage data volume per data type
    • Which computers sent a heartbeat in the last hour, and when they last sent it

https://jmespath.org/tutorial.html

Log Analytics (LA)

VIDEO, VIDEO: How to get started with Azure Monitor Log Analytics

Log Analytics reads data sources to create virtual tables in workspaces for use by KQL queries.

azure-log-analytics-711x306-35708

How you start Log Analytics within Azure Portal limits the scope of data available: ???

  • From Log Analytics workspaces menu
  • From an Application Insights application Overview page, select “Analytics”.
  • From an Azure resource menu, select “Logs”.

LA Workspaces

  1. In Portal search for “Log Analytics workspaces”.
    • Create (previously Add) an instance.
  3. Specify the resource group you defined earlier.
  4. Type name based on your Naming Guidelines.
  5. Next: Pricing

  6. Review + Create, Create.

    “OMS” (Operations Management Suite) is legacy branding which hasn’t been updated.

  7. Go to Resource.
  8. Click “«” at the top of the Portal menu to hide it for more room.
  9. Scroll down to “Workspace Data Sources” -> Virtual machines.
  10. Click a VM running.

Log Analytics

TASK: Connect Activity Logs from various Subscriptions to a Azure Montitor Logs (prev. aka Log Analytics) for up to 90 days.

  1. From the “Monitor” menu, select “Logs” for “Log Analytics”.

    Log Analytics monitors cloud and on-premises environments (to maintain availability and performance).

  • Azure Monitor (pane of glass for monitoring on Azure),
  • Azure Log Analytics (log ingestion and IaaS monitoring), and
  • Application Insights (application performance monitoring including availability, performance, and exception information)

  1. Click Monitor “Usage and estimated costs” : https://portal.loganalytics.io/demo

    ??? In each workspace, Log Analytics collects data from connected sources by configuring data sources and adding solutions to your subscription. ???

    Each workspace has its own data repository and configuration, and data sources and solutions are configured to store their data in a particular workspace.

    A Log Analytics workspace collects data from:

    • Azure resources under a subscription
    • On-premises computers monitored by System Center Operations Manager
    • Configuration Manager Device collections
    • Azure storage Diagnostics log data

    Data sources and solutions each create different record types, each with its own set of properties. But you can still analyze sources and solutions together in queries to the workspace. This capability allows you to use the same tools and methods to work with a variety of data collected by a variety of sources.

Azure Traffic Analytics

https://medium.com/microsoftazure/automation-to-block-malicious-flows-detected-by-azure-traffic-analytics-b010298ba347

Network Performance Monitor (NPM)

VIDEO:

  1. ”+ Create a resource”.
  2. In Search in the Marketplace, type enough of “Network Performance Monitor” to select it. Notice it’s “Solarwinds”.
  3. Click the blue Create.

  4. Create new Resource Group and define a VM server.

    …

  5. Get connected to a valid workspace.

  6. In a Log Analytics workspace, click General: Solutions.

  7. Download Windows Agent into each subnet.

    … synthetic transactions

  8. The full list of columns in table “NetworkMonitoring” is

    https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/NetworkMonitoring

Azure Resource Graph Explorer

  1. Click on “All Resources” powers

  2. Click “Open Query” at the top for a dialog to specify Kusto Query Language (KQL) for resource discovery and inventory.

    “Graph” refers to the relationship between resource entities, not visual graphics.

Kusto Query Language (KQL)

See my Kusto

Metrics (performance data)

Some metric data can be stored in Logs to combine them with other monitoring data for trending and other data analysis.

Service Health metrics

Dashboard for Metrics

  1. Click “Monitor” among Portal FAVORITES.

  2. Click “Dashboard” at the top.

    Note Dashboard display resources.

Application Insights

VIDEO:

Application Insights is an “APM” (Application Performance Management) service: it collects log, performance, and error data.

An Application Insights resource is created by default when a Function app is created.

https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview

App Insights have additional costs: https://portal.loganalytics.io/demo

Diagnostics

Types of diagnostic logs:

  • Tenant logs come from tenant-level services such as Azure Active Directory (Azure AD) which exist outside Azure subscriptions and their resources.

  • Resource-level logs come from Azure services under a subscription’s resources, such as rule counters by NSGs (Network Security Groups), Azure Key Vault audits, and storage accounts. Resource-level diagnostic logs provide insight into operations that were performed within that resource itself.

??? Diagnostic logs differ from activity logs . Activity logs provide insight into operations such as creating a VM or deleting a logic app, that Azure Resource Manager performed on resources in your subscription using. The activity log is a subscription-level log.

??? These logs also differ from guest operating system (OS)–level diagnostic logs. Guest OS diagnostic logs are those collected by an agent running inside a VM or other supported resource type. Resource-level diagnostic logs require no agent and capture resource-specific data from the Azure platform itself, whereas guest OS–level diagnostic logs capture data from the OS and applications running on a VM.

Monitor

Diagnostic Log Settings can be configured by:

  • Azure Portal GUI
  • PowerShell
  • Azure CLI
  • Rest API calls

Configuration:

  • Save Diagnostic logs to a storage account for auditing or manual inspection. Specify retention time (in days).

  • Stream Diagnostic logs to Event Hubs for ingestion by a third-party service or custom analytics solution, such as Power BI. An event hub is created in the namespace for each log category you enable. A diagnostic log category is a type of log that a resource may collect.

  • Analyze them with Azure Monitor, so they are immediately written to Azure Monitor with no need to first write the data to storage.

??? Logs are streamed to:

  • Azure Storage
  • Log Analytics workspaces
  • Azure Monitor Logs (previously called “”).
  • Event Hubs
  • Export

PowerShell: Set-AZDiagnosticSetting -ResourceID $ -WorkspaceID $ -Categories $list -Enabled $true

VIDEO: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/tutorial-monitoring Install Diagnostics extension -> Enable guest-level monitoring (into a storage account)

References:

Log alerts can come from any Azure resource:

  • server logs,
  • application server logs,
  • App logs are configured in “Diagnostics logs” menu, the output of pre-production runtime trace statements in app code, such as console.log(“Message”) and console.error(“Message”) to STDOUT an STDERR from JavaScript on Windows. On Linux machines, only errors (not blobs) are sent. Logging to the file system will be automatically reset to Off after 12 hours.
az webapp log config --application-logging true --level verbose \
      --name $app-name \
      --resource-group $resource-group-name

There is currently no way to disable application logging by using Azure CLI commands; however, the following command resets file system logging to error-level only.

az webapp log config –application-logging false –name &LT;app-name> –resource-group

To view the current logging status for an app, use this command.

az webapp log show –name &LT;app-name> –resource-group &LT;resource-group-name>

HANDS-ON LAB: Enable and Configure App Service Application Logging using the Azure Portal

gitRepo=https://github.com/MicrosoftDocs/mslearn-capture-application-logs-app-service Based on:

  • Tests for web site availability
  • Metric values threshold for metric alerts
  • Log search values
  • Activity Log events
  • Health
  • Heartbeats

Alerts

Alerts

FREE HANDS-ON LAB: Use metric alerts to alert on performance issues with an Azure environment resource-group shown as “MICROSOFT LEARN SANDBOX” subscription paid by Microsoft at https://portal.azure.com/learn.docs.microsoft.com.

  1. In the Portal Azure Cloud Shell, define custom-data in a file which generates enough stress on the CPU (1=100% usage) which trigger an alert:

    cat <<EOF > stress-cpu1.txt
#cloud-config
package_upgrade: true
packages:
       - stress
runcmd:
       - sudo stress --cpu 1
EOF

  2. Set up an Ubuntu Linux VM with a custom-data configuration file:

    az vm create \
 --resource-group learn-af57b7f1-6591-4a7d-8880-6f5db0c162d5 \
 --name vm1 \
 --image UbuntuLTS \
 --custom-data stress-cpu1.txt \
 --generate-ssh-keys

    Output:

    “SSH key files ‘/home/wilsonmar/.ssh/id_rsa’ and ‘/home/wilsonmar/.ssh/id_rsa.pub’ have been generated under ~/.ssh to allow SSH access to the VM. If using machines without permanent storage, back up your keys to a safe location.

  3. In Monitoring Overview, “+ New alert rule”.

  4. Under Scope, select Select resource. The Select a resource pane appears. The Filter by subscription dropdown list should already be populated with Concierge Subscription. In the Filter by resource type enough of “Virtual machines” to select it. Select the vm1 virtual machine under resource group:

    learn-3328fcbd-8b2a-4c63-9407-dea2ec4db84e

  5. Select Done at the bottom of the pane.

  6. Under the Condition (conditional logic) section, select Add condition for the Configure signal logic pane to appear. For Signal type, type “m” to select Metrics. For Monitor service, select All.

  7. In the “Search by signal name”, type list of available signals will change depending on the selected signal type. From the list of available signal types, select Percentage CPU.

  8. In the Configure signal logic pane, enter the following values for each setting.

    …

  9. Select Done.

…

Every log alert has an associated search rule. The composition of these rules is:

  • Log query: Query that runs every time the alert rule fires.
  • Time period: Time range for the query.
  • Frequency: How often the query should run.
  • Threshold: Trigger point for an alert to be created.

Actions are:

  • Send an email.
  • Send an SMS message.
  • Create an Azure app push notification.
  • Make a voice call to a number.
  • Call an Azure function.
  • Trigger a logic app.
  • Send a notification to a webhook.
  • Create an ITSM ticket. (Service Now)
  • Use a runbook (to restart a VM, or scale a VM up or down).

Two places

  • Blade
  • Menu: alert

  1. RESOURCE
  2. CONDITION: Configure signal logic (124 signals) : Signal Type Activity Log iconActivity Log
  3. Monitor service: Administrative, etc.
  4. Alert Level: All, Critical, Error, Warning, Informational, Verbose
  5. Status: Failed, Started, Succeeded
  6. ACTION Group Type: REMEMBER sets of notification preferences – VIDEO
    • Email/SMS/Push/Voice
    • Azure Function
    • LogicApp
    • Webhook
    • ITSM
    • Automation Runbook

Emails from:

  • azure-noreply@microsoft.com
  • azureemail-noreply@microsoft.com
  • alerts-noreply@mail.windowsazure.com

Severity Level 0 lowest, 4 highest

Application Insights

Alerts can be Enabled or Disabled at any time.

Learn Module: Handle transient errors in your app

Event Hubs

???

Analyzing

VIDEO

  • Analyze using Metrics Explorer
  • Visualize
  • Alert
  • Automate using AutoScale
  • Export
  • Retrieve
  • Archive

DDoS attacks

Other

Within Azure Monitor using Azure Log Analytics,

Pipe:

    Event | search "error" | take 10
    or 
    search in (Event) "error"
    Heartbeat 
   | summarize LastHeartbeat-max(TimeGenerated) by Computer
   | where LastHeartbeat < ago(5h)

CAUTION: Kusto language keywords are case-sensitive

Cost Management and Billing

https://raw.githubusercontent.com/benc-uk/icon-collection/master/azure-cds/migrate-285-Cost-Management-and-Billing.svg

https://uxbooth.com/articles/introduction-to-taxonomies

Tags

VIDEO

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/tag-resources Use tags to organize your Azure resources

(Get-AzResourceGroup -Name examplegroup).tags

RBAC

VIDEO

https://www.youtube.com/watch?v=Zr7LcSr6Ooo What to use for monitoring your applications in Azure | Azure Friday

https://myownpicloud.wordpress.com/2021/02/23/monitor-aks-infrastructure-using-prometheus-and-grafana/

Azure Security Center

Azure Security Center is a CSPM (Cloud Security Posture Management) solution. It lists prioritized security alerts and recommendations for attack investigation and remediation. It summarizes a tenant’s security posture with a “Secure Score” s based on the percentage of recommendations implemented.

DEFINITION: “Security posture” refers to cybersecurity policies and controls, as well as how well you can predict, prevent, and respond to security threats.

VIDEO DEMO LEARN

Host recommendations: Every VM includes vulnerability assessment from Qualys.

  • OS security settings configuration rules
  • System security & critical updates missing
  • Endpoint protection recommendations
  • Disk encryption validation
  • Remediate vulnerabilities assessment
  • Threat detection

Each can be exempted.

### Install agent

From the network; and connected partner solutions, like firewall and endpoint protection solutions, Security Center automatically collects, analyzes, and integrates log data from Azure resources to detect real threats and reduce false positives.

  1. Click “Getting Started” menu. Click “Install agent” for data collection on VMs.
  2. Click “Install agents” button to install on all VMs.

  3. “Pricing & settings” menu

    “Standard” tier includes threat protection, network hardening and VM EDR.

  4. Scroll down to see each resource can be enabled or disabled for security.

    az-mon-sec-pricing

    Each machine is $15/month.

  5. Data Collection

    By default, Azure Security Center stores data that it collects from agents in a Log Analytics workspace where it can be analyzed with other log data.

    Processed events that Azure Security Center produces are published to the Azure activity log, one of the log types available through Azure Monitor.

  6. “Continuous export”

    To stream log data from Azure Monitor to a SIEM tool (Azure Sentinel or Splunk), use Azure Event Hubs - a streaming platform and event ingestion service that transforms and stores data by using any real-time analytics provider or batching/storage adapters.

    Azure Monitor offers a consolidated pipeline for routing any of your monitoring data into a SIEM tool. This is done by streaming that data to an event hub, where it can then be pulled into a partner tool. This pipe uses the Azure Monitor single pipeline for getting access to the monitoring data from your Azure environment. This allows you to easily set up SIEMs and monitoring tools to consume the data. Currently, the exposed security data from Azure Security Center to a SIEM consists of security alerts.”

    Security Center “Adaptive network hardening” are recommendations about how NSGs should be locked down with remediation steps.

    Adaptive application controls which applications uses machine learning to alert about unauthorized applications that are running on its VMs, by creating exception rules for each resource group that holds the VMs.

    See code at https://github.com/ned1313/Monitor-Security-with-Azure-Security-Center

Azure Defender CWPP

Azure Defender is Security Center’s integrated Cloud Workload Protection Platform (CWPP). In addition to the built-in policies, custom policies and initiatives can be added – regulatory standards such as NIST and Azure CIS as well as the Azure Security Benchmark.

INTRO VIDEO at Ignite 2019 by Yinon Costica (@c0stica), Director PM, Cloud Security Group.

Several 3rd-party vendors also provide CWPP with recommendations:

  • https://securityboulevard.com/2021/01/a-quick-look-into-cloud-workload-protection-platforms-cwpp/
  • https://start.paloaltonetworks.com/gartner-market-guide-cwpp.html
  • https://www.zscaler.com/resources/security-terms-glossary/what-is-gartner-cwpp
  • https://blog.aquasec.com/gartner-cloud-workload-protection-platforms
  • https://www.cloudvisory.com/cspm-cwpp-solution.html
  • https://www.optiv.com/cybersecurity-dictionary/cwpp-cloud-workload-protection-platform
  • https://www.mcafee.com/enterprise/en-us/security-awareness/cloud/what-is-a-cwpp.html

CWPP

CWPP (Cloud Workload Protection Platform) aims to block the “Kill Chain” - see my notes on Kill Chain.

az-mon-threats-1015x426

PROTIP: Don’t set permissons for Local Admin on a laptop.

To protect containers in AKS:

az-protect-container-mining-1786x847

Security Center social

user Forum, Blog, Feature suggestion, documentation, API documentation

References:

  • https://docs.microsoft.com/en-us/azure/security-center/security-center-alerts-overview Security alerts and incidents in Azure Security CenterLearn how Azure Security Center generates security alerts and correlates them into incidents.docs.microsoft.com​[12:42 PM] Ken Lawson

  • https://docs.microsoft.com/en-us/azure/security-center/security-center-alerts-overview Security alerts and incidents in Azure Security CenterLearn how Azure Security Center generates security alerts and correlates them into incidents.docs.microsoft.com​[12:45 PM] Ken Lawson

  • https://docs.microsoft.com/en-us/azure/sentinel/create-incidents-from-alerts Create incidents from alerts in Azure SentinelLearn how to create incidents from alerts in Azure Sentinel.docs.microsoft.com​[12:46 PM] Ken Lawson

  • https://docs.microsoft.com/en-us/azure/logic-apps/quickstart-create-first-logic-app-workflow Quickstart - Create your first Logic Apps workflow - Azure portal - Azure Logic AppsBuild your first automated Logic Apps workflow in the Azure portal using this quickstart guide. Learn the basics of system integration and enterprise application integration (EAI) solutions in Logi…docs.microsoft.com​[12:>”X?51 PM] Ken Lawson

Just-In-Time VM access

VIDEO:

To reduce the attack surface, protect network ports by opening network traffic to VMs only during time of need:

  1. In Security Center, Configured tab, click the checkbox for a Virtual machine;
  2. Click “Request access” button.

  3. Toggle On/Off and select amount in Time range to allow.

    az-req-net-jit-1716x414

Azure Sentinel (SIEM)

BOOK:

DEPRECATION: Custom alert rules were retired from Azure Security Center on June 30, 2019 so they can be in the new Azure Sentinel product.

Azure Sentinal competes with Splunk and others to provide a SIEM (Security Information and Event Management) solution that collects monitor data to provide a centralized analysis and visualization for SOC (Security Operations Center).

Sentinal is called a SOAR (Security Orchestration Automated Response) solution because ???

Azure Sentinel stores data from data sources into a Log Analytics workspace.

Charges for Azure Sentinel are based on a fixed Capacity Reservation “commitment tiers” (in Log Analytics), from $123 per day for increments of 100GB, with overage beyond $2.46 per GB-ingested. WestUS costs 30% more than EastUS.

PRICING *

To onboard Azure Sentinel:

  1. Enable Azure Sentinel in Portal.

  2. Define subscriptions handled by each playbook.

    A Sentinel playbook is a collection of procedures that can be run from Azure Sentinel in response to an alert. Each Sentinel playbook can handle several subscriptions at once.

  3. Connect data sources.

    Investigation graph for visualizing and traversing the connections between entities like users, assets, applications, or URLs and related activities like logins, data transfers, or application usage to rapidly understand the scope and impact of an incident.

    Azure Sentinel comes with a number of connectors for Microsoft solutions, available out of the box and providing real-time integration, including Microsoft Threat Protection solutions, Microsoft 365 sources, including Microsoft 365, Azure AD, Azure ATP, Microsoft Cloud App Security, etc.

    In addition, there are built-in connectors to the broader security ecosystem for non-Microsoft solutions. You can also use common event format, Syslog or REST-API to connect your data sources with Azure Sentinel.

    Microsoft’s security researchers built the Azure Sentinel GitHub repository at https://github.com/Azure/Azure-Sentinel to cover 400 detection, exploratory, and hunting queries, plus Jupyter Notebooks samples and related Python libraries, playbooks samples, and parsers.

  4. Select from a gallery of dashboards to surface insights based on custom data.

  5. Customize your dashboard.

    ??? Incident detailed information includes severity, summary of the number of entities involved, the raw events that triggered this incident, and the incident’s unique ID.

  6. Analyze alerts

    Alerts can be triggered by a single event, be based on a threshold, by correlating different datasets, or by using built-in machine learning algorithms.

    Azure Sentinal makes use of AI Machine Learning.

    Azure Sentinel has more than 100 built-in alert rules, or you can create your own.

  7. Define a notebook.

    A notebook is a step-by-step playbook where one can walk through the steps of an investigation and hunt.

  8. Assign owner to incidents.

    Sentinel built-in roles are reader, responder, and contributor.

    All incidents start as unassigned. Add comments so that other analysts will be able to understand what was investigated and what concerns are around the incident.

Linking O365 to Log Analytics - https://docs.microsoft.com/en-us/azure/azure-monitor/insights/solution-office-365 Office 365 management solution in Azure - Azure MonitorThis article provides details on configuration and use of the Office 365 solution in Azure. It includes detailed description of the Office 365 records created in Azure Monitor.docs.microsoft.com​

Linking O365 to Sentinel - https://docs.microsoft.com/en-us/azure/sentinel/connect-office-365 Connect Office 365 logs to Azure SentinelLearn to use the Office 365 log connector to bring in information about ongoing user and admin activities in Exchange, Teams, and SharePoint, including OneDrive.docs.microsoft.com​

https://techcommunity.microsoft.com/t5/azure-sentinel/best-practices-for-designing-an-azure-sentinel-or-azure-security/ba-p/832574 Best practices for designing an Azure Sentinel or Azure Security Center Log Analytics workspace Note: alot has be updated since this article: we now have official guidelines in the documentation: Extend Azure Sentinel across workspaces and tenants. You may also want to review the Webinar on…techcommunity.microsoft.com​

https://techcommunity.microsoft.com/t5/azure-sentinel/become-an-azure-sentinel-ninja-the-complete-level-400-training/ba-p/1246310

References:

Azure ML

The JuPyter notebooks link to an Azure ML workspace ​
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-run-jupyter-notebooks#:~:text=How%20to%20run%20Jupyter%20Notebooks%20in%20your%20workspace,experiment.%20…%205%20Change%20the%20notebook%20environment. Run Jupyter notebooks in your workspace - Azure Machine LearningLearn how run a Jupyter notebook without leaving your workspace in Azure Machine Learning studio.docs.microsoft.com

Microsoft Defender

  • Identity
  • Endpoint forensic analyzes communication patterns
  • Cloud App Security (Caspia???) analyzes apps, conditional access proxies (DLP?). Was ATP

security.microsoft.com Office 365 Security Centerintegrates “Microsoft Defender for Endpoint” and “Microsoft Defender for Office 365”. It provides eXtended Detection and Response (XDR): incident management, automatic investigation and remediation, Microsoft Threat Experts, threat analytics, and cross-domain proactive hunting. Phishing protection. Provides a Secure Score summary metric; email threat alert policies.

Microsoft InTune policy engine manages Windows 10 & macOS client (device) security. Includes MDM (Mobile Device Management) can wipe entire device. MAM (Application) with app allow/deny policies.

compliance.microsoft.com has data classification and data connectors shown a Compliance Manager by Solution:

  • Catalog (of risks)
  • Audit (can go to 10 years with additional licensing)
  • Content search
  • Communication compliance (profanity, etc.)
  • Data loss prevention (DLP) exfiltration, watermarkx
  • Data subject requests
  • eDiscovery (hold)
  • Information governance
  • Information protection
  • Insider risk management
  • Records management

Privilege Access Management - task scope

Customer Lockbox in Office 365 to transfer files to Microsoft engineers.

Twitter @MSThreatProtect (integrated SIEM and XDR), @MsftSecIntel (Microsoft Security Intelligence)

Stay Up to Date

From Microsoft:

Azure Monitor: The essentials every admin should know by KnowOps

https://www.youtube.com/watch?v=-aMecR2Nrfc&list=PLLasX02E8BPCCsHzNLJjcElCwF52rnh6t

Microsoft’s YouTube channel for Azure Monitoring

VIDEO COURSE: Microsoft Azure IaaS Monitoring & Management

SCOM (System Center Operations Manager) monitoring tool have management packs for managing specific products

https://www.linkedin.com/learning/search?keywords=exam%20az-500%3A%20microsoft%20azure%20security%20technologies&u=3322

Monitor alerts - Actionable or noise happens..

Sev (Severity) levels: Sev 0 = Critical
Sev 1 = Error
Sev 2 = Warning
Sev 3 = Informational
Sev 4 = Verbose

Metrics Explorer

Analyze collected data using Metrics Explorer for charting and visual correlation and Log Analytics for ad-hoc queries, trending, and pattern recognition.

Azure Monitor allows you to manage and create alerts, notifications, and actions such as runbooks and autoscale based on metrics and logs.

Integrate Azure Monitor with other tools using Event Hubs to export data or APIs for ingestion and export.

Metrics Advisor

VIDEO: Metrics Advisor comes up with anomaly detection without you having to setup Machine Learning.

Load Testing Azure

https://k6.io/blog/k6-as-alternative-for-azure-and-visual-studio-load-tests/

References

HIGHLY RECOMMENDED: LAB: Become an Azure Security Center Ninja by Yuri Diogenes, Principal Program Manager at C+AI Security CxE Team at Microsoft

  • Microsoft CxE (Compliance and E): Protect, Detect, Defend

https://docs.microsoft.com/en-us/office365/servicedescriptions/office-365-platform-service-description/service-health-and-continuity

LEARN: Design a holistic monitoring strategy on Azure

Azure Metrics Advisor client library samples for Java

Social

https://github.com/Azure/Azure-Security-Center by Azure’s largest customers contains best practices and tools

More on Azure

This is one of a series on Azure:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API

  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow

  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options

  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)

  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications

  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)

  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute

  31. Azure Monitoring

  32. Digital Ocean

  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation

  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS

  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs

  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps

  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators

  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling

  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft

  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering

User Defined Routes and NVA (Network V A)

contosofashions25127.azurewebsites.net

Competitors

Alternatives to monitoring:

  • Dynatrace
  • New Relic
  • Naggios
  • Zabbix

More about Azure

This is one of a series about Azure cloud:

Azure Monitoring was published on .