Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Reaching to web servies on the web gives scripts data power

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Being able to get and send data within a PowerShell script enables them to be NOT static. Communication with APIs enable PowerShell scripts to:

  • Get input data

  • Send SMS messages and voicemails to any phone
  • Send emails
  • Send calendar appointments

There are now two major flavors of how requests and responses are formatted:

PowerShell has two commands to make web services calls:


Invoke-RestMethod documentation

Documentation on Invoke-RestMethod says the cmdlet was introduced in PS 3.0 to send HTTP and HTTPS requests to Representational State Transfer (REST) web services that returns richly structured data. No short alias is specified for it.

Its general syntax:

Invoke-RestMethod 
[-Uri] <Uri> 
[-Headers <IDictionary> ] 
[-Body <Object> ] 
[-Certificate <X509Certificate> ] [-CertificateThumbprint <String> ] 
[-ContentType <String> ] 
[-Credential <PSCredential> ] 
[-DisableKeepAlive] 
[-InFile <String> ] 
[-MaximumRedirection <Int32> ] 
[-Method <WebRequestMethod> {Default | Get | Head | Post | Put | Delete | Trace | Options | Merge | Patch} ] [-OutFile <String> ] 
[-PassThru] 
[-Proxy <Uri> ] [-ProxyCredential <PSCredential> ] [-ProxyUseDefaultCredentials] 
[-SessionVariable <String> ] [-TimeoutSec <Int32> ] 
[-TransferEncoding <String> {chunked | compress | deflate | gzip | identity} ] [-UseBasicParsing] [-UseDefaultCredentials] 
[-UserAgent <String> ] 
[-WebSession <WebRequestSession> ] 
[ <CommonParameters>]

HTTP requests have both a GET and POST approach.

GET Invoke-RestMethod

Web services which do not require some registration is getting more rare nowadays.

But in this blog, Jeff Hicks found that NewEgg still has an RSS feed for their daily deals at http://www.newegg.com/RSS/Index.aspx

  1. Run a basic HTTP GET in a PowerShell script containing:

    
    $uri = "http://www.newegg.com/Product/RSS.aspx?Submit=RSSDailyDeals&Depa=0"
    $response = Invoke-RestMethod -Uri $uri
    $response.title
    $response.count
    $response[0] | format-list
    

    BTW: Although “uri” means Universal Resource Identifier, to be technically correct, the inclusion of the access mechanism “http” makes it really is a URL, a type of URI.

    PROTIP: When output to a variable, show a count of how many items were returned into the response variable so that it doesn’t look like nothing happened.

    The response is overwhelming, so format-list is used to filter out just the first item returned in the response

    PROTIP: Lists starts from zero.

    This enables us to see the names of properties: Published, title, link

    103
    Published : 11/3/16 4:05:46 AM
    title     : $199.99 - SHIELD Series RSCM-0916B081 - 16-Channel 960H, 
             H.264-Level DVR Surveillance Kit + Eight 900TVL Cameras - Night 
             Vision Up to 65 Feet, Remote Viewing Supported (HDD Not Included)
    link      : http://www.newegg.com/Product/Product.aspx?Item=N82E16881147042&nm_
             mc=OTC-RSS&cm_sp=OTC-RSS-_-Surveillance%20Security%20Systems-_-SHIE
             LDeye-_-N82E16881147042
    
  2. BLAH: I get an error from this, but it’s here for future referene.

    To make the response clickable, feed the response through the out-gridview cmdlet introduced with v2 with an alias of ogv:

    
    $response | ogv -Title "Deal of the Day" -OutputMode Multiple | foreach { Start $_.link }
    
    
    [regex]$rx = "(?.)(?\d+\.\d{2})\s-\s(?.*)"
    </strong></pre>
    
    
    
  3. Another HTTP GET example (that no longer works) is:

    
    $url="http://www.seismi.org/api/eqs"
    $response = Invoke-RestMethod -Uri $uri
    # Extract child items:
    $people = $response.items
    

    This returns:

       
    count earthquakes                                                              
    ----- -----------                                                              
    21740 {@{src=us; eqid=c000is61; timedate=2013-07-29 22:22:48; lat=7.6413; lo...
    

POST Invoke-RestMethod

POST involve sending both a body and headers.

   $person = @{
      first='joe'
      lastname='doe'
   }
   $body = (ConvertTo-Json $person)
   $hdrs = @{}
   $hdrs.Add("X-API-KEY","???")
   $hdrs.Add("X-SIGNATURE","234j123l4kl23j41l23k4j")
   $hdrs.Add("X-DATE","12/29/2016")
   Invoke-RestMethod -Uri $url -Method Post -Body $body -ContentType 'application/json' -Headers $hdrs
   

ConvertTo-Json

The above on several lines is easier to read than one long line:

   $hdrs = @{"X-API-KEY"='???'; "X-SIGNATURE"='234j123l4kl23j41l23k4j'; X-DATE"='12/29/2016'"}
   

The power of Powershell vs wget are such helpers and how it can fluidly turn input into objects, and then to manipulate those objects in a granular way.

The API-KEY is obtained from the service’s website during sign-up.

WARNING: If -ContentType 'application/json is not added to REST calls, an error message is likely because when POST is specified, Invoke-RestMethod sets the content type to “application/x-www-form-urlencoded” for sending out forms, not REST calls.

Base64 Encoding for Authentication

See The example at Profitbricks for an example. You won’t be able to run the code if you don’t have an account.

But you’ll want to get an account because it’s a great service that is convenient and enables you to work with multiple clouds.

Authentication

For session authentication with cookies, see https://community.qualys.com/docs/DOC-5594 based on https://www.qualys.com/docs/qualys-api-v2-user-guide.pdf

  1. Define credentials in environment variables:

    $username = 'me_user'  
    $password = 'me_password'  
    $target = "Daily Whatsis Roundup"  
    
  2. Obtain a session variable sess

    $hdrs = @{"X-Requested-With"="powershell"}  
    $base = "https://qualysapi.qualys.com/api/2.0/fo"  
    $body = "action=login&username=$username&password=$password"  
    Invoke-RestMethod -Headers $hdrs -Uri "$base/session/" -Method Post -Body $body -SessionVariable sess
    

    This doesn’t work anymore https://community.qualys.com/docs/DOC-4523#jive_content_id_Windows_Powershell_30

    $username = "username"  
    $password = "password"  
    $password_base64 = ConvertTo-SecureString $password -AsPlainText -Force  
    $creds = New-Object System.Management.Automation.PSCredential ($username, $password_base64)  
    $headers = @{"X-Requested-With"="powershell"}  
    $url = "https://qualysapi.qualys.com/about.php"  
    Invoke-RestMethod -Headers $headers -Uri $url -Method Post -Credential $creds -OutFile response.xml  
    

Basic Authentication to GitHub

The code below makes a request sending the credentials in an Authorization header:

‘Basic [base64(“username:password”)]’

In PowerShell that would translate to something like:

function Get-BasicAuthCreds {
    param([string]$Username,[string]$Password)
    $AuthString = "{0}:{1}" -f $Username,$Password
    $AuthBytes  = [System.Text.Encoding]::Ascii.GetBytes($AuthString)
    return [Convert]::ToBase64String($AuthBytes)
}
$BasicCreds = Get-BasicAuthCreds -Username "Shaun" -Password "s3cr3t"
Invoke-WebRequest -Uri $GitHubUri -Headers @{"Authorization"="Basic $BasicCreds"}

Ignore Self-Signed Certs

http://www.datacore.com/RESTSupport-Webhelp/using_windows_powershell_as_a_rest_client.htm notes When using Windows PowerShell as a client, to avoid SSL Certificate trust issues if using HTTPS, enter this function in the PowerShell window:

function Ignore-SelfSignedCerts
{
    try
    {
        Write-Host "Adding TrustAllCertsPolicy type." -ForegroundColor White
        Add-Type -TypeDefinition  @"
        using System.Net;
        using System.Security.Cryptography.X509Certificates;
        public class TrustAllCertsPolicy : ICertificatePolicy
        {
             public bool CheckValidationResult(
             ServicePoint srvPoint, X509Certificate certificate,
             WebRequest request, int certificateProblem)
             {
                 return true;
            }
        }
"@
        Write-Host "TrustAllCertsPolicy type added." -ForegroundColor White
      }
    catch
    {
        Write-Host $_ -ForegroundColor "Yellow"
    }
    [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
}
Ignore-SelfSignedCerts

Invoke-WebRequest to get file

Here is an example of downloading a file from the internet into whatever path is specified in the environment variable $Temp.

echo "$Temp=${env:Temp}"
Invoke-WebRequest -Uri 'https://oneget.org/nuget-anycpu-2.8.3.6.exe' -OutFile "${env:Temp}\nuget.exe"

The Hey Scripting Guy article from 2013 by Doug Finke, author of Windows PowerShell for Developers, offered this example, which now returns a “401 (gone)” because it’s deprecated. Nevertheless, try the syntax on a working API:

  1. Filter the response through the PSCustomObject cmdlet and format it:

    Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell” | foreach {[PSCustomObject]@{Title=$.Title; Author=$.Author.name; Link=$_.content.src}} | Format-List

Resources

https://www.jokecamp.com/blog/invoke-restmethod-powershell-examples/

http://www.powershellmagazine.com/2014/12/24/using-azure-resource-management-rest-api-in-powershell/

https://www.youtube.com/watch?v=U3Ne_yX4tYo&index=1&list=PL5uoqS92stXioZw-u-ze_NtvSo0k0K0kq PowerShell Excel Module - ImportExcel Playlist of 5 vidoes Oct 5, 2017 by Doug Finke

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering