Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Deploy and control AWS, Azure, GCP, K8S, Serverless by coding JavaScript, TypeScript, Go, Python, etc.

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This is a hands-on introduction carefully sequenced to make complex material easier to understand quickly. This is not a “demo”, but an immersive step-by-step “deep dive” tutorial aimed to make you productive, without sales generalizations. “PROTIP” in the text highlight useful tools and insightful commentary based on experience by the author.

One source file to rule them all

Pulumi’s technologies enable developers to manage resources (Kubernetes, Serverless Lambdas/Functions, etc.) within multiple clouds by coding in general programming languages referencing Pulumi’s SDK libraries rather than editing declarative text files (in YAML of AWS Cloud Formation, HCL of Terraform).

BTW A footnote in CEO Joe Duffy’s personal June 18, 2018 blog announcing Pulumi says the company and product name “pulumi” is the Hawaiian word for “broom”, adopted as a homophone (mispronunciation) of Chris Brumme’s name, a friend and mentor.

Forbes article Oct 16, 2018

As an example, here’s JavaScript for Pulumi to create a new bucket in AWS S3:

   var aws = require("@pulumi/aws");
   const bucket = new aws.s3.Bucket("my-bucket");
   
  • When inside a .js file, “my-bucket” can be substituted with a variable.
  • The lines can be reused by being within a loop or placed in functions, and classes.
  • A single source code file can operate (transfer data) across several clouds (AWS, Azure, GCP, OpenStack, VMWare vSphere, etc.).
  • Use of general programming languages enable incorporation of code such as sending out emails, SMS texts, Slack notifications, etc.
  • Enforced error handling, compiler errors on invalid syntax, stack track debugging with line numbers.

The above is akin to calls to the AWS Boto3 Python library, but Pulumi provides more languages and for other cloud platforms and services in those clouds (such as Kubernetes and serverless lambdas/functions).

This is the same experience as Kris Nova who on July 13, 2017 blogged about transitioning from the YAML in kops cluster tool: “We replaced 1,000 lines of YAML with 10 Go structs and people started contributing again”.

The sample code above is specific to a particular cloud platform (aws).

multi-cloud coding

Pulumi is also working on building on top of the “@pulumi/aws” library to combine infrastructure definitions and application code in the same program source file using, initially, their @pulumi/cloud-aws API.

Pulumi is also working on “multi-cloud” coding that can be applied to several cloud platforms: the Pulumi Cloud Framework at https://github.com/pulumi/pulumi-cloud with higher-level “abstractions” such as

[Service], [Table], [Topic], and [API]

The plan is to offer an implementation of this on all major clouds rather than just aws currently. See Pulumi’s full Roadmap.

See https://pulumi.io/quickstart/cloudfx/index.html and [37:36] video

Infrastructure as Code (IaC)

https://github.com/pulumi/examples presents sample code for various clouds. The use cases have this naming convention:

cloud provider (platform or service) - language - use case

Cloud platforms are:

  • aws (Amazon Web Services)
  • azure (from Microsoft) - https://pulumi.io/reference/cd-azure-devops.html
  • gcp (Google Cloud Platform)
  • OpenStack and VMWare vSphere are coming

Cloud services:

  • kubernetes
  • multicloud
  • twilio (to send SMS texts, phone messages, etc.) https://github.com/pulumi/examples/blob/master/twilio-ts-component
  • https://github.com/pulumi/pulumi-openfaas

Pulumi provides an SDK of classes for programming clouds at repo
https://pulumi.io/reference/pkg/index.html (which is generated by Jekyll from the https://github.com/pulumi/docs repo).

Languages

Languages supported are

  • go for GoLang (as in aws-go-s3-folder)
  • js for JavaScript
  • ts for TypeScript (a wrapper over JavaScript)
  • py for Python 3.6+ (https://pulumi.io/reference/python.html) https://pypi.org/project/pulumi/

Pulumi works kinda like Microsoft’s .NET CLR (Common Language Runtime) which compiles several languages into a common set of instructions at the machine level. In fact, Pulumi’s engine was created by some of the people who worked on Microsoft’s CLR (Matt Ellis @ellismg, Pat Gavlin, etc.)

Dependencies & State like Terraform

The Pulumi engine is written in GoLang*

Pulumi determines the sequence which infrastucture components need to be created by compiling programming code languages. Internally, Pulumi resolves dependencies by building a data stucture (a Directed Acyclic Graph like Git does) needed to build the desired state requested from programs.

Pulumi stores its own copy of the current state of each stack in transactional snapshots called checkpoints at various points in time. This works like Puppet which makes a “diff” of the goal state and compares it versus current state to identify drift or what needs to be updated.

This is similar to what Hashicorp’s Terraform does under the covers.

Pulumi has Terraform plug-ins:

The ability to control IBM, Alibaba, Rackspace, and other clouds not yet defined can be converted from Terraform HCL files to Typescript programming code using the tf2pulumi utility (written in Golang).

  • https://github.com/pulumi/pulumi-terraform-template
  • https://github.com/pulumi/pulumi-alicloud by Pat Gavlin

The Pulumi Terraform Bridge and examples of its output is at https://github.com/pulumi/tf12-vs-pulumi

QUESTION: Provide “production-ready” code like Gruntwork does for Terraform users?

BTW gRPC utilizes protocol buffers to communicate the interface of the services and then has a compiler to generate source code in one of a variety of supported languages. https://github.com/pulumi/pulumi/tree/master/sdk/proto https://grpc.io/docs/tutorials/basic/node.html Protocol buffers (protobuf) https://forestgiant.com/articles/protocol-buffers/

#NoYAML for True DevOps + SRE

Pulumi’s approach solves a fundamental problem for DevSecOps.

PROTIP: Coding YAML to configure AWS CloudFormation or Terraform declarative DSL, with their thousands of cryptic settings, is a complexity which requires dedicated specialists. Because they hard to find, such experts are relegated to work in a silo, separate from development people who have to literally get in line to talk to them.

Such separation does not satisfy the need for increasing delivery speed, namely:

“Share ownership with developers by using the same tools and techniques across the stack”*

By making it trivial for developers and testers to spin up and tear down lots of stacks, Pulumi reduces the friction and thus enhances the speed to test out each Pull Request (individual feature) or entire tiers of technology.

https://noyaml.com/ from code at https://github.com/ghuntley/noyaml

  • https://arp242.net/weblog/yaml_probably_not_so_great_after_all.html

  • https://twitter.com/ellism/status/1008728148131733504 “What if you used the same language and tools you use to define your app to define your infrastructure?”

Cloud Console and account & organization

  1. Open an internet browser (Google Chrome is preferred).
  2. Login to your GitHub account at:

    https://github.com

    This establishes credentials in the browser for use by Pulumi Cloud in the next steps.

  3. In the same browser, paste this URL to enter the Pulumi Cloud Console :

    https://app.pulumi.com

  4. Establish a Pulumi account

    How Pulumi makes money

    Code for the pulumi CLI is open-sourced at https://github.com/pulumi/pulumi and the Pulumi company has joined the CNCF (Cloud Native Computing Foundation).

    BTW, Pulumi is funded by Madrona Venture Group, who also funded Lyft, Heptio, Tigera.

    Pulumi works on a Freemium business model, with a free tier for a single “stack” without enterprise scale and security features.

    Each Pulumi “stack” is an isolated instance of a particular environment such as dev, qa, staging, prod-west, prod-east, etc. See https://pulumi.io/reference/stack.html

    You are charged for the number of stacks. See https://www.pulumi.com/pricing/

    Pulumi also makes money from offering team, enterprise features, and on-premise self-hosting.

    See https://github.com/wilsonmar/docs-2/blob/master/reference/service/editions.md

  5. To use Pulumi’s Team features (for $600/year after a 14-day free trial), define a Pulumi organization at:

    https://app.pulumi.com/site/organizations/add

    https://help.github.com/articles/reviewing-your-authorized-integrations/

Installs

PROTIP: The brevity of Pulumi’s code makes it ideal for the next advancement in technical tutorials. The current approach on ACloud.guru, Pluralsight, Lynda.com, etc. is that you first view videos to learn concepts, then manually click through GUI. However, in this tutorial, you first run a script that does everything, then you get explanation of the automated steps as you customize the script for your own needs.

MEH: Pulumi’s instructions assume that readers manually enter pulumi cli commands. Sorry but that’s slow and error-prone.

PROTIP: So you can get to work quicker (without a background in other tools), I built a Bash script to automatically install all the pre-requisite utilities and files to enable Pulumi’s multi-cloud and multi-language capabilities.

https://github.com/wilsonmar/DevSecOps/blob/master/Pulumi/pulumi-setup.sh

The shell script above installs packages in this specific sequence: xcode-cli, homebrew, VSCode, git, Python3 > aws & azure-cli, Node.js, Go, Docker, Pulumi CLI.

Included in the bottom of the script is a sample use of a simple Pulumi deployment before cleaning up so charges do not linger.

The script is designed to be idempotent in that it can be rerun repeatedly. For this reason, the script deletes what is left over from a previous run. The script also has a flag which defaults to deleting what was created at the end of the script, so billing is minimized.

PROTIP: I recommend that you edit and add to the bottom of installation scripts I’ve provided. It only takes a few seconds to run. This ensure that you have the environment necessary in case of configuration changes and upgrades. And if you run into issues, the install portion of the script outputs version numbers to inform debugging efforts.

Every time it runs, it clones or pulls (updates) repos into its local .pulumi folder:

  • https://github.com/pulumi/examples
  • https://github.com/pulumi/templates

Pulumi CLI Install

Pulumi CLI installer for macOS and Windows package managers.

For macOS:

  1. Pulumi provides its own Bash shell script to install the Pulumi CLI program. This has the advantage of using the same command for first-time install as well as update:

    curl -fsSL https://get.pulumi.com | sh
    • It extracts to path $HOME/.pulumi/bin such as /Users/wilsonmar/.pulumi/bin.
    • It adds $HOME/.pulumi/bin to $PATH in /Users/wilsonmar/.bash_profile

  2. I prefer to use Homebrew for its remove command and because it does not require changes to my .bash_profile since it uses the same mecahnism as other packages:

    brew install pulumi

    In the response, note that the download is for a specific macOS version:

    ==> Downloading https://homebrew.bintray.com/bottles/pulumi-0.16.7.mojave.bottle.tar.gz
    
    Date Version # files Size
    2018-12-05 /usr/local/Cellar/pulumi/0.16.6 13 files 80.9MB
    2018-12-10 /usr/local/Cellar/pulumi/0.16.7 13 files 80.9MB

    NOTE: In the Info/Caveats: The installer adds code completion hints for Bash, ZSH.

    Bash completion has been installed to:
      /usr/local/etc/bash_completion.d
     
    zsh completions have been installed to:
      /usr/local/share/zsh/site-functions
    

    ??? Either way, VSCode autocomplete are installed for 500+ AWS APIs are installed with autocomplete for the JavaScript let commands which define properties.

  3. Verify the install

    pulumi version

    The response should show the same version as in the output message:

    v0.16.7
  4. Install bash completion:

    brew install bash-completion

    See https://pulumi.io/reference/commands.html#command-line-completion

    VSCode autocomplete for 500+ AWS APIs?

  5. To install the core API package, use either npm:

    npm install @pulumi/cloud

    Alternately, if you use yarn:

    yarn add @pulumi/cloud

Pulumi CLI commands

https://pulumi.io/tour/basics-projects.html (based on https://github.com/wilsonmar/docs-2/tree/master/tour)

  • Programs
  • Creating a project
  • Deploying code
  • Previewing
  • Spinning up
  • Performing updates
  • Destroying

https://www.youtube.com/watch?v=YMAe59BYzm4 Ep 002: How Pulumi Works [40:41] using AWS Fargate (not lower-level ECS)

  1. Make a Git-backed folder to contain your new Pulumi project repo.

  2. View the parameters for the “pulumi new” command:

    pulumi new --help
      Global flags:
       --color string                 Colorize output. Choices are: always, never, raw, auto (default "auto")
      -C, --cwd string                   Run pulumi as if it had been started in another directory
       --disable-integrity-checking   Disable integrity checking of checkpoint files
      -e, --emoji                        Enable emojis in the output (default true)
      -h, --help                         help for pulumi
       --logflow                      Flow log settings to child processes (like plugins)
       --logtostderr                  Log to stderr instead of to files
       --non-interactive              Disable interactive mode for all commands
       --profiling string             Emit CPU and memory profiles and an execution trace to '[filename].[pid].{cpu,mem,trace}', respectively
       --tracing string               Emit tracing to a Zipkin-compatible tracing endpoint
      -v, --verbose int                  Enable verbose logging (e.g., v=3); anything >3 is very verbose
    

    PROTIP: There is a list of templates available from https://github.com/pulumi/templates at the bottom of the response:

      aws-go                   A minimal AWS Go Pulumi program
      aws-javascript           A minimal AWS JavaScript Pulumi program
      aws-python               A minimal AWS Python Pulumi program
      aws-typescript           A minimal AWS TypeScript Pulumi program
      azure-go                 A minimal Azure Go Pulumi program
      azure-javascript         A minimal Azure JavaScript Pulumi program
      azure-python             A minimal Azure Python Pulumi program
      azure-typescript         A minimal Azure TypeScript Pulumi program
      gcp-go                   A minimal Google Cloud Go Pulumi program
      gcp-javascript           A minimal Google Cloud JavaScript Pulumi program
      gcp-python               A minimal Google Cloud Python Pulumi program
      gcp-typescript           A minimal Google Cloud TypeScript Pulumi program
      go                       A minimal Go Pulumi program
      hello-aws-javascript     A simple AWS serverless JavaScript Pulumi program
      javascript               A minimal JavaScript Pulumi program
      kubernetes-javascript    A minimal Kubernetes JavaScript Pulumi program
      kubernetes-typescript    A minimal Kubernetes TypeScript Pulumi program
      openstack-go             A minimal OpenStack Go Pulumi program
      openstack-javascript     A minimal OpenStack JavaScript Pulumi program
      openstack-python         A minimal OpenStack Python Pulumi program
      openstack-typescript     A minimal OpenStack TypeScript Pulumi program
      python                   A minimal Python Pulumi program
      typescript               A minimal TypeScript Pulumi program
      

    A new command crates in the present working directory a Pulumi.yaml file containing some minimal metadata about your project (including a name and description which you may wish to change).

  3. To create a new project by specifying a template name:

    pulumi new template-name

    NOTE: pulumi new has these parameters:

      -c, --config stringArray   Config to save
      -d, --description string   The project description; if not specified, a prompt will request it
       --dir string           The location to place the generated project; if not specified, the current directory is used
      -f, --force                Forces content to be generated even if it would change existing files
      -g, --generate-only        Generate the project only; do not create a stack, save config, or install dependencies
      -h, --help                 help for new
      -n, --name string          The project name; if not specified, a prompt will request it
      -o, --offline              Use locally cached templates without making any network requests
      -s, --stack string         The stack name; either an existing stack or stack to create; if not specified, a prompt will request it
       --suppress-outputs     Suppress display of stack outputs (in case they contain sensitive values)
      -y, --yes                  Skip prompts and proceed with default values
    

    If you issue “pulumi new” before login:

    warning: A new version of Pulumi is available. To upgrade from version '0.16.6' to '0.16.7', visit https://pulumi.io/install for manual instructions and release notes.
    Manage your Pulumi stacks by logging in.
    Run `pulumi login --help` for alternative login options.
    Enter your access token from https://app.pulumi.com/account
     or hit  to log in using your browser            : 
    </pre>
    
    
  4. Configuration by Enterprise users can encrypt configuration in your stack, you’ll need to re-run

    pulumi config set --secret key value 
    

    This is because pulumi.com uses a different key to encrypt your secrets than the local endpoint.

    To create a stack in a different Pulumi organization, prefix the stack’s name with the organization’s login:

    pulumi stack init acme-corp/widget-server
  5. To Deploy code and/or resource changes

    pulumi up
  6. To preview and deploy changes to your program and/or infrastructure

    https://github.com/pulumi/apps ???

  7. To configure variables such as keys, regions, etc.

    pulumi config
  8. Log into the Pulumi service then display current logged in user:

    pulumi login
    pulumi whoami

    This is equivalent to manually logging into the https://app.pulumi.com Pulumi Cloud Console using your GitHub or GitLab credentials.

  9. Manage language and resource provider plugins

    pulumi plugin

  10. To list key-value pairs managed by pulumi, such as aws:region us-west-2

    pulumi stack ls
  11. To manage stacks (at least one is required to perform an update):

    pulumi stack select name
    pulumi stack output

    lists content of “export let publicIp = server.publicIp;”

  12. To preview your changes explicitly before deploying

    pulumi preview

    [9:09] Like Terraform, internally, Pulumi builds a graph that represents the desired goal state, then diff the current state to create a plan.

  13. To build using yarn (instead of Maven or Ant):

    yarn build
  14. Refresh the resources in a stack:

    pulumi refresh
  15. Edit the current stack’s state:

    pulumi state

  16. Update:

    pulumi update
  17. Show aggregated logs for a stack:

    pulumi logs

    Destroy and Logout

    While you are testing pulumi commands:

  18. Destroy your program and its infrastructure when you’re done

    pulumi destroy
  19. Log out of the Pulumi service

    pulumi logout

BTW, functional in the CLI can be accessed in code:
https://pulumi.io/reference/pkg/nodejs/@pulumi/pulumi/index.html

Builds

Pulumi has automated integrations for CI/CD (Continuous Integrations & Deployment):

https://pulumi.io/reference/cd-github-actions.html makes use of https://developer.github.com/actions/ beta

To update your Pulumi stack on-demand from AWS CodeBuild and other AWS Code Services products, configure your CloudBuild project.

https://pulumi.io/reference/cd-travis.html

TravisCI can bet set to add a comment to specific GitHub repos about the result of pull requests by installing https://github.com/apps/pulumi, as described at here. Access to source code is not needed for this.

To avoid multiple commits being merged into the master branch at the same time, To avoid conflicting resource updates or corrupting resource state. https://github.com/pulumi/travisqueue limits build concurrency on a per-branch basis. This allows you to limit the number of concurrent builds for any branches that are configured to perform a Pulumi update. So Travis will only have one build for the master branch at a time, but could be running any number of conrrent builds for other branches. .travis.yml file ???

https://pulumi.io/reference/cd-gitlab-ci.html

Contribution to open source

Pulumi created and open-sourced a tool: Kubespy at https://github.com/pulumi/kubespy

Rock stars

https://www.youtube.com/channel/UC2Dhyn4Ev52YSbcpfnfP0Mw YouTube channel

https://www.youtube.com/watch?v=YMAe59BYzm4 Ep 002: How Pulumi Works [40:41]

http://joeduffyblog.com/ Joe Duffy is Founder/CEO https://github.com/joeduffy

https://www.youtube.com/watch?v=DM8Wd4f1MNA Twitch stream Ep 000: Pulumi Prolog Jun 5, 2018 Luke Hoban, CTO https://www.linkedin.com/in/lukejhoban/ from Amazon & Microsoft

Marc Holmes CMO https://www.linkedin.com/in/marcholmes/ from Chef and Docker

https://www.linkedin.com/company/pulumi/

https://hire.withgoogle.com/public/jobs/pulumicom/view/P_AAAAAAFAAC1NXwlGmpMEPP Developer Advocate job

Sean Gillespie https://twitter.com/swgillespie @swgillespie

davidk01 David karapetyan https://dev.to/davidk01/bootstrapping-a-consul-cluster-with-pulumi-35d4

Alex Clemmer wrote the Kubernetes provider https://pulumi-community.slack.com/team/UBAJ7TD53

https://www.cncf.io/ck Certified Kubernetes Conformance tests

https://cloudbootup.com/

References

  • tour.
  • Pulumi’s QuickStart
  • Delivering Cloud Native Infrastructure as Code, a Pulumi white paper

  • June 18, 2018 - [[CMO Joe Duffy blogs “Hello Pulumi” announcing the platform]]cite web|url=http://joeduffyblog.com/2018/06/18/hello-pulumi/|title=Hello Pulumi|publisher=joeDuffyBlog.com

  • October 22, 2018 - [[Pulumi raises $15M for its infrastructure as code platform]]cite web|url=https://techcrunch.com/2018/10/22/pulumi-raises-15m-for-its-infrastructure-as-code-platform/|title=Pulumi raises $15M for its infrastructure as code platform|publisher=TechCrunch.com

  • https://techcrunch.com/2018/06/18/pulumi-wants-to-let-you-manage-your-infrastructure-with-code/

Pulumi US offices are at 520 E. Denny Way, Seattle, WA 98122

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud services comparisons (across vendors)
  16. Cloud regions (across vendors)
  17. AWS Virtual Private Cloud

  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell
  21. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  22. Digital Ocean
  23. Cloud Foundry

  24. Packer automation to build Vagrant images
  25. Terraform multi-cloud provisioning automation
  26. Hashicorp Vault and Consul to generate and hold secrets

  27. Powershell Ecosystem
  28. Powershell on MacOS
  29. Powershell Desired System Configuration

  30. Jenkins Server Setup
  31. Jenkins Plug-ins
  32. Jenkins Freestyle jobs
  33. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  34. Docker (Glossary, Ecosystem, Certification)
  35. Docker Setup
  36. Dockerize apps

  37. Maven on MacOSX

  38. Ansible

  39. MySQL Setup

  40. SonarQube static code scan

  41. API Management Microsoft
  42. API Management Amazon

  43. Scenarios for load

More on Amazon

This is one of a series on Amazon: