Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Pick which crew of robots to build your servers

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This tutorial describes the options AWS provides to automate setup of multi-stage (dev+QA+prod) enterprise environments within EC2.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

The options:


Manually configure component services

Each environment within AWS for enterprise use requires several services. Here is the sequence of dependencies:

  1. VPN
  2. VPC
  3. NAT

  4. DNS
  5. ELB

  6. AMI by Region, with Auto-scale

Elastic Beanstalk

AWS Simple Icons Compute AWSElasticBeanstalk

Before doing this, setup default VPC, subnets, and Security Groups.

  1. Services > Compute > Beanstalk

    NOTE: You can’t SSH into individual servers.

  2. Specify an Application Name.

    PROTIP: Define a convention that applies to apps, such as a project, feature, and version number, such as PS1-bean2-node-v01.

    PROTIP: Include in the name a code for the platform being used.

  3. Select a Platform.

  4. Click Configure more options.

    Beanstalk is considered a “Platform-as-a-Service” (PaaS), that does the “heavy lifting” to get infrastructure online, with load balancing, autoscaling, and health monitoring.

    The Virtual Machine is for the Platform chosen in the previous step.

    The Low cost configuration preset is the default.

    In the Scaling section, the Environment type is single instance.

  5. Click the Highly available configuration preset.

    Notice the Environment type changed to “loadbalancing, autoscaling” with Scale instance: 1-4.

    Beanstalk is free to use. You only pay for servers deployed by it.

  6. Click Modify in the Notifications section and input your email address.

  7. Switch temporarily to your email to confirm the subscription.

  8. Since this is a tutorial, select the Low cost single instance.

  9. Scroll down to click Create app.

Beanstalk Settings

Configuration information stored in the .ebextensions folder containing:

  • a dynamodb.config

    The file contains functions definitions such as Fn::GetOptionSetting: with parameters.

  • a options.config files.

  • The .elasticbeanstalk folder ???


Opsworks Chef

Opsworks is a higher level tool than CloudFormation, offering more customization than Elastic Beanstalk.

  1. Services > Management Tools > Opsworks

  2. Click Add your first stack.

    NOTE: You cannot mix and match Windows with Linux servers.

Opsworks is called a “configuration as code service” because it sets up servers by running Chef recipies obtained from a Cookbook repository.

NOTE: There is no equivalent for Puppet.

Each “layer” is a blueprint and container for instances. (JSON) defining stacks:

  • OpsWorks
  • ECS
  • RDS

    A different Chef recipie for each event within the lifecycle :

    • Setup
    • Configure
    • Deploy
    • Undeploy
    • Shutdown

Opsworks Cookbooks from @Danilop.

NOTE: Each server has a Chef agent installed.

The lack of agents is why Ansible is becoming more popular.

Ansible using CloudFormation

My tutorial on AWS High-Availability using CloudFormation, which is the current rage (June 2016).


EC2


Amazon EC2 Cloud Images

GUIDE

A hardened Amazon Machine Image (AMI) containing Visual Studio 2017 Community Edition on Windows Server 2016 costs 28 cents per hour on a t2.medium in the US. 20 cents of that goes pays for support from Cognosys, its creator. 888.489-2723

Cost per hour increases in a straight line (linear) way for number of CPUs: ec2-cognosys-vcpu-per-dollar-562x217-54344
Cost per hour increases in the same way for amount of Memory (RAM): ec2-cognosys-mem-per-dollar-562x225-62637

ec2instances.info provides a spreadsheet.

WARNING: On AWS EC2, Windows Server 2016 Nano servers do not support RDP, only Windows PowerShell.

A t2.medium has 2 cores and 4 GB.

PROTIP: With Amazon, you pay for hourly increments. With Google, you pay per minute.

Connect to an instance in AWS EC2:

  1. Create an Amazon EC2 account at http://aws.amazon.com/ec2/.

Client WorkSpaces in AWS cloud

This approach works not just a MacOS laptop, but for any computer running a modern browser, such as on a Chromebook. No files are transferred, just graphic images of a screen on servers within the AWS cloud. This makes for more stringent security, but also means significant lag that affect productivity.

  1. Use an appropriate AWS IAM account to login to the WorkSpaces Console at

    https://console.aws.amazon.com/workspaces/home

    windows-on-mac-189x434-6552.jpb

  2. Select the region.
  3. Click “Directories” menu and set up a Directory.
  4. Launch. Select the Directory.
  5. For the simplest approach, select “Simple AD” Next. Small AWS Managed Microsoft AD”.
  6. Use a password generator UoyJhssxbcQzrDwT8ciF. Next.
  7. Choose VPC and two subnets.
  8. Wait for Status to go from “Requested” to “Creating” to “Active”.

    Launch Workspaces

    An image contains only the OS, software, and settings. A bundle is a combination of both that image and the hardware from which a WorkSpace is launched.

    The Free Tier provides two Standard bundle WorkSpaces for up to 40 hours of combined use per month, for two calendar months, from the time you create your first WorkSpace. Usage time accrues while you’re actively using your WorkSpace as well as the time it takes to stop after a specified period of inactivity, which by default is set to one hour. If you exceed the Free Tier limits, you will be charged the standard Amazon WorkSpaces hourly rate for the additional resources you use. At the end of two calendar months, the WorkSpaces you launched in the Free Tier will automatically be billed at the applicable hourly rate.

    Amazon’s regular pricing is $25 to $75 per month per user, which Amazon estimates is 59% less than traditional Virtual Desktop Infrastructures (VDI) from Citrix and VMware.

    Amazon’s approach uses newer tech than VDI.

  9. Click “WorkSpaces” in menu.
  10. Click blue “Launch WorkSpaces”.
  11. Select the Directory and Subnets.

    Set-up Users

  12. Specify for each user his/her Username, First Name, Last Name, and Email for the Bundle selected.

    PROTIP: Use a email as the Username.

  13. Click “Create Users”.
  14. Check the user you want.

    If a new user input was already defined, the form is cleared.

  15. Click “Show All Users”.
  16. Check the user.
  17. Click “Next Step”.
  18. Select Bundle for the OS (with default 80 GB root and 50 GB user volume).
  19. Select Running Mode (AlwaysOn or AutoStop hours).
  20. Click “Next Step”.

  21. Click Launch Workspace (for all users).
  22. Wait (about 20 minutes) for the Workspace Console goes from PENDING to AVAILABLE (in green letters).
  23. To refresh the page, type command+R or click the recycle icon.

    Amazon WorkSpaces Application Manager (WAM)

  24. Switch to return to managed users in the Amazon WorkSpaces Application Manager (WAM) for your current region at:

    https://us-west-2.console.aws.amazon.com/wam/home

    CAUTION: There is no moving WorkSpaces from one region to another.

    There is a WAM Standard for additional functionality.

    Install WorkSpaces client

  25. In each user’s email client, open the welcome email and click the link.
  26. Set your WorkSpaces credentials with a password.s
  27. Highlight the registration code in the email and copy to your Clipboard.

  28. Choose the link for your laptop model at
    http://clients.amazonworkspaces.com

  29. Click to download the “WorkSpaces.pkg” to your Downloads folder.

    On a MacOS, it’s file “WorkSpaces.pkg” (38.2 MB taking 115.3 MB space).

    On a Chromebook, click “ADD TO CHROME” at the upper-right corner.

  30. Switch to Finder and click to invoke the installer the Chrome application to enable the proprietary PC over IP (PCoIP) protocol (from Teradici) to compress, encrypt and rapidly transport image pixels between client and server.

  31. Double-click on the installer and click Continue and finally, Install. Provide your password when requested.

  32. Move the installer file (WorkSpaces.pkg) to Trash, to recover disk space.

  33. Do a Chromebook search to verify that the Amazon WorkSpaces client app icon appears.

  34. BLAH: The full graphic streaming virtual desktops tends to eat up much bandwidth. So measure how much you have used before and after sessions.

    Optionally, your company’s existing on-premises Active Directory (AD) can be reached by Amazon via an Amazon Virtual Private Cloud (VPC) with a hardware virtual private network (VPN) connection or a dedicated connection with AWS Direct Connect.

    Once linked up, you use the AWS Management Console to select the users in your Active Directory who will receive a WorkSpace.

  35. Open the app for the first time.
  36. Type your user name and password and choose Sign In.
  37. Switch to your user’s email client and highlight the registration code from the “Your Amazon WorkSpace” email, then paste on the form.
  38. Click Register.

    Repeat Login

    NOTE: 1Password cannot auto-fill Username and Password on the MacOS WorkSpaces app nor on Chrombook.

  39. Login using the Username and password for the WorkSpaces client.

    If your Amazon WorkSpaces administrator has enabled multi-factor authentication for your organization’s WorkSpaces, you are prompted for a passcode to complete your login.

    CAUTION: Only one browser can be used at the same time. Amazon logs off a session when another session is started on another computer.

  40. If your Amazon WorkSpaces administrator has not disabled the “Remember Me” feature, you are prompted to save your credentials securely so that you can connect to your WorkSpace easily in the future. Your credentials are securely cached while the application is running.

    After the client application connects to your WorkSpace, your WorkSpace desktop is displayed.

  41. (Optional) If your WorkSpace uses an AD Connector directory, update the maximum lifetime of the Kerberos ticket by following the steps in Configuring Kerberos Policies in the Microsoft TechNet Library.

  42. If you need to disable the “Remember Me” feature, search for help in the Amazon WorkSpaces forum.

    Configure Remote Assistance

  43. Open PowerShell window.
  44. Install Remote Assistance using this PowerShell command:

    Add-WindowsFeature Remote-Assistance </tt>

    Open port 3389 in the firewall and in the Security Group, Remote Desktop should work using the username and password in the traditional way.

    Configure Windows 7 Folder Options

  45. Click the Windows Start round icon at the bottom left of the screen.
  46. Type “folder options” (without the quotes) until the line “Folder Options” appears for you to click at the top of the menu.
  47. In the “Folder Options” dialog box, click the “View” tab at the top of the window.
  48. Select “Show hidden files, folders, and drives”.
  49. Click to uncheck the box for “Hide extensions for known file types”.
  50. Click the “OK” button at the bottom of the dialog box.

    Configure Windows 7 Toolbar

  51. Click the Windows Start round icon at the bottom left of the screen.
  52. Click All Programs, Accessories. All the usual tools are there.
  53. Drag Notepad and drop it on the tool bar at the bottom of the screen.

  54. Click the Windows PowerShell folder.
  55. Drag “Windows PowerShell” and drop it on the tool bar at the bottom of the screen.

  56. Open Windows Explorer from the tool bar at the bottom of the screen.
  57. Click on “Computer”. Notice there is no C: drive and no access to C:\Windows internals.
  58. Double-Click on “User Profile D:” drive.

    Notice there are 50 GB for you.

  59. Double-Click on D: and navigate into folder Users, your account name.

    PROTIP: Here is the default location when command line windows open by default. So place scripts here (among folders).

  60. Right-Click Windows PowerShell to select “Run as Administrator”.
  61. Type:

    echo $Env:USERPROFILE

    This is your user home folder.

  62. Set permissions:

    set-executionpolicy remotesigned

    Execution Policy Change
    The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
    you to the security risks described in the about_Execution_Policies help topic at
    http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the execution policy?
    [Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y
    
  63. Type Y to confirm.

    See How to write a PowerShell script

  64. Within PowerShell you can also go to your home folder by typing a tilde:

    cd ~

  65. Verify whether you can create a PowerShell script file:

    Add-Content helloworld.ps1 ‘Write-Host “Hello World”’

    This is the PowerShell equivalent of echo "Hello World" >helloworld.ps1.

  66. List directory:

    dir

  67. Type the first letter h and press Tab to auto-complete:

    ./helloworld.ps1

    Instead of “Hello World”, if you get this, it means executionpolicy was not set correctly:

     + CategoryInfo          : SecurityError: (:) [], PSSecurityException
     + FullyQualifiedErrorId : UnauthorizedAccess
    

    Install posh-git for PowerShell

    TODO: file:///C:/Program%20Files%20(x86)/AWS%20Tools/Documentation/AWSToolsForWindows.html AWS Tools for Windows AWS SDK for .NET

  68. To exchange files among a group of people, setup:
    https://amazonworkdocs.com/en/clients

    Install clients using Chocolatey

    [Find-Package from OneGet included in Windows Management Framework 5.0 Preview gets packages from the Chocolatey installer repository]

  69. Right-click on cmd and select “Run as Administrator”.
  70. Copy this and right-click in the command window:

    
    @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
    
  71. Install Java Development Kit for version 8 using Chocolatey:

    choco install jdk8 -y

  72. Install Chrome browser:

    choco install googlechrome -y

  73. Install Git for Windows client:

    choco install git -y

    Alternately, to add a Git client manually, open Firefox, search for “Git for Windows”. Click Download. Click Save file. Click the down arrow for a list of downloads. Click to Open File. Run. Yes to UAC. Next all, but Git LFS. then Finish. Close Firefox. Click Windows icon. All Programs. Click Git, Git Bash.

  74. Install Posh Git for PowerShell:

    choco install poshgit -y

    • Launch each and configure the window properties to enable Quick Edit and set Layout Height (scroll buffer) to 9999 lines.

    https://www.develves.net/blogs/asd/articles/using-git-with-powershell-on-windows-10/#fn:start

    Clone Samples

    Download Git repository containing bootstrap script:

  75. Open a Git Bash window.
  76. cd to where you add Git repositories:

    cd gits;
    git clone https://github.com/wilsonmar/loadrunner.git --depth=1
    git clone https://github.com/wilsonmar/git-utilities.git --depth=1

    Shut-down and Resume

    BLAH: It takes many minutes to stop and resume.

    Custom WorkSpaces

  77. As an administrator, in the Console, select the WorkSpace and select “Create Image” to create an image with your applications and settings.

    NOTE: Custom images created from Amazon WorkSpaces Graphics bundles can only be used with Graphics bundles, and custom images created from Value, Standard, Performance, or Power bundles can only be used with those bundles. Most Amazon WorkSpace images are available within 45 minutes.

    See http://docs.aws.amazon.com/console/workspaces/images

Docker

Docker is now a favored approach because it runs the same (unaltered) (Dockerized apps) on Windows, Mac, and Linux platforms.

Docker provides a transparent interface to operating systems.

More on Amazon

This is one of a series on Amazon:

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering