Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Use Flood.io Element TypeScript to measure GUI performance accessing website stood up in AWS cloud using Docker, as measured by NewRelic

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Video introduction


Below is the narration (transcript) of the video above.

Many are familiar with this website: “the-internet” running on herokuapp.com

The website was created by Dave Hoeffner to present 43 controls which provide challenges to those learning to code Selenium scripts that automate manual actions real users perform on an internet browser (such as Google Chrome). Dave created the site as the basis for his tutorials at ElementalSelenium.com and SeleniumGuidebook.com.

Selenium makes use of older “Web Driver” APIs that control browsers from code written in a variety of programming languages.

But the new Google “Lighthouse” API exposes a comprehensive set of metrics that include performance timings for every manual action. They’ve been added to the “Audits” section of Chrome Developer Tools UI we’ve been using to figure out the classes and identifiers we code into Selenium scripts.

But since these “Lighthouse” diagnostics require manual effort each time, it can be tedious to keep repeating manual actions during each regression test. So we create scripts to automate manual actions in the Typescript programming language run by the Flood Element program. The program is installed locally using a shell script.

Typescript is a superset of the JavaScript programming language that control browsers. Typescript is used because its transpiler checks for errors sooner than JavaScript.

After scripts are git pushed to a remote GitHub repository, others can git clone onto their machine to run.

Both Element CLI and Selenium control a single user GUI at a time. But we also want to see what happens when many users on many browser instances run at the same time exercising the website. We want to see how much each JavaScript control can impact both the client and server environment’s memory, CPU, and other resources. «<

App Build

But we don’t want our experiments to overload Dave’s public site for everyone else.

So we run the app as a Docker container within the AWS or other cloud. The container and environment under test is built by invoking a build script that retrieves files from GitHub which makes use of a Docker image housed in Docker Hub and uses it to instantiate an app server for testing. It uses credentials for an AWS account associated with the appropriate groups with applicable permissions and roles needed.

Emulate using Flood

We use the Flood.io service in the cloud (or on-premises) to emulate those many users by running automation scripts pulled from a GitHub repository.

Flood Element scripts are a new innovation because it emulates manual actions within each user’s browser. Historically, Java code used by JMeter or Scala code used by Gatling emulate load by simply emulating just the exchange of what is sent between client browser and server. But a lot of work now occur inside the client browser.

Multi-user Flood runs are controlled by run parameters such as the number of virtual users being emulated. Run shell scripts can be used to manage various runs, test data, and results over various variations in run conditions.

Instrumentation

It’s important to have a metrics dashboard that helps people make sense of measurements collected over time. This article talks about use of New Relic.

We have an instrumentation script which installs an agent (newrelic-infra) to run alongside the app. Because the sample app under test was written in the Ruby language, the agent is installed as a rpm file (newrelic.rpm) specified in the Gemfile referenced during installation.

During runs, the agent sends notifications about events to a process installed using a Docker image in Docker Hub from New Relic. The monitoring process transfers events collected by agents to a metrics dashboard at newrelic.com.

To validate communications, a license key obtained manually from the New Relic website is installed when the agent is installed.

The license key, plus IP address and port number of the metrics collector are provided to Flood so that it can add its metrics to New Relic over time. These metrics include the number of users, transaction response times, the rate of transactions per second processed, network bandwidth throughput, and transaction pass/fail error rates.

When we also add the cost of each run, we would be able to identify which configurations would provide the most profitable number of transactions per dollar.

Recap: Architectural components

flood-the-internet-v12-1900x959.jpg

While learning on a local machine such as a Mac, Linux laptop, or Windows PC:

  1. Chrome Developer Tools is used to extract identifiers, classess, and selectors in HTML and CSS to programmatically reach each GUI control used in each TypeScript.

  2. The Flood Element CLI is installed to run Element TypeScript to emulate a single user’s manual actions on a Google’s Chrome browser.

The system under (load) test (SUT) instantiated using Docker:

  1. The “the-internet” web app from a Docker image
  2. The monitoring process, which can be in another container or as another pod within the same container as the web app.

Additionally, the GUI of two cloud services:

  1. https://flood.io which runs TypeScript to emulate the browsers for multiple users, which imposes an artificial load.

  2. A New Relic dashboard which provides visualization (line graphs) of metrics collected during runs over time and under various configurations.

How To Manual Steps

Here are the manual steps to make use of instructions in this tutorial:

NOTE: This assumes that you know how to open and use a Terminal program on a Mac or a Git Bash on Windows.

  1. Study the app under test and manual actions captured into Selenium and Flood Element test automation script code (below).
  2. Invoke the Flood Element CLI install locally using the Flood Element TypeScript code provided, which calls on Google’s Pupetter technology to control client browsers.

  3. Get an AWS service account linked to AWS Roles assigned to Groups with Permissions. See https://wilsonmar.github.io/aws-onboarding, which describes editing files which store credentails (UserName and Password) in environment files which are invoked during runs to populate environment variables referenced by automation scripts.

  4. Install within AWS “the-internet” app under test from DockerHub Flood Script Update-aws-ec2

    NOTE: Automation for installation within the Azure cloud is on our Roadmap.

  5. Obtain logins to accounts for New Relic APM, New Relic Browser, and New Relic Infrastructure.
  6. Install NewRelic’s monitoring process

    NewRelicAgentInstall-aws-ec2.sh

  7. Define initial run parameters to control runs of flood.io in the cloud (Flood.io account, etc.) in flood-run-e2e.sh.
  8. Run “flood-run-e2e” to launch runs in flood.io at scheduled times to take advantage of AWS Spot Rates.

    PROTIP: Automation (shell or Python) scripts to bring up servers and conduct runs are designed to minimize spend (not waste money on idle resources). Services are deleted after each run.

  9. Configure New Relic dashboard
  10. Review warning and error messages. Analyze statistics collected by the Chrome Browser’s Lighthouse and sent to NewRelic’s dashboard in the cloud.
  11. Identify the fastest realistic ramp-up time (without causing errors) by re-running with different rates which running users are added.

  12. Experiment with changes to JavaScript, HTML, and CSS in client application code in attempts to reduce timings and resource usage on browsers.
  13. Modify TypeScript accordingly to identify the impact of changes made to the application before reruns.

    NOTE: Concern about metrics is more about what is happening in client browsers than in the application server. The application under test here (“the-internet”) does not make use of separate authentication servers, a database, nor other back-end services.

  14. Run separate isolated scenarios to isolate transactions to study:

    1. Registration (to establish new users)
    2. Login (to load authentication)
    3. Menus and static pages (when users are exploring)
    4. User data entry filling out forms (editing and saving form data into a database)
    5. Client-side performance during batch reporting, backup, restore, or other back-end processing


  15. Identify the mix of transactions in a standardized run of different transactions at a time.

    PROTIP: This exercise stands up only one instance each and not multiple instances in a cluster for High Availability (HA).

  16. Identify optimal scale-up and scale-down (instance type) configurations
  17. Identify optimal scale-out and scale-in configurations

Automating the-internet using Selenium

Dave Haeffner spoke about his “the-internet” in 2015 part 1 and part 2 of “Selenium Test Automation: Practical Tips & Tricks” presentation recorded in Israel. In his May 2016 “How to use Selenium successfully” slidedeck he said:

    "An example application that captures prominent and ugly functionality found on the web. Perfect for writing automated acceptance tests against."

Ruby code to create “the-internet” is at https://github.com/tourdedave/the-internet

T.J. Myer wrote in his website June - July 2015 a series describing his adventures coding Selenium on Dave’s website:

  1. Sketch out the simple manipulation of a Login page
  2. Draft Common Utilities
  3. Storing Constants: static finals vs enums
  4. Storing Locators for Web Elements
  5. The Page Object Model
  6. Writing the Automated Test

References:

Many of the issues addressed above also need to be addressed by any app automation tool.

Challenges on The-Internet app

Click on to see the sample app’s UI on-line at
https://the-internet.herokuapp.com/
Click on flood.io Element script to view the Flood Element TypeScript at
https://github.com/flood-io/element/tree/master/examples/internet-herokuapp

Click on YouTube to view a video about manual actions and analysis of the UI page source code as the basis for Flood Element TypeScript creation.
Additional columns of icons may be added to show sample run results for each test item.

  1. flood.io Element scriptYouTube A/B Test Control (also known as split testing)
  2. flood.io Element script Add/Remove Elements
  3. flood.io Element script Basic Auth (Sign in Username and Password: admin)
  4. flood.io Element script Broken Images
  5. flood.io Element script Challenging DOM - this is the one impacting server resource
  6. flood.io Element script Checkboxes
  7. flood.io Element issue Context Menu
  8. flood.io Element script Digest Authentication (user and pass: admin)
  9. flood.io Element script Disappearing Elements
  10. flood.io Element issue Drag and Drop
  11. flood.io Element script Dropdown
  12. flood.io Element script Dynamic Content
  13. flood.io Element script Dynamic Controls
  14. flood.io Element scriptYouTube Dynamic Loading (using explicit wait for resilency)
  15. flood.io Element script Entry Ad
  16. flood.io Element issue Exit Intent
  17. flood.io Element scriptYouTube File Download (query HTTP HEADER first to make sure file contains something)
  18. flood.io Element script File Upload (issue in Element handling Windows vs Linux, also in Selenium)
  19. flood.io Element script Floating Menu
  20. flood.io Element scriptYouTube Forgot Password

  21. flood.io Element scriptYouTube Form Authentication (Login)
    assertion on message Logout (appears after login, not listed in the app’s menu)

  22. flood.io Element script Frames (Nested)
    flood.io Element script Frames (iFrames)
  23. flood.io Element script Geolocation
  24. flood.io Element script Horizontal Slider
  25. flood.io Element script Hovers
  26. flood.io Element script Infinite Scroll
  27. flood.io Element script Inputs
  28. flood.io Element script JQuery UI - Menus
  29. flood.io Element script JavaScript Alerts
  30. flood.io Element script JavaScript onload event error
  31. flood.io Element script Key Presses
  32. flood.io Element scriptYouTube Large & Deep DOM
  33. flood.io Element script Multiple Windows
  34. flood.io Element script Nested Frames
  35. flood.io Element scriptYouTube Notification Messages (Growl listener)
  36. flood.io Element script Redirect Link
  37. flood.io Element script Secure File Download
  38. flood.io Element script Shifting Content
  39. flood.io Element scriptYouTube Slow Resources (blacklist slow resource with proxy server)
  40. flood.io Element script (Sortable) Data Tables
  41. flood.io Element script Status Codes
  42. flood.io Element script Typos
  43. flood.io Element script WYSIWYG Editor

NOTE: Some scripts are under development.


Project folders from GitHub

These steps are done manually on your local machine.

  1. Create or navigate to a project folder for this effort.
  2. Clone automation scripts:

    git clone https://github.com/flood-io/element/tree/master/examples/internet-herokuapp
  3. Review Flood Element TypeScript coding.

  4. Review the installation scripts:

    See: https://github.com/flood-io/load-testing-playground/tree/master/element

Coding Flood Element Typescript

Typescript is a superset of the JavaScript programming language that control browsers. Flood Element makes use of the Typescript language, which transpiles to JavaScript. Typescript is used because its transpiler checks for errors sooner than with JavaScript.

QUESTION: Typescript

  • Typescript variables are statically typed (not dynamic as in JavaSript)
  • Typescript can be coded using object-oriented constructs

Rather than directly referencing application control IDs, so that when an app changes, just one change is needed to make all tests pass again. This is like Selenium YouTube page object helpers from http://bit.ly/po-html-elements Yandex and http://bit.ly/po-page-factory built into Selenium. YouTube Similarly, Base Page Object library to migrate commands from one version to another.

Run Element script from client CLI

An example summary report:

This flood simulated up to 1 users across 1 grid in us-east-1 for 7 minutes. The mean response time was 916ms with a standard deviation of 1,285 ms. The median was 169 ms and the 90th percentile was 3,010 ms. The maximum was 3,014 ms. A maximum of 12 rpm with a mean of 4 rpm was observed. 101 transactions passed with 0 failed.

Setup AWS manually

Based on https://wilsonmar.github.io/aws-onboarding

On an internet browser such as Google Chrome, Apple Safari, or Microsoft Edge:

  1. To limit financial exposure (to like $25 or whatever), buy a pre-paid reloadable Visa gift (debit) card pre-paid online (which has an expiration date and some have a monthly service fee). The Drawpay card provides a 1% refund on purchases. Others provide fee-Free cash withdrawal at over 25,000 MoneyPass ATMs
  2. Open AWS master account with email.
  3. In IAM, lock down master account.
  4. Create Security Group.
  5. In IAM, create service account.
  6. Define service account with permissions.
  7. Store key pair (credentials) for service account locally.
  8. Select your AWS region.

    Script A : Instantiate AWS Docker in EC2 build script

  9. Get to AWS EC2.
  10. Choose and AMI - Ubuntu 16.04 LTS
  11. Select EC2 instance type (t2.micro Free Tier eligible can handle up to 50 users), or “m5axlarge”.
  12. [10:23] Define Security Group add “All TCP Traffic”.
  13. Assign Key Pair name ___
  14. [2:29] Download Key Pair
  15. Save to file ???
  16. [2:37] Launch Instance
  17. PROTIP: Name instance “the-internet-app” so that files referring to this name (such as newrelicc-infra.yml) don’t have to be changed.

    Terminal - AWS Key Pair

  18. In Terminal store .pem file downloadd
  19. SSH into instance.

    Terminal - AWS Key Pair

    Inside “the-internet” terminal:

  20. [4:08] Within app server to hold “the-internet”, install prerequisites (GPG certs, Docker):

    sudo apt update
    sudo apt install apt-transport-https ca-certificates curl software-properties-common
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
    sudo apt update
    apt-cache policy docker-ce
    sudo apt install docker-ce
    # verify:
    sudo systemctl status docker
    sudo docker --version
    DOCKER_IMAGE="gprestes/the-internet"
    sudo docker pull "$DOCKER_IMAGE"
    sudo docker run -d -p 7080:5000 "$DOCKER_IMAGE"
    CONTAINER_ID=$( docker ps | grep "$DOCKER_IMAGE" | cut -d " " -f 1 )
    WANIP4=$( curl http://canhazip.com" )  # public IP4 address.
    
  21. [12:04] On a browser, verify external access to “the-internet” app using the external IP address from AWS, such as:

    http://52.91.73.157:7080/

    Setup “NewRelic” in a EC2 Ubuntu containing Docker

    In a browser:

  22. Login to AWS using your service account.
  23. Get to AWS EC2.
  24. Choose and AMI - Ubuntu 16.04 LTS
  25. Select EC2 instance type (t2.micro Free Tier eligible can handle up to 50 users), or “m5axlarge”.
  26. [10:23] Define Security Group add “All TCP Traffic”.
  27. Assign Key Pair name ___
  28. [2:29] Download Key Pair
  29. Save to file ??? [15:07] ssh -i “ubuntu.pem” ubuntu@ec2-18-208-170-2.compute-1.amazonaws.com
  30. [2:37] Launch Instance
  31. Name instance “NewRelic”

    Save NewRelic License Key

    Based on *

  32. [8:13] On the NewRelic web page Account Settings, highlight and save the License Key text

  33. Open file ./secrets.env

    NOTE: The secrets.env file is referenced in the docker-compose.yml file cloned from GitHub.

    docker-compose build
    docker-compose up
    
  34. [15:29] Update newrelicc-infra.yml with license_key value.

    Install NewRelic instrumentation agent

  35. [8:13] Switch back to the terminal
  36. TODO: Script to do this:

    # TODO: Replace with reference to secrets.env by docker-compose
    echo "license_key: a46bf7d3b4043cdfffcab3aaef677d29cc60d6be" | sudo tee -a /etc/newrelic-infra.yml
    curl https://download.newrelic.com/infrastructure_agent/gpg/newrelic-infra.gpg | sudo apt-key add -
     
    cat /etc/lsb-release 
    # [13:33] Based on NewRelic web page  
    # https://docs.newrelic.com/docs/infrastructure/new-relic-infrastructure/installation/
    printf "deb [arch=amd64] https://download.newrelic.com/infrastructure_agent/linux/apt bionic main" \
    | sudo tee -a /etc/apt/sources.list.d/newrelic-infra.list
    sudo apt-get update
    sudo apt-get install newrelic-infra -y
    # Verify: more /etc/newrelic-infra.yml
     
    # TODO: automate this:
    vi  /etc/newrelic-infra.yml
    sudo vi /etc/newrelic-infra.yml
     
    # TODO: Identify the Docker ID to a variable: 
    $CONTAINER_ID=$(sudo docker ps)
    # (value such as ba965ff40ef7)
    sudo docker exec -i -t "$CONTAINER_ID" /bin/bash
     
    # TODO: From inside NewRelic process:
    sudo docker cp ba965ff40ef7:/app/server.rb .
     
    # [19:31] Reboot:
    sudo systemctl restart newrelic-infra
    
  37. [19:57] Verify that NewRelic recognizes events from “the-internet-app” (subsituting the account number), such as:

    https://infrastructure.newrelic.com/accounts/2256749/hosts

    Install NewRelic agent in running “the-internet-app”

  38. [20:38] Get inside Docker container:

    # TODO: Identify the Docker ID to a variable: 
    $CONTAINER_ID=$(sudo docker ps)
    # (value such as ba965ff40ef7)
    sudo docker exec -i -t "$CONTAINER_ID" /bin/bash
    

    Based on https://docs.newrelic.com/docs/agents/ruby-agent/installation/install-new-relic-ruby-agent

  39. [23:17] Add gem 'newrelic_rpm' in Gemfile.

    # TODO: From inside NewRelic process (example root@ba965ff40ef7):
    sudo docker cp "$CONTAINER_ID:/app/server.rb" .
    sudo docker cp server.rb "$CONTAINER_ID:/app/"
    # Backup existing
    cp Gemfile Gemfile.backup
    # TODO: Automate 
    ??? gem 'newrelic_rpm
    # For the server.rb which is larger it’s not possible to echo and cat the file,
    # so I copied it locally and edit it to include the require 'newrelic_rpm'
    sudo docker cp ba965ff40ef7:/app/server.rb .
    # And copy it back to the container:
    sudo docker cp server.rb ba965ff40ef7:/app/
    
  40. Verify

    Script B : Add in server.rb and create new Docker image

    This is so the Docker image can be used for scaling.

    sudo docker run -d -p 7080:5000 ruby-bundle-update
     
    # TODO: Identify the Docker ID to a variable: 
    $CONTAINER_ID=$(sudo docker ps)
    # (value such as 363ddc8f7439)
    sudo docker exec -i -t "$CONTAINER_ID" /bin/bash
    # Save updated files:
    sudo docker cp server.rb "$CONTAINER_ID:/app/"
    sudo docker cp Gemfile "$CONTAINER_ID:/app/"
    # Get inside:
    sudo docker exec -i -t  "$CONTAINER_ID" /bin/bash
    sudo docker ps
    sudo docker commit "$CONTAINER_ID" ruby-bundle-update
    sudo docker stop "$CONTAINER_ID"
     
    sudo docker ps
    sudo docker run -d -p 7080:5000 ruby-bundle-update
    sudo docker ps
     
    # TODO: copy file newrelic.yml from external
    touch 1
    vi q
    vi 1
    mv 1 newrelic.yml
    vi newrelic.yml 
    ls -lart  # to verify manually
     
    # TODO: Identify the Docker ID to a variable: 
    $CONTAINER_ID=$(sudo docker ps)
    # (value such as 178e6dc45ab7)
     
    sudo docker cp newrelic.yml "$CONTAINER_ID:/app/"
    sudo docker commit "$CONTAINER_ID" final-version
    sudo docker stop "$CONTAINER_ID"
    # Verify:
    sudo docker images
    # Restart:
    sudo docker run -d -p 7080:5000 final-version
     
    sudo docker ps
    history > /tmp/history.file
    

    Update Element .ts script for flood.io

  41. Update the IP address in the script (several locations):

    await browser.visit('http://18.208.170.2:7080/')

    Instrument script for NewRelic

  42. Get license from newrelic.com
  43. Insert license into script

    NOTE: https://github.com/ThyWoof/geek-movie-shop

    Script C : Save instrumented Docker image to DockerHub for reuse

  44. Docker save

Perpare for Flood

  1. If you don’t have a flood.io account, get one (manually).
  2. Confirm your account via email.
  3. Log into Flood.io.
  4. Get license token.

Run flood against app in AWS under instrumentation

This step runs a shell script file at
https://github.com/wilsonmar/DevSecOps/master/flood-io/flood-run-e2e.sh

It is customized from an example in Flood docs.

  1. If you don’t have a GitHub account, get one.
  2. Use an internet browser to view:

    https://github.com/wilsonmar/DevSecOps/

  3. Click “Fork” button to copy the repository under your own account.

  4. Open a Terminal.
  5. Navigate to the containing folder where GitHub creates folders. Make a folders as necessary.

  6. Clone the whole DevSecOps repo:

    git clone https://github.com/wilsonmar/DevSecOps/ 
  7. Navigate into the folder:

    cd DevSecOps
    cd flood-io
  8. Edit the environment file using your favorite editor, such as:

    nano flood-env.sh
  9. Insert the license token from flood.io.
  10. Save the file
  11. Run the environment file to load variables into memory.

  12. Edit the script using your favorite editor, such as:

    nano flood-run-e2e.sh
  13. Define other parameters: run conditions.
  14. Validate run pre-conditions.

  15. Initiate run.
  16. Stop run.
  17. Collect run results.
  18. Analyze run results / Generate visualizations.
  19. Display summary statistics.

  20. Git add, commit, and push the changed script to GitHub.

Config New Relic Dashboard

Questions about several other dimensions, such as:

  • What is the impact on the cloud bill (costs) of that cool JavaScript UI code?
  • What is the capacity of a chosen instance type (such as the free tier t2.micro)?
  • How much more is needed to run the anticipated peak load?
  • What happens when that peak load is exceeded?

https://docs.newrelic.com/docs/insights/use-insights-ui/manage-dashboards https://learn.newrelic.com/get-started-with-apm https://learn.newrelic.com/dashboards-and-data-apps https://docs.newrelic.com/docs/plugins/plugins-new-relic/custom-dashboards-custom-views https://blog.newrelic.com/product-news/steal-this-dashboard/

Other APM

A full list of APM tools:

  • New Relic
  • Dynatrace OneAgent
  • Dynatrace AppMon
  • AppDynamics
  • NudgeAPM

Security scans

In today’s ransomware enviornment, we all need to be extra vigilant to ensure security.

  1. Install https://inspec.io (created by Chef)

  2. Install https://github.com/docker/doccker-bench-security

  3. Install CIS (Center for Internet Security) benchmarks for specific distributions and versions of Linux:

    • Distribution independent Linux
    • Debian Linux 8
    • Ubuntu Linux 16.04 LTS
    • Amazon Linux 2
    • Centos Linux 7
    • Oracle Linux 7
    • Red Hat Enterprise Linux 7
    • SUSE Linux Enterprise 12

The Linux Audit Framework is used to identify potential security weaknesses or policy violations

  1. Install

    sudo apt install auditd
    pidof auditd
  2. Add rules and list them

    sudo auditctl -w /usr/bin/dockerd -k docker
    sudo auditctl -l
  3. Turn auditing on.
  4. Analyze report

    sudo aureport

Securing the Docker Platform by Nigel Brown Released 21 Jun 2018


Resources