Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Tracing within the AWS Cloud

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This tutorial is about AWS X-Ray, which trace interactions among independent and autonomous distributed services to identify where errors and delays are occuring in production.

First, let’s clarify differences among the various monitoring-related services from Amazon:

  • CloudWatch saves logs sent from custom apps STDOUT and STDERR, for debugging
  • CloudTrail saves every action performed by AWS, for auditing
  • X-Ray tracing

Get on AWS

If you don’t already have an AWS account, …

Competitors

Zipkin OpenTracing

https://www.youtube.com/watch?time_continue=1571&v=JQPOPV_VH5w Kelsey Hightower

Architecture

aws-xray-475x349-20026.jpg

  1. In IAM Roles for the EC2 worker node instance, Attach policy and Add the AWSXRayDaemonWriteAccess managed policy so worker nodes can make API calls.

    The AWSXRayWriteOnlyAccess managed policy

    The AWSXRayReadOnlyAccess managed policy

  2. Build the X-Ray daemon Docker image using the Dockerfile from

    https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/Dockerfile

    FROM amazonlinux:1
     
    # Download latest 2.x release of X-Ray daemon
    RUN yum install -y unzip && \
     cd /tmp/ && \
     curl https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-2.x.zip > aws-xray-daemon-linux-2.x.zip && \
     unzip aws-xray-daemon-linux-2.x.zip && \
     cp xray /usr/bin/xray && \
     rm aws-xray-daemon-linux-2.x.zip && \
     rm cfg.yaml
     
    # Expose port 2000 on udp
    EXPOSE 2000/udp
     
    ENTRYPOINT ["/usr/bin/xray"]
     
    # No cmd line parameters, use default configuration
    CMD ['']
    

    Notice the daemon listens by default on UDP port 2000, gathers raw segment data, and relays them to the AWS X-Ray cloud service’s API.

    docker build -t xray-daemon .
  3. Add instrumentation code into you custom app code by importing the X-Ray SDK for your programming language.

    https://github.com/linuxacademy/eks-deep-dive-2019/tree/master/4-3-XRay/demo-app

    var XRay = require('aws-xray-sdk');            // Initialize X-ray SDK
    var AWS = XRay.captureAWS(require('aws-sdk')); // Capture all AWS SDK calls
    var http = XRay.captureHTTPs(require('http')); // Capture all HTTP/HTTPS calls
    

    A sample “service-a” app sends to “service-b”, which, after random intervals, sends good and error responses back.

    To begin capturing segments (units of work):

    XRay.config([XRay.plugins.EC2Plugin, XRay.plugins.ECSPlugin]);
    XRay.middleware.enableDynamicNaming();
     
    // Start capturing the calls in the application
    app.use(XRay.express.openSegment('service-a'));
    

    Before exiting, stop capturing calls:

    // Stop capturing the calls in the application
    app.use(XRay.express.closeSegment());
    
  4. Create a repository to hold segments captured:

    aws ecr create-repository --repository-name xray-daemon
  5. Create a repository to hold segments captured:

    docker tag xray-daemon:latest 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
    docker push 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
  6. Edit the sample daemonset.yam to define the AWS account ID at https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/xray-k8s-daemonset.yaml

       containers:
       - name: xray-daemon
         image: 1232456789012.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
    
  7. Apply the configuration

    kubectl apply -f xray-k8s-daemonset.yaml
  8. View the deamonset

    kubectl describe daemonset xray-daemon
  9. View logs

    kubectl logs -l app=xray-daemon
  10. Build service-a

    docker build -t service-a
  11. Build service-b

    docker build -t service-b
  12. Configure the X-Ray Daemonset running in all ECS clusters.

    https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/demo-app/k8s-deploy.yaml

  13. Each segment collected is stored in XML format, and includes request request and response timing.

  14. The AWS X-Ray Console displays statistical sampling of individual segments grouped together into a “trace”.

    PRICING: To encourage adoption, traces recorded cost $5 per million ($0.000005 per trace) after the first free 100,000 traces each month. Traces retrieved or scanned cost $0.50 per million ($0.0000005 per trace) after the first 1,000,000 traces each month.

  15. The “service map” provides an aggregate view of traces.

More on Amazon

This is one of a series about Amazon: