Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


Observability: Tracing within the AWS Cloud

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


This tutorial is about AWS X-Ray, which trace interactions among independent and autonomous distributed services to identify where errors and delays are occuring in production.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

First, let’s clarify differences among the various monitoring-related services from Amazon:

  • CloudWatch saves logs sent from custom apps STDOUT and STDERR, for debugging
  • CloudTrail saves every action performed by AWS, for auditing
  • X-Ray tracing

Get on AWS

If you don’t already have an AWS account, …


Zipkin OpenTracing

https://www.youtube.com/watch?time_continue=1571&v=JQPOPV_VH5w Kelsey Hightower



  1. In IAM Roles for the EC2 worker node instance, Attach policy and Add the AWSXRayDaemonWriteAccess managed policy so worker nodes can make API calls.

    The AWSXRayWriteOnlyAccess managed policy

    The AWSXRayReadOnlyAccess managed policy

  2. Build the X-Ray daemon Docker image using the Dockerfile from


    FROM amazonlinux:1
    # Download latest 2.x release of X-Ray daemon
    RUN yum install -y unzip && \
     cd /tmp/ && \
     curl https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-2.x.zip > aws-xray-daemon-linux-2.x.zip && \
     unzip aws-xray-daemon-linux-2.x.zip && \
     cp xray /usr/bin/xray && \
     rm aws-xray-daemon-linux-2.x.zip && \
     rm cfg.yaml
    # Expose port 2000 on udp
    EXPOSE 2000/udp
    ENTRYPOINT ["/usr/bin/xray"]
    # No cmd line parameters, use default configuration
    CMD ['']

    Notice the daemon listens by default on UDP port 2000, gathers raw segment data, and relays them to the AWS X-Ray cloud service’s API.

    docker build -t xray-daemon .
  3. Add instrumentation code into you custom app code by importing the X-Ray SDK for your programming language.


    var XRay = require('aws-xray-sdk');            // Initialize X-ray SDK
    var AWS = XRay.captureAWS(require('aws-sdk')); // Capture all AWS SDK calls
    var http = XRay.captureHTTPs(require('http')); // Capture all HTTP/HTTPS calls

    A sample “service-a” app sends to “service-b”, which, after random intervals, sends good and error responses back.

    To begin capturing segments (units of work):

    XRay.config([XRay.plugins.EC2Plugin, XRay.plugins.ECSPlugin]);
    // Start capturing the calls in the application

    Before exiting, stop capturing calls:

    // Stop capturing the calls in the application
  4. Create a repository to hold segments captured:

    aws ecr create-repository --repository-name xray-daemon
  5. Create a repository to hold segments captured:

    docker tag xray-daemon:latest 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
    docker push 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
  6. Edit the sample daemonset.yam to define the AWS account ID at https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/xray-k8s-daemonset.yaml

       - name: xray-daemon
         image: 1232456789012.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
  7. Apply the configuration

    kubectl apply -f xray-k8s-daemonset.yaml
  8. View the deamonset

    kubectl describe daemonset xray-daemon
  9. View logs

    kubectl logs -l app=xray-daemon
  10. Build service-a

    docker build -t service-a
  11. Build service-b

    docker build -t service-b
  12. Configure the X-Ray Daemonset running in all ECS clusters.


  13. Each segment collected is stored in XML format, and includes request request and response timing.

  14. The AWS X-Ray Console displays statistical sampling of individual segments grouped together into a “trace”.

    PRICING: To encourage adoption, traces recorded cost $5 per million ($0.000005 per trace) after the first free 100,000 traces each month. Traces retrieved or scanned cost $0.50 per million ($0.0000005 per trace) after the first 1,000,000 traces each month.

  15. The “service map” provides an aggregate view of traces.

More on Amazon

This is one of a series about Amazon: