AWS X-Ray

Observability: Tracing within the AWS Cloud

Overview

Get on AWS
Competitors
Architecture
More on Amazon

This tutorial is about AWS X-Ray, which trace interactions among independent and autonomous distributed services to identify where errors and delays are occuring in production.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

First, let’s clarify differences among the various monitoring-related services from Amazon:

CloudWatch saves logs sent from custom apps STDOUT and STDERR, for debugging
CloudTrail saves every action performed by AWS, for auditing
X-Ray tracing

Get on AWS

If you don’t already have an AWS account, …

Competitors

Zipkin OpenTracing

https://www.youtube.com/watch?time_continue=1571&v=JQPOPV_VH5w Kelsey Hightower

Architecture

In IAM Roles for the EC2 worker node instance, Attach policy and Add the AWSXRayDaemonWriteAccess managed policy so worker nodes can make API calls.

The AWSXRayWriteOnlyAccess managed policy

The AWSXRayReadOnlyAccess managed policy

Build the X-Ray daemon Docker image using the Dockerfile from

https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/Dockerfile

FROM amazonlinux:1
 
# Download latest 2.x release of X-Ray daemon
RUN yum install -y unzip && \
 cd /tmp/ && \
 curl https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-2.x.zip > aws-xray-daemon-linux-2.x.zip && \
 unzip aws-xray-daemon-linux-2.x.zip && \
 cp xray /usr/bin/xray && \
 rm aws-xray-daemon-linux-2.x.zip && \
 rm cfg.yaml
 
# Expose port 2000 on udp
EXPOSE 2000/udp
 
ENTRYPOINT ["/usr/bin/xray"]
 
# No cmd line parameters, use default configuration
CMD ['']

Notice the daemon listens by default on UDP port 2000, gathers raw segment data, and relays them to the AWS X-Ray cloud service’s API.

docker build -t xray-daemon .

Add instrumentation code into you custom app code by importing the X-Ray SDK for your programming language.

https://github.com/linuxacademy/eks-deep-dive-2019/tree/master/4-3-XRay/demo-app

var XRay = require('aws-xray-sdk');            // Initialize X-ray SDK
var AWS = XRay.captureAWS(require('aws-sdk')); // Capture all AWS SDK calls
var http = XRay.captureHTTPs(require('http')); // Capture all HTTP/HTTPS calls

A sample “service-a” app sends to “service-b”, which, after random intervals, sends good and error responses back.

To begin capturing segments (units of work):

XRay.config([XRay.plugins.EC2Plugin, XRay.plugins.ECSPlugin]);
XRay.middleware.enableDynamicNaming();
 
// Start capturing the calls in the application
app.use(XRay.express.openSegment('service-a'));

Before exiting, stop capturing calls:

// Stop capturing the calls in the application
app.use(XRay.express.closeSegment());

Create a repository to hold segments captured:

aws ecr create-repository --repository-name xray-daemon

Create a repository to hold segments captured:

docker tag xray-daemon:latest 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest

docker push 123456778.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest

Edit the sample daemonset.yam to define the AWS account ID at https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/xray-k8s-daemonset.yaml
```
   containers:
   - name: xray-daemon
     image: 1232456789012.dkr.ecr.us-east-1.amazonaws.com/xray-daemon:latest
```

Apply the configuration

kubectl apply -f xray-k8s-daemonset.yaml

View the deamonset
```
kubectl describe daemonset xray-daemon
```
View logs
```
kubectl logs -l app=xray-daemon
```
Build service-a
```
docker build -t service-a
```
Build service-b
```
docker build -t service-b
```
Configure the X-Ray Daemonset running in all ECS clusters.

https://github.com/linuxacademy/eks-deep-dive-2019/blob/master/4-3-XRay/demo-app/k8s-deploy.yaml
Each segment collected is stored in XML format, and includes request request and response timing.
The AWS X-Ray Console displays statistical sampling of individual segments grouped together into a “trace”.

PRICING: To encourage adoption, traces recorded cost $5 per million ($0.000005 per trace) after the first free 100,000 traces each month. Traces retrieved or scanned cost $0.50 per million ($0.0000005 per trace) after the first 1,000,000 traces each month.
The “service map” provides an aggregate view of traces.

More on Amazon

This is one of a series about Amazon:

Wilson Mar

AWS X-Ray

Get on AWS

Competitors

Architecture

More on Amazon

You might also enjoy (View all posts)