Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

https://wilsonmar.github.io/aws-lambda

Kinda like Java Lambdas, but not really


Overview

This tutorial provides a deep dive into the basics of creating and using functions within AWS Lambda. This is a hands-on guided tour. Take one step at a time and we point out PROTIPs and notes along the way.

  1. Use an internet browser to get on the AWS Console at
    http://aws.amazon.com

    BTW, competitors to AWS Lambda include:

    TODO: I hope to have a competitive comparison in the future.

  2. Click Sign In to the AWS Lambda Console. If you have signed in before, the URL changes to include your working region.

    https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2

    NOTE: AWS Lambda acts like a giant single server. But you get to pick which Availability Zone.

  3. Change the region if you need.

  4. Select Services from the top menu.

    COMMENT: Compute services are at the top of the list among all Amazon services because it’s the hottest thing right now (June 2016)?

  5. Select Lambda from among Compute services.

    If no Lambda functions have been created, you will see a “Get Started Now” button.

    If custom Lambda functions have been created on your account, you’ll see a list of those functions.

    No worries about achieving massive scale. Addition of enough machines to handle load are taken care of by Amazon people behind the scenes.

    BTW, if you’re worried about vendor lock-in, know that Amazon is not the only ones who can run lambdas. IronWorker from Iron.io and Serverless can run Lambdas on your own servers.

    Pricing

  6. On a new browser tab, click
    http://aws.amazon.com/lambda/, the home page for Amazon’s Lambda service.

  7. Click http://aws.amazon.com/lambda/pricing

    It says the first million requests are free. That’s 20 cents you’ll save each month.

    For now, the first 400,000 GB-seconds (x 1024 = 409,600,000 MB-seconds) are free.

    @lambdatips: If you run only 128 MB Lambdas, you can make one request every 1.23 seconds during a 30-day month, for free. (409,600,000 / 128 = 3,200,000 / 30 days / 24 hours / 60 minutes / 60 seconds = 1.23 )

    QUESTION: What is the date that free allocations flip to the next month? On the 1st?

    NOTE: Lambda functions created (and sitting around with no activity) incur no charges.

    On-line TOOL: Pricing Calculator at
    https://s3.amazonaws.com/lambda-tools/pricing-calculator.html

    NOTE: How much each Lambda request consumes in memory time is described in CloudWatch, below.

Why?

Andrew Baird, AWS Solutions Architect, listed all the questions that developers DON’T have to graple with in his March 2016 webinar “Getting Started with Serverless Architectures”.

“There are tools and entire industries whose entire value proposition are about answering just one of these.”

Why NOT?

An app is NOT a good candidate for using AWS Lambda if you interact with components in EC2 instances:

  • App requires a custom AMI (extra programs on servers rather than via REST API)
  • Need to SSH into server using multiple accounts
  • Need to use LDAP for authentication
  • OS updates impact your app?
  • OS configuration such as memory settings, max. file handlers, etc.
  • OS snapshots during security incidents
  • Intrusion detection is needed (not available in AWS Lambda)
  • Network speeds are important (not known in AWS Lambda)
  • VPC is needed (not yet supported in AWS Lambda)

Create a Lambda function

There are several options for creating Lambda functions:

  1. Click “Create a Lambda Function”.

Source of code

There are several ways to get programming code into AWS Lambda:


Select Blueprint of Amazon-defined code - Hello JSON

Function name

  1. Click the right-arrow to Page forward if you don’t see “Hello World”.

    PROTIP: Highlight the blueprint’s name (in bold letters) and copy it to your Clipboard so your can paste it into the function name during the next step.

  2. PROTIP: Use your mouse to highlight the function name for use in a later step.

  3. Click on the name “Hello World” Node.js function.

    Amazon calls Lambda a “compute service” because programmers write code as discrete API handler functions responding to events such as an image being uploaded into S3.

    'use strict';
    console.log('Loading function');
    
    exports.handler = (event, context, callback) => {
       //console.log('Received event:', JSON.stringify(event, null, 2));
     console.log('value1 =', event.key1);
     console.log('value2 =', event.key2);
     console.log('value3 =', event.key3);
     callback(null, event.key1);  // Echo back the first key value
     // callback('Something went wrong');
    };
    

    NOTE: The callback function is like a return statement. The callback value is the JSON file returned after asynchronous execution.

    Additional observations about Node.js programming is at AWS Lambda Node JavaScript Programming.

  4. PROTIP: Construct a function name with more metadata, like this example:

    learn1-hello-json-node43-v01
    
    • Prefix the function with a project name (such as “learn1”).
    • Use dashes or underscores instead of spaces (which are not allowed in the name).
    • Include the use case (“hello-json”) for Hello World receiving key-value pairs in a static JSON file.
    • Include in the name the language and its version (such as node43 for Node.js 4.3).
    • Specify a version number (v01) for different versions you want to keep simultaneously. (Git can keep history of alterations to the same version)

    Description

    PROTIP: In the description, put in a URL to a wiki ?

    Runtime

    The code associated with each blueprint is for a particular runtime. So runtime isn’t a choice that can be changed on this form.

    Run-times exist within a Amazon Linux AMI. More specifically, this page describes the specific Linux server and utility libraries used in each region. (ami-f0091d91) in Oregon is amazon/amzn-ami-hvm-2015.09.1.x86_64-gp2 with Linux kernel 4.1.19-24.31.amzn1.x86_64.

    Linux AMI 2014.09 Packages - source RPMs

    • AWS SDK for JavaScript version 2.3.8
    • Python images contain AWS SDK for Python (Boto 3) version 1.3.1.

    WARNING: These underlying versions can change at anytime, unannounced. So have a way of being notified if errors are detected in run logs.

    Scroll down beyond the script to more input fields.

    Handler

    NOTE: index.handler specifies the default index module.

    Leave the default alone for now.

    Role

  5. If empty, click on the Role field for a list:

    The Execution role defines the permissions.

  6. Select Basic execution role if your function does not access input data from S3, Dynamo DB, Kinesis, etc.

    This example is for performing CloudWatch actions on logs:

      { "Version": "2012-10-17", "Statement": [
        { "Effect": "Allow",
          "Action": [ "logs:*" ],
          "Resource": "arn:aws:logs:*:*:*" }
        ]
      

    NOTE: AWS IAM provides for roles based on specific resources.

  7. Click Allow in the pop-up browser window and be returned to the Lambda definition.

    Memory

    PROTIP: Initially select the lowest memory (128 MB) and add more as necessary.

    NOTE: Billing is in minimum 128 MB increments even if less memory was actually used.

    PROTIP: Keep input and output data of a known small size. Memory used to hold input and output data is included in the memory used. Uncertainties about the size of data used require a larger allocation.

    1,536 MB is the current maximum memory size.

    CAUTION: If a function needs more memory than allocated, it would fail, and AWS would still charge you for the allocated memory.

    Timeout

    The longest allowed is 5 minutes.

    But the maximum execution duration per request is 300 seconds.

    NOTE: AWS Lambda is a multi-tenancy environment much like what Salesforce.com provides developers. Different developers (who don’t know each other) use the same physical server. So limits are necessary to keep one function to affect all others.

    VPC

    Our initial example does not use the internet.

    Save and Test

  8. Click Next.

  9. Review, then click Create function.

  10. Click Test.

    Input Test Event

  11. Select from Sample event template. The default is “Hello World”.

  12. Click Save and test.

    On first invocation is a pop-up Input test event. This can later be obtained by Actions > Configure test event.

    Input test sample event data template

    NOTE: Events provide input into the function.

  13. Click Test. If no input data has been defined, the Input test event form appears in a pop-up.

    The default data that appears is from the Hello World sample event template.

    CAUTION: Selection of “Hello World” from the Sample event template replaces what has been saved for the current Lambda.

  14. Scroll down the list to marvel at other options to trigger the function:

    lambda test sample 2016-06-03

    • API Gateway Authorizer
    • CloudFormation Create REPORT
    • SES Email receiving
    • CloudWatch Logs
    • SNS

    • DynamoDB Update
    • Cognito Sync trigger
    • Kinesis
    • S3 Put
    • S3 Delete

    • Alexa Start Session
    • Alexa End Session
    • Alexa Smart Home - Control
    • Alexa Intent - MyColorIs

    • Hello World
    • Mobile Backend

  15. Make a change to “value1”.

    {
      "key3": "value3",
      "key2": "value2",
      "key1": "value1"
    }
    

    NOTE: Each Key-Value pair (KVP) is called a “structure” (two pieces of data together as a single object). Keys used to lookup values are also called “identifiers”.

  16. Click Save to exit the modal dialog.

    Action: Configure test event

  17. Return to the “Input test event” dialog by selecting from the Actions dropdown “Configure test event”.

  18. Click Save and test to exit the modal dialog and run the function.

    Execution result

  19. Scroll down the page for the “Execution result” section.

    The result shows “value1” because of this code line:

    callback(null, event.key1);  // Echo back the first key value
    

    Variable event.key1 in the script refers to “key1” and its value “value1” in the JSON file above.

    CloudWatch Logs

    Notice that output from the initial console.log('Loading function'); does not appear in the run Log Output.

  20. Click the Monitoring tab.

  21. PROTIP: Mouse over a point on a line for more detail.

    • Invocation count measures the number of times a function has been invoked and billed (in response to an event or API call). This includes both successful and failed invocations, but not throttled attempts.

    • Invocation duration measures the elasped wall clock time from when the function code starts executing (as a result of an invocation) to when it stops executing. This is used in billing, rounded up to the nearest 100 milliseconds. The maximum data point value possible is the function timeout configuration.

    • Invocation errors measure the number of invocations that failed due to some error. Failed invocations may trigger a retry attempt that succeeds:

      • Handling exceptions such as context.fail(error).
      • Unhandled exceptions causing the code to exit the function since it can’t handle the error.
      • Out of memory exceptions
      • Timeouts
      • Permission errors

    • Throttled invocations measures the number of Lambda functon invocation attempts not executed because the customer concurrent limit has been reached for the period (error 429).

    NOTE: AWS Lambda only sends non-zero value metrics to CloudWatch.

  22. Click View logs in CloudWatch link.
  23. Click a Log stream entry to see the Event Data for it.

    Notice the time stamp such as “Last modified date2016-06-03T13:23:27.021+0000” is UTC/GMT time (London with no Summertime).

  24. Click the line that begins with “REPORT RequestId:” to expand it to see the amount of memory actually used.

    REPORT RequestId: 0ddf5949-29b1-11e6-b8de-a70c7c47033a Duration: 1.00 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 36 MB
    

    These statistics also appear in the Summary pane.

    The amount charged (Billed) is in increments of 100 milliseconds, even if the Duration is 1 ms.

    The number of Max Memory Used used can vary from one execution to another for different sizes of files handled by the function.

    QUESTION: Can the highest MB Used by the function be a metric shown in the list of functions (along with Max. memory)?

    QUESTION: Are charges for memory based on the maximum allocated or the amount actually used?

  25. In the CloudWatch Console, click “Lambda” under the Metrics category on the left menu.

    CloudWatch Log stream items

  26. Click on a log stream item. Notice the “Loading function” there.

  27. Click on the Lambda Management Console to return.

    PROTIP: AWS has a free tier of several gigabytes of log storage. Set the log retention time to the amount of time to fit your budget.

Programming model

The full programming model for Lambdas consists of:

  • Handlers (shown above)
  • Logging (shown above)
  • Context Objects (next)
  • Exceptions

Context objects

  1. Uncomment the first console.log with stringify and add these console.log lines from docs about the Context object:

    'use strict';
    console.log('Loading function');
    exports.handler = (event, context, callback) => {
       console.log('Received event:', JSON.stringify(event, null, 2));
       console.log("context.awsRequestId: "  + context.awsRequestId);
       console.log("context.logGroupName: "  + context.logGroupName);
       console.log("context.functionName: "  + context.functionName);
       console.log("context.logStreamName: " + context.logStreamName);
       console.log('context.getRemainingTimeInMillis: ', context.getRemainingTimeInMillis());
       console.log("context.identity: "      + context.identity);
       console.log('clientContext =', context.clientContext);
       if (typeof context.identity !== 'undefined') {
           console.log('Cognito identity ID =', context.identity.cognitoIdentityId);
       }    
       callback(null, event.key1);  // Echo back the first key value
    // callback('Something went wrong');
    };
     
  2. Run again and view the CloudWatch log.

    • The RequestId is shown in the log without being asked.
    • The logGroupname is the name of the function, such as “/aws/lambda/learn1-context-node43-v01”.
    • The functionName does not include the “/aws/lambda” path.
    • The logStreamName is the input, such as “2016/06/13/[$LATEST]f64505ba061e4ecea26d59abeb513f72”.
    • The RemainingTime is milliseconds before timeout occurs. Kinda like “headroom”.
    • The identity is unidentified.

Exception handling

See http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-mode-exceptions.html

Functions context.done(), context.succeed(), and context.fail() have been deprecated in favor of the callback() function. These v0.10.36 context functions still work in v4.3.2, but are no longer recommended.

  1. Change AWS_RUNTIME value to nodejs in the .env file.

Create Event Source S3

See https://github.com/awslabs/lambda-refarch-fileprocessing

This is an example of a “push” model where Lambda is triggered by an event external to it.

Prior to AWS Lambda, servers had to “ping” S3 frequently to identify when a file has been added.

The Introduction to AWS Lambda course in qwiklabs.com provides step-by-step instruction on creating an Lambda Event Source triggered by an upload to S3 bucket.

  1. Open another tab on your browser to create a bucket in S3 with a name (such as “lambdabkt001”).

    WARNING: Bucket names must be unique among all S3 users.

    TUTORIAL NOTE: All links on this page are for Oregon region us-west-2.

    For now, no logging, versioning, replication, tags, etc. for the bucket.

    Create Lambda Function

    On Dec 9, 2014 Dominick Peluso talks about his Lambda that, when it detects an image has been upload, it uses ImageMagick to create a thumbnail in S3, then sends an email.



  2. On the AWS Console, select Services, select Lambda, click Get Started Now

  3. Click the All languages drop-down. The default is Node.JS 4.3.

    But also available are Java 8, Node.JS 0.10, Node.JS 4.3, Python 2.7.

    COOL: With Lambda developers don’t need to provision servers and install and update JVM run-time software. Servers accepting Lambda requests are Node JS, Python, and JVM run-times that run Lambda functions.

  4. If you want to use, for example, blueprint “s3-get-object”, type “s3” in the Filter field to select a blueprint (sample configurations of event sources and Lambda functions).

    Alternately, to provide your own script, scroll down to click Skip.

  5. S3 is the default among event sources:

  6. Event Type “Object Created (All)” rather than specific action:

  7. Click Next. For now, no file prefix or suffix specification.

    Configure function

    The default Description is supplied from the blueprint.

  8. Select “Node.j2 4.3” for Runtime.

    Sample code for the language version selected appears. NOTE: Yes, that’s 2006:

    let aws = require('aws-sdk');
    let s3 = new aws.S3({ apiVersion: '2006-03-01' });
    

    Alternately:

    • If you selected Edit code inline, paste code copied from a Gist or GitHub.
    • If you selected Upload a ZIP file, navigate to the file on your machine.

      WARNING: Each upload is limited to under 50 MB (compressed), 10 MB in uploads.

    • If you selected Upload a file from Amazon S3, paste in the bucket ID.

    PROTIP: Install the AWS Eclipse plugin to author and deploy Lambda functions for Java and Node.js.

    NOTE: The file holding the code is index.js.

  9. The Event Source Type would be “S3”.
  10. The Event Type would be “Object Created (All)”.

  11. Leave Handler as “index.handler”.

    “index” Lambda calls a module in error messages.

  12. The Execution role defines the permissions.

    If you are usign S3, select * S3 execution role. The asterisk (*) designates the default selection.

    Alternately, select Basic execution role if your function does not need to input data.

  13. Click View Policy Document for accessing S3. An example:

    {
      "Version": "2012-10-17",
      "Statement" : [
      { "Effect" : "Allow",
        "Action" : [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ],
        "Resource" : "arn:aws:logs:*:*:*"
      },
      { "Effect" : "Allow", "Action" : [ "s3: GetObject" ],
        "Resource" : "arn:aws:s3::: your-bucket-name"
      }
    ]
     }
    
  14. Click Allow in the IAM window opened for you to accept the role name suggested.

  15. PROTIP: Start with 128 MB of memory (the lowest), and increase it later.

  16. Set Timeout to 5 seconds.

  17. No VPC during tutorial runs. Click Next.

  18. Click Enable event source until testing has occurred. Click Create function.

    Test Trigger S3

  19. Switch back to the S3 tab.
  20. Click on the bucket link.
  21. Click Upload tab.
  22. Click Add Files.
  23. Select any file from your local machine’s hard drive.
  24. Click Start Upload.

    Done.

Test Trigger Custom

  1. In the Lambda Functions screen, select Actions since Test should not be clicked until it’s configured.

  2. The default data that appears is from the Hello World sample event template. But there are others:

  3. If your function does not use test data, save an empty set of JSON curly braces:

    { }
    
  4. Click Test. Scroll to the bottom of the screen to see the Execution result.

Create Lambda function with no input data for API

  1. In an internet browser go to Services > Compute > Lambda
    https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions

  2. Create an Amazon Lambda function to work with if you haven’t already.
  3. Click on the name of a Lambda function to expose using an API endpoint.

    Notice a set of numbers and letters (such as tsdwtdl0r1) has been assigned next to your Lambda name.

  4. Click the API endpoints tab.
  5. Click Add API endpoint.
  6. Select API Gateway in the pop-up dialog.

    QUESTION: Why is this necessary if there is only one choice?

  7. Change the Resource Name (such as /faq) to define part of the URL path.
  8. The initial Method must be GET.
  9. Change Deployment stage to test.
  10. Select Open for Security (no tracking by access key or authentication with IAM).

    The screen before clicking.

  11. Click Submit to have a URL assigned, such as:

    https://tsdwtdl0r1.execute-api.us-west-2.amazonaws.com/prod/faq

Dynamo DB Pull

This is an example of a “pull model” where Lambda watches a data stream (DynamoDB or Kinesis streams). A change event triggers additional actions.

https://www.youtube.com/watch?v=_-hnJC4IXJI

To build mobile back-ends that retrieve and transform data from Amazon DynamoDB:

  • https://github.com/abalone0204/serverless-demo-with-dynamodb-node

  • https://www.youtube.com/watch?v=TuGyyTXPQ-U1 How to use AWS API Gateway to expose AWS DynamoDB database backend by mattua

  • http:// blog.import.io/ post/ using-amazon-lambda-and-api-gateway

  • https://www.youtube.com/watch?v=XByPxb_VvpY AWS re:Invent 2015 | (DEV303) Practical DynamoDB Programming in Java

Kinesis streams

Server-less processing of streaming data using Amazon Kinesis.

Python program

The most basic lambda in Python:

def lambda_handler(event, context):
  return "Hello World!"
   

The most basic lambda in Java:

package example;

import com.amazonaws.services.lambda.runtime.Context;

public class Hello {
    public String lambdaHandler(String event, Context context) {
        return "Hello World!";
    }
}
   

AWS CLI

Lambdas can be invoked from a command-line window or shell script.

This example provides an input.txt file containing JSON text:

aws lambda invoke-async --function-name learn1-hello-world1-node43-v01 --region us-west-2 --invoke-args input.txt --profile adminuser

Return:

{ “Status”: 202 }

CloudWatch Command-line Tool

NOTE: CloudFront logs can also be read using command-line tool awslogs.

CloudFormation Deploy

https://github.com/jorgebastida/gordon λ Gordon is a tool to create, wire and deploy AWS Lambdas using CloudFormation.

http://gordon.readthedocs.io/en/latest/ provides examples in Java, JavaScript (for Node), and Python.

Create EC2 instance from Lambda

An example to start an EC2 instance:

var AWS = require('aws-sdk');
exports.handler = function(event, context) {
   var ec2 = new AWS.EC2({region: 'us-east-1'});
   ec2.startInstances({InstanceIds : ['i-0114833f8ffc9151c'] },function (err, data) {
      if (err) console.log(err, err.stack);
      else console.log(data);
      context.done(err,data);
   });
};

In-built libraries

ImageMagick

ImageMagick is built-in to Lambda

  • http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html mentioned among available libraries.
* https://www.npmjs.com/package/imagemagick
  wrapper around the imagemagick CLI.

Use of /tmp folder

AWS Lambda projects shared

Some Lambda developers share their stories:

Lena and Sergey Barinova blogged about how she used AWS, React, and Node.js to build her Price Tracker website hosted in Amazon S3. Code for it is exposed at https://github.com/LenaBarinova/PriceTracker, which include Gulp tasks and Mocha tests. Concern is expressed about the latency involved in data retrieval from DynamoDB.



John Stamper, AWS Solution Architect, wrote Fanout S3 Event Notifications to Multiple Endpoints 24 JUL 2015



Daniele Stroppa, AWS Solution Architect, described how he uses Jenkins and Grunt to deploy AWS Lambdas 09 JUL 2015.

David Barnett wrote AWS LAMBDA TUTORIAL: PLAYING MATH GAMES WITH LAMBDA May 24 2016

 * Visualizing the Mandelbrot sets
 * Playing Monte Carlo simulations
 * Visualizing Apache Spark and Databricks

Russ Matney: What I am using AWS Lambda for

  * gif-to-mp4
  * create-timelapse
  * file-to-png
  * pngs-to-mp4
  * mp4s-to-timelapse
  * upload-to-vimeo

AWS Lambda Tools

  • BLESS from Netflix is an SSH Certificate Authority that runs as a AWS Lambda function in an isolated AWS account with restricted IAM roles to sign short-lived ssh public keys. Written in Python.

  • https://www.npmjs.com/package/aws-lambda-toolkit

Social

#AWSLambda is the Twitter tag.

  • http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Resources:

  • https://blog.fugue.co/2016-05-05-architecting-a-serverless-web-application-in-aws.html

  • https://github.com/dwyl/learn-aws-lambda

  • https://www.jayway.com/2015/11/07/error-handling-in-api-gateway-and-aws-lambda/

  • http://codurance.com/2016/05/11/aws-lambdas/

  • https://stelligent.com/2016/03/23/serverless-delivery-bootstrapping-the-pipeline-part-2/

  • https://www.youtube.com/watch?v=fXZzVzptkeo

@johncmckim John McKim’s Express to AWS Lambda series on Medium:

Justin Yoo Readify ‏@readify

Trivia

  • The HTML 4 character entity references for the Greek capital and small letter lambda are “Λ” (&#923) and “λ” (&#955).

  • The Unicode numbers for lambda are U+039B and U+03BB.

Tutorial Rock Stars

Danilo Poccia (@danilop of Luxembourg, http://danilop.net/), Technical Evangelist at Amazon Web Services

Eric Hammond

Vyon Nagrani, AWS Lambda Sr. Product Manager,

}, “Region” : { “Value” : { “Ref” : “AWS:: Region” }, “Description” : “The region this template was launched in.” } } }

Typer Cross

  • wrote an ebook “AWS Lambda: The Ultimate Guide To Serverless Microservices - Learn Everything You Need To Know About Microservices Without Servers!” I do not recommend this at all. The title is a total lie as the author offers no comments to the code copied from elsewhere.

When code is copied from Kindle, line endings are removed and everything appears as a single line.

AWS People

  • Tim Wagner, @timallenwagner, General Manager of AWS Lambda at Amazon, Apr 10, 2015:

    Lambda was launched publicly November, 2015. Can run 100 Lambdas at the same time by default. Added Synchronous (Request/Response).

  • Ajay Nair @ajaynairthinks is Lead Product Manager for AWS Lambda

  • Michael LaFrenier @MLaFrecruiter Talent Sourcer and Executive Recruiter for Amazon Web Services

More on Serverless

This is one of a series on Serverless computing