DEV Community

Cover image for A crash course on Serverless with AWS - Image resize on-the-fly with Lambda and S3
Adnan Rahić
Adnan Rahić

Posted on • Updated on

A crash course on Serverless with AWS - Image resize on-the-fly with Lambda and S3

Handling large images has always been a pain in my side since I started writing code. Lately, it has started to have a huge impact on page speed and SEO ranking. If your website has poorly optimized images it won't score well on Google Lighthouse. If it doesn't score well, it won't be on the first page of Google. That sucks.

meme about page 2 of google

TL;DR

I've built and open-sourced a snippet of code that automates the process of creating and deploying an image resize function and an S3 bucket with one simple command. Check out the code here.

But if you want to follow along and learn how to do it yourself, keep reading.

Where do we start?

Luckily enough, there is a way to solve the issue of bad image optimization with little to no hassle. Today we'll build an AWS Lambda function to resize images on-the-fly.

The images will be stored in an S3 bucket and, once requested, will be served from it. If you need a resized version, you'll request the image and supply the height and width. This will trigger a function. It'll grab the existing image, resize it, return it back to the bucket and then serve it from the bucket.

This scenario will only resize an image for a given set of dimensions once. Every subsequent request to the image of that size will be served from the bucket. Pretty cool huh? Here's a diagram, because who doesn't love diagrams.

diagram

Because I already assume you know how to use the Serverless Framework and have already been introduced to the basics of serverless, Docker and AWS, I'll immediately jump into the gist of things. Here's an overview of what we'll be doing.

  • Create the project structure
  • Create a secrets file
  • Write the AWS Lambda function configuration
  • Write AWS Lambda function source code
  • Write S3 Bucket configuration
  • Deploy with Docker
  • Test with Dashbird

Note: Please install Docker and Docker Compose before continuing with this tutorial.

The funny thing here is that we need to use Docker to deploy this service because of Sharp. This image-resize module has binaries that need to be built on the same operating system as it will run on. Because AWS Lambda is running on Amazon Linux, we need to npm install the packages on an Amazon Linux instance before running sls deploy.

With that all out of the way let's build some stuff.

Create the project structure

The best way of explaining this complex structure will be with an image.

folder structure

Folders in green and files in blue. Check out the repo for further insight.

Create a secrets file

We'll create a secrets.env file in the secrets folder to hold our secrets and inject them into our Docker container once it spins up.

SLS_KEY=XXX # replace with your IAM User key
SLS_SECRET=YYY # replace with your IAM User secret
STAGE=dev
REGION=us-east-1
BUCKET=images.yourdomain.com
Enter fullscreen mode Exit fullscreen mode

Note: The deploy.sh script will create a secondary secrets.json file only to hold the domain name of our API Gateway endpoint.

Write the AWS Lambda function configuration

Open up the functions folder, and start by creating the package.json file. Paste this in.

{
  "name": "image-resize",
  "version": "1.0.0",
  "description": "Serverless image resize on-the-fly.",
  "main": "resize.js",
  "dependencies": {
    "serverless-plugin-tracing": "^2.0.0",
    "sharp": "^0.18.4"
  }
}
Enter fullscreen mode Exit fullscreen mode

No need to run any install, because we need to run the Docker container first. Note that we're adding a tracing plugin to enable X-Ray because it's awesome.

Moving on, let's create the serverless.yml file to configure our function.

service: image-resize-on-the-fly-functions

plugins:
  - serverless-plugin-tracing

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${env:STAGE, 'dev'}
  profile: serverless-admin # or change to 'default'
  tracing: true
  region: ${env:REGION, 'us-east-1'}
  environment:
    BUCKET: ${env:BUCKET}
    REGION: ${env:REGION}
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:ListBucket"
      Resource: "arn:aws:s3:::${env:BUCKET}"
    - Effect: "Allow"
      Action:
        - "s3:PutObject"
      Resource: "arn:aws:s3:::${env:BUCKET}"
    - Effect: "Allow"
      Action:
        - "xray:PutTraceSegments"
        - "xray:PutTelemetryRecords"
      Resource:
        - "*"

functions:
  resize:
    handler: resize.handler
    events:
      - http:
          path: resize
          method: get
Enter fullscreen mode Exit fullscreen mode

As you can see we're grabbing a bunch of values from the env and also allowing the function to access the BUCKET we specified and some X-Ray telemetry.

Write AWS Lambda function source code

Only the code left now. Here's what we'll do. Because images can get quite large and we don't want to risk loading several megabytes into the lambda function's memory, I'll show you how to use Node.js streams to read the image from an S3 stream, pipe it to Sharp, and then write it back to S3, once again, as a stream.

Ready? Let's do it in two steps. First, define the helper functions we need in order to create the streams, then create the handler function itself which our lambda will invoke.

// resize.js

// require modules
const stream = require('stream')
const AWS = require('aws-sdk')
const S3 = new AWS.S3({
  signatureVersion: 'v4'
})
const sharp = require('sharp')

// create constants
const BUCKET = process.env.BUCKET
const URL = `http://${process.env.BUCKET}.s3-website.${process.env.REGION}.amazonaws.com`

// create the read stream abstraction for downloading data from S3
const readStreamFromS3 = ({ Bucket, Key }) => {
  return S3.getObject({ Bucket, Key }).createReadStream()
}
// create the write stream abstraction for uploading data to S3
const writeStreamToS3 = ({ Bucket, Key }) => {
  const pass = new stream.PassThrough()
  return {
    writeStream: pass,
    uploadFinished: S3.upload({
      Body: pass,
      Bucket,
      ContentType: 'image/png',
      Key
    }).promise()
  }
}
// sharp resize stream
const streamToSharp = ({ width, height }) => {
  return sharp()
    .resize(width, height)
    .toFormat('png')
}
Enter fullscreen mode Exit fullscreen mode

Here we're defining constants and the constructor functions for our streams. What's important to note is that S3 doesn't have a default method for writing to storage with a stream. So you need to create one. It's doable with the stream.PassThrough() helper. The abstraction above will write the data with a stream and resolve a promise once it's done.

Moving on, let's check out the handler. Just underneath the code above, paste this in.

exports.handler = async (event) => {
  const key = event.queryStringParameters.key
  // match a string like: '/1280x720/image.jpg'
  const match = key.match(/(\d+)x(\d+)\/(.*)/)
  // get the dimensions of the new image
  const width = parseInt(match[1], 10)
  const height = parseInt(match[2], 10)
  const originalKey = match[3]
  // create the new name of the image, note this has a '/' - S3 will create a directory
  const newKey = '' + width + 'x' + height + '/' + originalKey
  const imageLocation = `${URL}/${newKey}`

  try {
    // create the read and write streams from and to S3 and the Sharp resize stream
    const readStream = readStreamFromS3({ Bucket: BUCKET, Key: originalKey })
    const resizeStream = streamToSharp({ width, height })
    const { 
      writeStream, 
      uploadFinished 
    } = writeStreamToS3({ Bucket: BUCKET, Key: newKey })

    // trigger the stream
    readStream
      .pipe(resizeStream)
      .pipe(writeStream)

    // wait for the stream to finish
    const uploadedData = await uploadFinished

    // log data to Dashbird
    console.log('Data: ', {
      ...uploadedData,
      BucketEndpoint: URL,
      ImageURL: imageLocation
    })

    // return a 301 redirect to the newly created resource in S3
    return {
      statusCode: '301',
      headers: { 'location': imageLocation },
      body: ''
    }
  } catch (err) {
    console.error(err)
    return {
      statusCode: '500',
      body: err.message
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The query string parameters will look like this /1280x720/image.jpg. Which means we grab the image dimensions from the parameters by using a regex match. The newKey value is set to the dimensions followed by a / and the original name of the image. This will create a new folder named 1280x720 with the image image.jpg in it. Pretty cool.

Once we trigger the streams and await the promise to resolve we can log out the data and return a 301 redirect to the location of the image in the bucket. Note that we will enable static website hosting on our bucket in the next paragraph.

Write S3 Bucket configuration

The bucket configuration will mostly consist of default CloudFormation templates, so if you're a bit rusty, I'd advise you brush up a bit. But, the gist of it is simple enough to understand. Open up the bucket folder and create a serverless.yml file.

service: image-resize-on-the-fly-bucket

custom:
  secrets: ${file(../secrets/secrets.json)} # will be created in our deploy.sh script

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${env:STAGE, 'dev'}
  profile: serverless-admin # or change to 'default'
  region: us-east-1
  environment:
    BUCKET: ${env:BUCKET}

resources:
  Resources:
    ImageResizeOnTheFly:
      Type: AWS::S3::Bucket # creating an S3 Bucket
      Properties:
        AccessControl: PublicReadWrite
        BucketName: ${env:BUCKET}
        WebsiteConfiguration: # enabling the static website option
          ErrorDocument: error.html
          IndexDocument: index.html
          RoutingRules: # telling it to redirect for every 404 error
            - 
              RedirectRule:
                HostName: ${self:custom.secrets.DOMAIN} # API Gateway domain
                HttpRedirectCode: "307" # temporary redirect HTTP code
                Protocol: "https"
                ReplaceKeyPrefixWith: "${self:provider.stage}/resize?key=" # route
              RoutingRuleCondition:
                HttpErrorCodeReturnedEquals: "404"
                KeyPrefixEquals: ""
    ImageResizeOnTheFlyPolicy:
      Type: AWS::S3::BucketPolicy # add policy for public read access
      Properties: 
        Bucket: 
          Ref: ImageResizeOnTheFly
        PolicyDocument: 
          Statement: 
            - 
              Action: 
                - "s3:*"
              Effect: "Allow"
              Resource:
                Fn::Join: 
                  - ""
                  - 
                    - "arn:aws:s3:::"
                    - 
                      Ref: ImageResizeOnTheFly
                    - "/*"
              Principal: "*"
Enter fullscreen mode Exit fullscreen mode

Once we enable the static website hosting option on our bucket, it will behave like any website. This lets us serve images from it and utilize the 404 error redirect rule.

If no image is found, the bucket will trigger the 404, which redirects to our lambda function. Then resize the original image to the requested dimensions, and put it back to the bucket at the exact route it was requested from.

Just like magic!

Deploy with Docker

Here comes the fun part! We'll create a Dockerfile and docker-compose.yml file to create our Amazon Linux container and load it with .env values. That's easy. The hard part will be writing the bash script to run all commands and deploy our function and bucket.

Starting with the Dockerfile, here's what you need to add.

FROM amazonlinux

# Create deploy directory
WORKDIR /deploy

# Install system dependencies
RUN yum -y install make gcc*
RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
RUN yum -y install nodejs

# Install serverless
RUN npm install -g serverless

# Copy source
COPY . .

# Install app dependencies
RUN cd /deploy/functions && npm i --production && cd /deploy

#  Run deploy script
CMD ./deploy.sh ; sleep 5m
Enter fullscreen mode Exit fullscreen mode

Because Amazon Linux is rather basic, we need to install both gcc and Node.js at the get-go. Then it's as simple as any Dockerfile you've seen. Install the Serverless Framework globally, copy over the source code, install the npm modules in the functions directory and run the deploy.sh script.

The docker-compose.yml file is literally only used to load the .env values.

version: "3"
services:
  image-resize-on-the-fly:
    build: .
    volumes:
      - ./secrets:/deploy/secrets
    env_file:
      - ./secrets/secrets.env
Enter fullscreen mode Exit fullscreen mode

That's it. The Docker part is done. Let's write some bash.

Starting out simple, we'll define the initial variables, configure our serverless installation and deploy our function.

# deploy.sh

# variables
stage=${STAGE}
region=${REGION}
bucket=${BUCKET}
secrets='/deploy/secrets/secrets.json'

# Configure your Serverless installation to talk to your AWS account
sls config credentials \
  --provider aws \
  --key ${SLS_KEY} \
  --secret ${SLS_SECRET} \
  --profile serverless-admin

# cd into functions dir
cd /deploy/functions

# Deploy function
echo "------------------"
echo 'Deploying function...'
echo "------------------"
sls deploy
Enter fullscreen mode Exit fullscreen mode

Once we have deployed it, we need to grab the domain name of our API Gateway endpoint and put it in a secrets.json file which will be loaded into our serverless.yml file from the bucket directory. To do this I just did some regex magic. Add this at the bottom of your deploy.sh.

# find and replace the service endpoint
if [ -z ${stage+dev} ]; then echo "Stage is unset."; else echo "Stage is set to '$stage'."; fi

sls info -v | grep ServiceEndpoint > domain.txt
sed -i 's@ServiceEndpoint:\ https:\/\/@@g' domain.txt
sed -i "s@/$stage@@g" domain.txt
domain=$(cat domain.txt)
sed "s@.execute-api.$region.amazonaws.com@@g" domain.txt > id.txt
id=$(cat id.txt)

echo "------------------"
echo "Domain:"
echo "  $domain"
echo "------------------"
echo "API ID:"
echo "  $id"

rm domain.txt
rm id.txt

echo "{\"DOMAIN\":\"$domain\"}" > $secrets
Enter fullscreen mode Exit fullscreen mode

Now you have a secrets.json file in the secrets directory. All that's left is to run the bucket deployment. Paste this final snippet at the bottom of the deploy.sh script.

cd /deploy/bucket

# Deploy bucket config
echo "------------------"
echo 'Deploying bucket...'
sls deploy

echo "------------------"
echo 'Bucket endpoint:'
echo "  http://$bucket.s3-website.$region.amazonaws.com/"

echo "------------------"
echo "Service deployed. Press CTRL+C to exit."
Enter fullscreen mode Exit fullscreen mode

There we have it! The coding part is done. Would you believe me we only need to run one command now? Open up a terminal window in the root of your project and run:

$ docker-compose up --build
Enter fullscreen mode Exit fullscreen mode

Let it do its magic, and you'll see everything get created automagically! Please note that the terminal will show you the bucket endpoint which you'll use to access the images.

Test with Dashbird

The final step is to check if it all works. Let's upload an image to the bucket, so we have something to resize. Go ahead and find an image you like and want to resize, or just take this one.

It's rather huge and around 6 MB in size. Here's the command you want to run to upload it.

$ aws s3 cp --acl public-read IMAGE s3://BUCKET
Enter fullscreen mode Exit fullscreen mode

So, if your bucket name is images and your image name is the-earth.jpg, then it should look like this if you run the command from the directory where the image is located.

$ aws s3 cp --acl public-read the-earth.jpg s3://images
Enter fullscreen mode Exit fullscreen mode

Keep in mind you need to have the AWS CLI installed on your machine, or upload the image through the AWS Console.

Now, try out to request the image through the bucket. Type into your browser the S3 bucket URL.

http://BUCKET.s3-website.REGION.amazonaws.com/the-earth.jpg
Enter fullscreen mode Exit fullscreen mode

You'll see the original image appear. But, now add dimensions to the URL.

http://BUCKET.s3-website.REGION.amazonaws.com/400x400/the-earth.jpg
Enter fullscreen mode Exit fullscreen mode

This will trigger the resize function and create a 400x400 pixel version of the image. It'll take a few hundred milliseconds to create and once it's done, the browser will redirect you to the newly created image. When you try to refresh the URL you'll see that it's now fetching the new image, without calling the resize function.

Let's check the logs in Dashbird and make sure it all works fine under the hood.

Dashbird logs

Looks fine, but I did make some beginner mistakes when I first tried setting up the function. One of them was forgetting to require a module before using it. Luckily I immediately got an alert that explained what was wrong. Slack alerts are life-savers.

slack alert

The way I fixed the issue was with the live-tailing feature. It let me check the invocation logs with a few seconds of latency so I could debug the issue. Pretty cool.

live tailing

Wrapping up

To conclude this little showdev session, I'd like to point out using serverless as a helper to support your existing infrastructure is incredible. It's language agnostic and easy to use.

We at Dashbird use clusters of containers for our core features that interact heavily with our databases while offloading all the rest to lambda functions, queues, streams and other serverless services on AWS.

And, of course, here's the repo once again, give it a star if you want more people to see it on GitHub. If you follow along with the instructions there, you'll be able to have this image resize on-the-fly microservice up and running in no time at all.

If you want to read some of my previous serverless musings head over to my profile or join my serverless newsletter!

I had an absolute blast writing this open-source code snippet. Writing the article wasn't that bad either! Hope you guys and girls enjoyed reading it as much as I enjoyed writing it. If you liked it, slap that tiny unicorn so more people here on dev.to will see this article. Until next time, be curious and have fun.


This months sponsor is Zeet.

Disclaimer: Zeet is sponsoring this blogpost for the next month. I tried it out the other day. It's like serverless but for running entire back ends. You can host and scale apps automagically. Pretty neat.


Top comments (14)

Collapse
 
realph profile image
Raphael Essoo-Snowdon • Edited

I'm having a problem triggering the resize function. The 400px x 400px version doesn't yet exist in S3, calling it for the first time should trigger the resize and put the resized image in S3, however the response fails in Chrome Dev Tools when I request it inside an image tag:

However, if I go to that URL (someUrl/400x400/default.jpg) directly in Chrome, the resize function is triggered and the image is added to S3.

Is there any reason why it's failing when used inside an img tag, and why it doesn't kick-off the resize function?

UPDATE: I'm getting an 'ERR_TOO_MANY_REDIRECTS' error. See the image below:

image

Collapse
 
mattpackwood profile image
Matthew Packwood

Hi,

I am getting an error: Initialization error: Node.js v8 is not supported by Serverless Framework v2. Please upgrade - Am I missing something, is this the version of Node.js in the docker container?

Matt

Collapse
 
ewoks profile image
Beeblebrox • Edited

One more great post!!! Thanks for it... Looking forward to next one ;)

Collapse
 
erebos-manannan profile image
Erebos Manannán

Personally I've found kraken.io/ to be a perfect service for this. Easily affordable, probably less work to set up than your own homebrew solution.

Collapse
 
adnanrahic profile image
Adnan Rahić

I quite literally built it to be deployed with one command. How's that less work than paying for a 3rd party service?

Collapse
 
erebos-manannan profile image
Erebos Manannán

Well, you built it, and you will have to maintain it.

Collapse
 
orliesaurus profile image
orliesaurus

Tell me about the cost of running this? Got any data?

Collapse
 
adnanrahic profile image
Adnan Rahić

I've been running it in production for almost a year. Have never paid a cent for it.

Why? Well, you only ever need a fixed set of dimension for your image. So it's a thumbnail, an HD, and a FullHD version. You'll rarely ever see some serious traffic even if you have a decent website.

Collapse
 
rnanwani profile image
Rahul Nanwani

As the website evolves, in terms of design and layout, improperly resized images are in fact one of the biggest factors contributing to page bloat. And at that time, one has to make do with the closest size available. So, you should account for requiring different sizes in the future to ensure that your resizing strategy is future-proof.

Collapse
 
jon_tarrant profile image
Jon Tarrant

Still works 05/19, thanks a lot really helped me quickly add a little extra layer of AWS awesomeness into a showcase i've been working on! :)

Collapse
 
adnanrahic profile image
Adnan Rahić

Glad to hear! This article is due for an update. It can be done with Layers now, or even with a nice Makefile setup. Hmmm, so many ideas. 😄
Anyhow, glad my article helped!

Collapse
 
robincingh profile image
Robin

Would be interesting to see the result between the above and set up something like Thumbor and compare the overall optimisation and Google Results for that as thumbor also serves webp file output.

Collapse
 
adnanrahic profile image
Adnan Rahić

That would be cool indeed. Does Thumbor work with serverless?

Collapse
 
robincingh profile image
Robin

I don't see it not working with serverless as it is similar to the approach you followed.