From Background Job to a Serverless Application

Photo by engin akyurt on Unsplash

What is Serverless

Imagine a world where you don’t have to worry about servers, scaling, or maintenance. That’s the serverless wonderland for you! AWS Lambda is one of the magical tools that make this possible. In this land, you write code, deploy it, and Lambda does the rest. No more midnight server crashes or grumpy sysadmins (unless, of course, you miss them).

In a serverless setup, you don’t provision or manage servers. AWS Lambda automatically runs your code in response to events like HTTP requests, database updates, or even a tweet from your favorite celebrity! You pay only for the compute time you consume, and there’s no charge when your code isn’t running. It’s like having a personal butler who only gets paid when you ask for a cup of tea.

Other service providers like Azure Functions and Google Cloud Functions also offer serverless capabilities, each with their own set of bells and whistles. But today, let’s focus on AWS Lambda.

Perks of the Serverless Applications

Why should you care about serverless? Here’s a quick rundown of the benefits:

• Cost Efficiency: Pay-as-you-go pricing means you only pay for what you use. It’s like having a buffet where you only pay for the calories you actually consume.
• Scalability: Automatically scales your application in response to incoming traffic. No more frantic calls to IT asking, “Can we handle this Black Friday sale?”
• Maintenance-Free: No need to patch servers or worry about uptime. AWS takes care of the heavy lifting while you sip your coffee.
• Speed to Market: Deploy applications faster without the need to provision infrastructure. Think of it as instant noodles for your app — just add code and it’s ready to serve!

Quest of Moving a Rails Background Job to AWS Lambda

I had a background job in my app that was like a relentless hamster on a wheel — running constantly throughout the day, fetching and updating database records. It was consuming resources like a teenager raiding the fridge, and I needed to lighten the load on my application server. My mission was to move this background job to AWS Lambda.

There are shiny, ready-made solutions like Lamby and Jets for deploying entire Rails applications into Lambda. These tools can package your app and send it flying into the serverless cloud. But I decided go with a custom solution. Why? I only needed ActiveRecord, and sometimes it’s fun to take the road less traveled (and then write a post about it).

Docker Dance: Creating a Deployable Image

I decided to create a Docker image for my Lambda function. For those who haven’t worked with Docker, it’s like packing everything your app needs into a container — think of it as a lunchbox with all your favorite snacks.

Here’s the Dockerfile I whipped up:

# Pull official ruby 3.2 image
FROM public.ecr.aws/lambda/ruby:3.2

# Copy Gemfile and Gemfile.lock
COPY Gemfile Gemfile.lock ${LAMBDA_TASK_ROOT}/

# Install dependencies required for the lambda
RUN yum update -y && \
    yum install -y gcc make amazon-linux-extras \
    openssl openssl-devel postgresql13 && \
    amazon-linux-extras enable postgresql13 && \
    yum install -y libffi libffi-devel 
    postgresql postgresql-server postgresql-devel \
    mysql-devel


# Install Bundler and the specified gems
RUN gem install bundler:2.5.7 && \
    bundle config set --local path 'vendor/bundle' && \
    bundle install

# Copy function code
COPY . ${LAMBDA_TASK_ROOT}/

# Create lib directory for native gem dependencies
RUN mkdir ${LAMBDA_TASK_ROOT}/lib/
RUN mkdir /usr/mysql/
RUN cp -r /usr/lib64/mysql/* /usr/mysql/
RUN cp -r /usr/lib64/mysql/* $LAMBDA_TASK_ROOT/lib/
RUN bundle config --local build.mysql --with-mysql-config=/usr/local/mysql/bin/mysql_config
RUN bundle config --local set deployment true
RUN bundle install

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "scan.LambdaFunction::Handler.process" ]

This Dockerfile sets up a Ruby environment with all the necessary dependencies. It’s like packing a Swiss Army knife, a flashlight, and a bag of trail mix for a weekend camping trip — prepared for anything!

Here’s what the directory structure looked like:

lambda_function
├── app
│   ├── models
│       ├── application_record.rb
│       └── user.rb
│
│
│
├── lambda_function.rb
├── Gemfile
├── Gemfile.lock
└── README.md

Minimal, tidy, and ready for action.

Setting Up the Database

To access the models from my Rails application in my lambda_function.rb file, I added the database configuration:

require 'active_record'
require 'pg'
require 'mysql2'

# Load model files from the app/models directory
Dir["app/models/*.rb"].each { |file| require_relative file }

# Build database connection to your Rails database
ActiveRecord::Base.establish_connection(
  adapter: ENV['database_adapter'],
  encoding: ENV['database_encoding'],
  pool: "5", #ENV['database_pool']
  host: ENV['database_host'],
  port: ENV['database_port'],
  username: ENV['database_user'],
  password: ENV['database_pass'],
  database: ENV['database_name'],
)

module LambdaFunction
  module Handler

    def self.process(event:, context:)
      if event['source'] == 'aws.apigateway'
        invoke_lambda_method # triggered by api call
      else
        return {
          statusCode: 400,
          body: {
            error: "Unknown event source"
          }.to_json
        }
      end
    end
  end
end

In this code, I established a connection to my database and set up the handler to process events. It’s like setting up a telephone line and waiting for calls (hopefully not from telemarketers).

Models from the Rails app

I created an application_record.rb file to serve as a base class for my models:

require 'active_record'

class ApplicationRecord < ActiveRecord::Base
  primary_abstract_class
end

And here I am providing a sample model that should contain only the bare minimum required for the successful run of your lambda function:

require_relative 'application_record'

class User < ApplicationRecord

  # define minimum required relations
  # include only relevant validations, attributes, methods, etc.
end

Prerequisites and Local Testing

You can deploy your Lambda as a Docker container on your personal machine, you’ll need Docker Desktop, Ruby, and AWS CLI version 2. The image is built locally with the following commands:

# First install necessary gems
bundle install

# Then build the Docker image locally
docker build --platform linux/amd64 -t lambda-function:latest .

To test it, you can connect to a remote database server (like Amazon RDS) by setting necessary environment variables for the ActiveRecord::Base.establish_connection() and run your container locally with:

docker run --platform linux/amd64 -p 9000:8080 lambda-function:latest

While the container is running on your machine, you can trigger the Lambda function with a curl command:

curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{ "source": "aws.apigateway" }'

Based on the response from your curl command, you can debug or fine-tune your Lambda function. Keep in mind that Lambda is billed per usage, so it’s better to minimize database calls and optimize any slow code. Think of it as making every second count — like a speed dating session with your database!

Wrapping Up

Migrating my background job to AWS Lambda was like upgrading from a horse-drawn carriage to a rocket ship. It took some elbow grease and a bit of learning, but the rewards were worth it. Now, the job runs smoothly, and the application server isn’t panting like it just ran a marathon.

If you’re considering a move to serverless, give it a shot. It might just be the adventure you need, and who knows, you might end up writing a blog post about it too!