Unable to use custom layer in AWS lambda (Ruby 3.2.0)

Question

I have a problem running a lambda (Runtime: Ruby 3.2) using a custom layer (pg gem).

I have followed the Ruby file structure as per this document to create the custom layer (.zip):

pg.zip
└ ruby/gems/3.2.0/
               | build_info
               | cache
               | doc
               | extensions
               | gems
               | └ pg-1.5.3
               └ specifications
                 └ pg-1.5.3.gemspec

This custom layer is already added in the lambda.

I tested and I got an error:

{
  "errorMessage": "cannot load such file -- pg",
  "errorType": "Init<LoadError>",
  "stackTrace": [
    "/var/lang/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/lang/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/task/lambda_function.rb:2:in `<top (required)>'",
    "/var/lang/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
    "/var/lang/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'"
  ]
}

I built the package in Mac (x86_64) with ruby@3.2 installed.

The ruby script works well when I ran it locally.

Questions:

1. Is my file structure for the pg.zip (custom layer) correct?
2. Or could this be a mismatch when bundling it in my current environment?

Answer 1

I faced a similar error too using lambda layers w ruby 3.2 runtimes!

To your qns:

1. Yes its correct usually! In our case, we will have an additional lib folder to store the native extensions.
2. Yes since pg depends on native extensions, you have to build the gem using in the same environment as the runtime environment.

These were the steps I took to solve it:

1. Build the gem pg using the lambda ruby 3.2 native extensions instead.

Dockerfile

FROM public.ecr.aws/lambda/ruby:3.2

RUN yum install -y amazon-linux-extras \
    && amazon-linux-extras enable postgresql14 \
    && yum group install "Development Tools" -y

RUN yum install -y postgresql postgresql-devel

ADD Gemfile Gemfile.lock ${LAMBDA_TASK_ROOT}

ENV GEM_HOME=${LAMBDA_TASK_ROOT}

RUN bundle config set without 'development test'
RUN bundle config set path 'vendor/bundle'

RUN bundle install

Gemfile

# frozen_string_literal: true

source 'https://rubygems.org'

gem 'pg'

2. and then build the Dockerfile and copy the necessary extensions into the layer folder (i.e $DIST_DIR folder). Example script:

#!/bin/sh -e

rm -rf $DIST_DIR
mkdir -p "$DIST_DIR/lib"
mkdir -p "$DIST_DIR/ruby/gems"

cd "$SOURCE_DIR" >/dev/null || exit

docker build -t ruby-builder -f Dockerfile .
CONTAINER=$(docker run -d ruby-builder)

# See https://northsail.io/articles/aws-lambda-ruby-2-7-pg-gem-libldap-error
# for what to libpq binaries to copy for Amazon Linux 2
docker cp \
  $CONTAINER:/usr/lib64/libpq.so.5.14 \
  $DIST_DIR/lib/libpq.so.5

docker cp \
  $CONTAINER:/usr/lib64/libldap_r-2.4.so.2.10.7 \
  $DIST_DIR/lib/libldap_r-2.4.so.2

docker cp \
  $CONTAINER:/usr/lib64/liblber-2.4.so.2.10.7 \
  $DIST_DIR/lib/liblber-2.4.so.2

docker cp \
  $CONTAINER:/usr/lib64/libsasl2.so.3.0.0 \
  $DIST_DIR/lib/libsasl2.so.3

docker cp \
  $CONTAINER:/usr/lib64/libssl3.so \
  $DIST_DIR/lib/

docker cp \
  $CONTAINER:/usr/lib64/libsmime3.so \
  $DIST_DIR/lib/

docker cp \
  $CONTAINER:/usr/lib64/libnss3.so \
  $DIST_DIR/lib/

docker cp \
  $CONTAINER:/usr/lib64/libnssutil3.so \
  $DIST_DIR/lib/

docker cp \
  $CONTAINER:/var/task/vendor/bundle/ruby/3.2.0 \
  $DIST_DIR/ruby/gems/3.2.0

docker stop $CONTAINER
docker rm $CONTAINER

The docker cp commands is used to copy over the missing native extension files needed by pg gem when running in the lambda environment which is running on Amazon Linux 2.

Your resulting layer contents should look like this:

├── lib
│   ├── liblber-2.4.so.2
│   ├── libldap_r-2.4.so.2
│   ├── libnss3.so
│   ├── libnssutil3.so
│   ├── libpq.so.5
│   ├── libsasl2.so.3
│   ├── libsmime3.so
│   └── libssl3.so
└── ruby
    └── gems
        └── 3.2.0
            ├── bin
            │   ├── bundle
            │   └── bundler
            ├── build_info
            ├── cache
            │   ├── bundler-2.4.6.gem
            │   └── pg-1.5.3.gem
            ├── doc
            ├── extensions
            │   └── aarch64-linux
            ├── gems
            │   ├── bundler-2.4.6
            │   └── pg-1.5.3
            ├── plugins
            └── specifications
                ├── bundler-2.4.6.gemspec
                └── pg-1.5.3.gemspec

3. .zip $DIST_DIR create aruby3.2 lambda layer using that same document you have: https://docs.aws.amazon.com/lambda/latest/dg/creating-deleting-layers.html