Skip to main content

AWS RDS Extensions in the Atlas Dev Database

Your RDS schema declares an AWS extension and builds on it: CREATE EXTENSION aws_lambda, then a trigger calls aws_lambda.invoke or aws_s3 for a bulk import. It runs on RDS, but Atlas commands that use a dev database (migrate diff, schema diff, schema plan) can't compile it because a stock postgres image has no aws_lambda to create.

This is the question RDS users ask us most:

How do you handle the AWS extensions (aws_commons, aws_lambda, aws_s3) when they aren't in the dev Docker image?

The answer is simple: mock them. Build a dev image that carries its own copy of each extension, with real function signatures but empty bodies, so CREATE EXTENSION and everything downstream compiles while the real implementations keep running on RDS. The AWS extensions are the example here; the pattern fits any cloud-only extension with no public binary.

Prerequisites​

  1. Docker, to build and run the dev-database container.
  2. Atlas installed:

To download and install the latest release of the Atlas CLI, simply run the following in your terminal:

curl -sSf https://atlasgo.sh | sh
Atlas Pro Feature

The docker and devblocks used in this guide are available to Atlas Pro users. To use them, run:

atlas login

Background​

The dev database is a throwaway PostgreSQL database that Atlas uses to compile and validate your schema before planning a change. Since aws_commons, aws_lambda, and aws_s3 have no public binary, loading a schema that opens with CREATE EXTENSION aws_commons onto a stock image fails on the first line:

Error: schema.sql:2: pq: extension "aws_commons" is not available (0A000)

You could use a second RDS instance that has the real extensions as a dev database since a dev block takes any PostgreSQL URL, but that's slow and costs money to keep running. Mocking them on a local image is cheaper and does the job. At plan time Atlas never calls aws_lambda.invoke, it only needs it to exist.

Install Extensions on the Dev Image​

The dev database has to run your CREATE EXTENSION lines, so the image needs those extensions installed as real, installable packages whose scripts define stubs.

Set up your project. The content for each file will be explained below.

.
├── atlas.hcl
├── Dockerfile             # builds the dev image (official postgres + the mocks)
├── schema.sql             # your desired schema
├── ext/                   # the mock extension package
│   ├── aws_commons.control
│   ├── aws_commons--1.2.sql
│   ├── aws_lambda.control
│   ├── aws_lambda--1.0.sql
│   ├── aws_s3.control
│   └── aws_s3--1.2.sql
└── migrations/

Define a schema that enables all three extenstions, invokes a Lambda on insert, and bulk-loads from S3:

schema.sql
CREATE EXTENSION aws_commons;
CREATE EXTENSION aws_lambda;
CREATE EXTENSION aws_s3;

CREATE TABLE "public"."events" (
    "id"         bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    "payload"    jsonb NOT NULL,
    "created_at" timestamptz NOT NULL DEFAULT now()
);

-- On insert, invoke a Lambda function via the aws_lambda extension.
CREATE FUNCTION "public"."on_event_insert"() RETURNS trigger
    LANGUAGE plpgsql AS $$
BEGIN
    PERFORM aws_lambda.invoke(
        aws_commons.create_lambda_function_arn('process-event', 'us-east-1'),
        to_jsonb(NEW)
    );
    RETURN NEW;
END;
$$;

CREATE TRIGGER "trg_on_event_insert"
    AFTER INSERT ON "public"."events"
    FOR EACH ROW EXECUTE FUNCTION "public"."on_event_insert"();

-- Bulk-load rows from S3 via the aws_s3 extension.
CREATE FUNCTION "public"."import_events"("bucket" text, "path" text) RETURNS text
    LANGUAGE sql AS $$
    SELECT aws_s3.table_import_from_s3(
        'public.events', '', '(format csv)',
        aws_commons.create_s3_uri("bucket", "path", 'us-east-1')
    );
$$;

The extension package​

PostgreSQL installs an extension from a control file "named the same as the extension with a suffix of .control" in SHAREDIR/extension, plus a version script. So each extension is two files:

  • aws_commons.control: the metadata PostgreSQL reads before installing.
  • aws_commons--1.2.sql: the script it runs inside CREATE EXTENSION.

The control file uses postgresql.conf syntax: parameter = value, one per line, with # for comments (not SQL's --):

ext/aws_commons.control
# Mock of the RDS aws_commons extension for the Atlas dev database.
default_version = '1.2'
comment = 'AWS commons mock'

The field to get right is default_version. Atlas installs the mock at that version, then records it in the migration as CREATE EXTENSION ... VERSION "1.2" (you'll see it below). RDS must offer that same version when the migration runs there, or apply fails. So set it to what RDS reports:

SELECT name, default_version FROM pg_available_extensions
  WHERE name IN ('aws_commons', 'aws_lambda', 'aws_s3');
--     name     | default_version
-- -------------+-----------------
--  aws_commons | 1.2
--  aws_lambda  | 1.0
--  aws_s3      | 1.2

The script creates the extension's schema and the stub objects. Everything it creates during CREATE EXTENSION becomes an extension member, so Atlas treats the schema and functions as extension-owned and never manages them itself:

ext/aws_commons--1.2.sql
CREATE SCHEMA aws_commons;

-- Composite types (array types are auto-created).
CREATE TYPE aws_commons._s3_uri_1              AS (bucket text, file_path text, region text);
CREATE TYPE aws_commons._lambda_function_arn_1 AS (function_name text, region text);
CREATE TYPE aws_commons._aws_credentials_1     AS (access_key text, secret_key text, session_token text);

CREATE FUNCTION aws_commons.create_s3_uri(bucket text, file_path text, region text DEFAULT NULL)
  RETURNS aws_commons._s3_uri_1 LANGUAGE sql AS $$ SELECT NULL::aws_commons._s3_uri_1 $$;
CREATE FUNCTION aws_commons.create_lambda_function_arn(function_name text, region text DEFAULT NULL)
  RETURNS aws_commons._lambda_function_arn_1 LANGUAGE sql AS $$ SELECT NULL::aws_commons._lambda_function_arn_1 $$;
CREATE FUNCTION aws_commons.create_aws_credentials(access_key text, secret_key text, session_token text)
  RETURNS aws_commons._aws_credentials_1 LANGUAGE sql AS $$ SELECT NULL::aws_commons._aws_credentials_1 $$;

aws_lambda and aws_s3 have the same shape. Their control files add requires = 'aws_commons', so PostgreSQL knows to install aws_commons first:

ext/aws_lambda.control
# Mock of the RDS aws_lambda extension for the Atlas dev database.
default_version = '1.0'
comment = 'AWS Lambda mock'
requires = 'aws_commons'
relocatable = true
ext/aws_s3.control
# Mock of the RDS aws_s3 extension for the Atlas dev database.
default_version = '1.2'
comment = 'AWS S3 mock'
requires = 'aws_commons'
relocatable = true
ext/aws_lambda--1.0.sql
CREATE SCHEMA aws_lambda;

CREATE FUNCTION aws_lambda.invoke(function_name aws_commons._lambda_function_arn_1, payload jsonb,
  invocation_type text DEFAULT 'RequestResponse', log_type text DEFAULT 'None', context jsonb DEFAULT NULL,
  qualifier varchar DEFAULT NULL, OUT status_code integer, OUT payload jsonb, OUT executed_version text, OUT log_result text)
  LANGUAGE sql AS $$ SELECT NULL::integer, NULL::jsonb, NULL::text, NULL::text $$;
CREATE FUNCTION aws_lambda.invoke(function_name aws_commons._lambda_function_arn_1, payload json,
  invocation_type text DEFAULT 'RequestResponse', log_type text DEFAULT 'None', context json DEFAULT NULL,
  qualifier varchar DEFAULT NULL, OUT status_code integer, OUT payload json, OUT executed_version text, OUT log_result text)
  LANGUAGE sql AS $$ SELECT NULL::integer, NULL::json, NULL::text, NULL::text $$;
CREATE FUNCTION aws_lambda.invoke(function_name text, payload jsonb, region text DEFAULT NULL,
  invocation_type text DEFAULT 'RequestResponse', log_type text DEFAULT 'None', context jsonb DEFAULT NULL,
  qualifier varchar DEFAULT NULL, OUT status_code integer, OUT payload jsonb, OUT executed_version text, OUT log_result text)
  LANGUAGE sql AS $$ SELECT NULL::integer, NULL::jsonb, NULL::text, NULL::text $$;
CREATE FUNCTION aws_lambda.invoke(function_name text, payload json, region text DEFAULT NULL,
  invocation_type text DEFAULT 'RequestResponse', log_type text DEFAULT 'None', context json DEFAULT NULL,
  qualifier varchar DEFAULT NULL, OUT status_code integer, OUT payload json, OUT executed_version text, OUT log_result text)
  LANGUAGE sql AS $$ SELECT NULL::integer, NULL::json, NULL::text, NULL::text $$;
ext/aws_s3--1.2.sql
CREATE SCHEMA aws_s3;

CREATE FUNCTION aws_s3.table_import_from_s3(table_name text, column_list text, options text,
  bucket text, file_path text, region text, access_key text DEFAULT NULL, secret_key text DEFAULT NULL, session_token text DEFAULT NULL)
  RETURNS text LANGUAGE sql AS $$ SELECT NULL::text $$;
CREATE FUNCTION aws_s3.table_import_from_s3(table_name text, column_list text, options text,
  s3_info aws_commons._s3_uri_1, credentials aws_commons._aws_credentials_1 DEFAULT NULL)
  RETURNS text LANGUAGE sql AS $$ SELECT NULL::text $$;
CREATE FUNCTION aws_s3.query_export_to_s3(query text, bucket text, file_path text, region text DEFAULT NULL,
  options text DEFAULT NULL, kms_key text DEFAULT NULL, OUT rows_uploaded bigint, OUT files_uploaded bigint, OUT bytes_uploaded bigint)
  LANGUAGE sql AS $$ SELECT NULL::bigint, NULL::bigint, NULL::bigint $$;
CREATE FUNCTION aws_s3.query_export_to_s3(query text, s3_info aws_commons._s3_uri_1 DEFAULT NULL,
  options text DEFAULT NULL, kms_key text DEFAULT NULL, OUT rows_uploaded bigint, OUT files_uploaded bigint, OUT bytes_uploaded bigint)
  LANGUAGE sql AS $$ SELECT NULL::bigint, NULL::bigint, NULL::bigint $$;

Bake the package into the dev image​

The image is the official postgres plus the ext/ files, copied into PostgreSQL's extension directory:

Dockerfile
# Match your RDS major version
FROM postgres:18

COPY ext/ /tmp/mock-ext/
RUN set -eux; \
    install -d "$(pg_config --sharedir)/extension"; \
    install -m 0644 -t "$(pg_config --sharedir)/extension" \
        /tmp/mock-ext/*.control /tmp/mock-ext/*.sql; \
    rm -rf /tmp/mock-ext

Point the docker block's build config at it. Atlas builds the image once and tags it with image:

atlas.hcl
docker "postgres" "dev" {
  // Local image name Atlas builds and tags
  image = "postgres:18-mock-aws"
  build {
    context    = "."
    dockerfile = "Dockerfile"
  }
}

env "local" {
  url = "your database URL here"
  src = "file://schema.sql"
  dev = docker.postgres.dev.url
}

Now the schema compiles, and migrate diff writes the migration:

atlas migrate diff initial --env local
migrations/20260710181259_initial.sql
-- Create extension "aws_commons"
CREATE EXTENSION "aws_commons" WITH SCHEMA "public" VERSION "1.2";
-- Create extension "aws_s3"
CREATE EXTENSION "aws_s3" WITH SCHEMA "public" VERSION "1.2";
-- Create extension "aws_lambda"
CREATE EXTENSION "aws_lambda" WITH SCHEMA "public" VERSION "1.0";
-- Create "events" table
CREATE TABLE "public"."events" (
  "id" bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
  "payload" jsonb NOT NULL,
  "created_at" timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY ("id")
);
-- Create "on_event_insert" function
CREATE FUNCTION "public"."on_event_insert" () RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
    PERFORM aws_lambda.invoke(
        aws_commons.create_lambda_function_arn('process-event', 'us-east-1'),
        to_jsonb(NEW)
    );
    RETURN NEW;
END;
$$;
-- Create trigger "trg_on_event_insert"
CREATE TRIGGER "trg_on_event_insert" AFTER INSERT ON "public"."events" FOR EACH ROW EXECUTE FUNCTION "public"."on_event_insert"();
-- Create "import_events" function
CREATE FUNCTION "public"."import_events" ("bucket" text, "path" text) RETURNS text LANGUAGE sql AS $$
SELECT aws_s3.table_import_from_s3(
        'public.events', '', '(format csv)',
        aws_commons.create_s3_uri("bucket", "path", 'us-east-1')
    );
$$;

That's the full migration. A follow-up diff is a no-op:

atlas migrate diff followup --env local
The migration directory is synced with the desired state, no changes to be made

Deploy to RDS​

Your RDS environment is likely already configured. This example authenticates with a short-lived IAM token via data "aws_rds_token":

atlas.hcl
variable "endpoint" {
  // host:port, e.g. mydb.abc123.us-east-1.rds.amazonaws.com:5432
  type = string
}

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "database" {
  type = string
}

variable "username" {
  type = string
}

docker "postgres" "dev" {
  image = "postgres:18-mock-aws"
  build {
    context    = "."
    dockerfile = "Dockerfile"
  }
}

data "aws_rds_token" "migrator" {
  region   = var.region
  endpoint = var.endpoint
  username = var.username
}

env "rds" {
  src = "file://schema.sql"
  url = "postgres://${var.username}:${urlescape(data.aws_rds_token.migrator)}@${var.endpoint}/${var.database}?sslmode=require"
  dev = docker.postgres.dev.url
}

On RDS, the same CREATE EXTENSION aws_lambda installs the real extension, and aws_lambda.invoke calls the real AWS integration at runtime:

atlas migrate apply --env rds \
  --var endpoint=mydb.abc123.us-east-1.rds.amazonaws.com:5432 \
  --var database=appdb \
  --var username=<privileged-role>
Migrating to version 20260710181259 (1 migrations in total):
  -- migrating version 20260710181259
    -> CREATE EXTENSION "aws_commons" WITH SCHEMA "public" VERSION "1.2";
    -> CREATE EXTENSION "aws_s3" WITH SCHEMA "public" VERSION "1.2";
    -> CREATE EXTENSION "aws_lambda" WITH SCHEMA "public" VERSION "1.0";
    -> ... (events table, functions, trigger)
  -- ok (6.216017584s)
  -- 1 migration, 7 sql statements
The applying role needs rds_superuser

CREATE EXTENSION for the AWS extensions requires membership in rds_superuser, so a least-privilege deploy role can't create them. Apply the extension-creating migrations (in practice, just the first) with a privileged role; later ones need no special rights.

aws_rds_token signs the token from your ambient AWS credentials, so IAM auth must be configured for the database user. See the RDS IAM migration role guide.

Adding PostGIS or pgvector Alongside the Mocks​

Both PostGIS and pgvector ship a real binary, so you install the real extension on the dev image, right next to the AWS mocks.

One Dockerfile does both: install PostGIS (or start from an image that ships it), then copy the ext/ mocks as before:

Dockerfile
FROM postgres:18

# Real PostGIS: it has a public binary, so install the package.
RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends postgresql-18-postgis-3; \
    rm -rf /var/lib/apt/lists/*

# Mock AWS extensions: no binary, so copy the package in (as above).
COPY ext/ /tmp/mock-ext/
RUN set -eux; \
    install -d "$(pg_config --sharedir)/extension"; \
    install -m 0644 -t "$(pg_config --sharedir)/extension" \
        /tmp/mock-ext/*.control /tmp/mock-ext/*.sql; \
    rm -rf /tmp/mock-ext

A schema that declares CREATE EXTENSION postgis alongside the AWS extensions now compiles on the dev database: PostGIS is real, the AWS functions are stubs.

Since PostGIS is a real extension, Atlas manages it like the AWS ones, so it lands in the migration pinned to the dev image's version. That means migrate apply needs rds_superuser and a matching PostGIS version on RDS (here the package installs 3.6.4; RDS may ship 3.6.1). Pin the package version, or start from a postgis/postgis tag that matches RDS. See Extension version mismatch.

Apply This to Other Extensions​

The technique is general. Grep for the extension's qualified calls (aws_lambda., aws_s3.) and mock only those. Read the exact signatures from a database that has the extension installed; the catalog prints them ready for CREATE FUNCTION, with OUT parameters and defaults intact:

SELECT format('%I.%I', n.nspname, p.proname) AS func,
       pg_get_function_arguments(p.oid)      AS args,
       pg_get_function_result(p.oid)         AS returns
FROM pg_proc p
JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('aws_commons', 'aws_lambda', 'aws_s3')
ORDER BY 1, 2;

Composite types the same way:

SELECT format('%I.%I', n.nspname, t.typname) AS type,
       string_agg(a.attname || ' ' || format_type(a.atttypid, a.atttypmod), ', '
                  ORDER BY a.attnum)         AS fields
FROM pg_type t
JOIN pg_namespace n ON n.oid = t.typnamespace
JOIN pg_class c     ON c.oid = t.typrelid
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum > 0 AND NOT a.attisdropped
WHERE n.nspname = 'aws_commons' AND t.typtype = 'c'
GROUP BY 1
ORDER BY 1;

And the version and dependencies from the control metadata, without installing anything:

SELECT name, version, requires FROM pg_available_extension_versions
  WHERE name IN ('aws_commons', 'aws_lambda', 'aws_s3');
--     name     | version |   requires
-- -------------+---------+---------------
--  aws_commons | 1.2     |
--  aws_lambda  | 1.0     | {aws_commons}
--  aws_s3      | 1.2     | {aws_commons}

From there it's mechanical: for each extension, write a control file (default_version, requires) and a script that stubs the objects your schema uses, then copy both into the image. Most schemas only touch functions and composite types, but operators, casts, and the rest stub the same way.

References​