DynamoDB

Connect & Ingest data from / to AWS DynamoDB

DynamoDB is AWS's managed key-value and document database: a table is a set of items, each addressed by a primary key (a partition key, optionally plus a sort key). See https://aws.amazon.com/dynamodb/ for more details.

Sling connects through the AWS SDK for Go v2 and talks to the DynamoDB API directly. There is no SQL engine behind the connection: Sling renders each read into a scan descriptor (table, filter expression, projected fields and limit) and applies writes item by item, where a PutItem batch upserts by primary key.

Setup

The following credentials keys are accepted:

• region (required) -> AWS region of the tables, e.g. us-east-1.

• access_key_id (optional) -> AWS access key ID. When omitted, the AWS credential chain is used (environment variables, shared config/credentials, instance metadata).

• secret_access_key (optional) -> AWS secret access key.

• session_token (optional) -> AWS session token, for temporary credentials.

• profile (optional) -> AWS profile name from ~/.aws/credentials.

• endpoint (optional) -> Custom endpoint, for example DynamoDB Local (http://localhost:8000) or another compatible service.

For DynamoDB Local, set endpoint and region only. It accepts any credentials, so Sling signs with placeholder keys and no AWS account is needed:

$ docker run -p 8000:8000 amazon/dynamodb-local
$ sling conns set DYNAMODB type=dynamodb endpoint=http://localhost:8000 region=us-east-1

Using sling conns

Here are examples of setting a connection named DYNAMODB. We must provide the type=dynamodb property:

# using the AWS credential chain (env vars, ~/.aws/credentials, instance role)
$ sling conns set DYNAMODB type=dynamodb region=us-east-1

# explicit keys
$ sling conns set DYNAMODB type=dynamodb region=us-east-1 \
    access_key_id=AKIAIOSFODNN7EXAMPLE \
    secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# a named profile, or temporary credentials with a session token
$ sling conns set DYNAMODB type=dynamodb region=us-east-1 profile=prod
$ sling conns set DYNAMODB type=dynamodb region=us-east-1 \
    access_key_id=<access_key_id> \
    secret_access_key=<secret_access_key> \
    session_token=<session_token>

# DynamoDB Local
$ sling conns set DYNAMODB type=dynamodb endpoint=http://localhost:8000 region=us-east-1

$ sling conns test DYNAMODB

Environment Variable

See here to learn more about the .env.sling file.

export DYNAMODB='{ type: dynamodb, region: "us-east-1" }'
export DYNAMODB='{
  type: dynamodb,
  region: "us-east-1",
  access_key_id: "AKIAIOSFODNN7EXAMPLE",
  secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}'

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  DYNAMODB:
    type: dynamodb
    region: us-east-1

    # use explicit keys, or omit them to use the AWS credential chain
    access_key_id: <access_key_id>
    secret_access_key: <secret_access_key>
    session_token: <session_token>   # for temporary credentials
    profile: default                 # AWS profile from ~/.aws/credentials

  # DynamoDB Local
  DYNAMODB_LOCAL:
    type: dynamodb
    endpoint: http://localhost:8000
    region: us-east-1

Primary Keys

A DynamoDB table cannot exist without a primary key, and Sling upserts by that key. Sling takes it from the stream's primary_key (or unique key), then from the source's primary key, and when the stream declares none it adds a _sling_id column and keys the table on it — keying on a data column instead would collapse rows whose values repeat. Set the key explicitly whenever the data has a natural one:

source: POSTGRES
target: DYNAMODB

defaults:
  mode: incremental
  primary_key: [id]
  update_key: updated_at

streams:
  public.orders:
    object: default.orders

DynamoDB has no schemas, so Sling always uses the default schema in object names. A composite key is declared as multiple columns (primary_key: [id, code]), which maps to a partition key plus a sort key.

Reading Data

• A stream is read with Scan, so a bare table name returns every item.

• select with explicit fields and a limit are pushed down to the scan (select id, code from orders limit 100).

• Filtered reads use the where stream option with a JSON filter expression:

streams:
  default.orders:
    object: public.orders_pg
    where: '{"code": {"$gt": 15}}'

Supported filter operators are $eq, $ne, $lt, $lte, $gt, $gte, $in, $between, $begins_with and $exists. Values are stored with the type of their column (numbers as N, booleans as BOOL, timestamps as ISO strings).

• incremental and backfill translate update_key (and range) into the same filter, so no SQL is rendered for the source.

• order by is not pushed down, since a scan returns items unordered. Sort downstream, or in the target.

• A select ... where ... written by hand is rejected with an error rather than silently dropping the condition. Use the where option above.

Writing Data

• full-refresh recreates and reloads the table, truncate empties it while keeping the key schema, and incremental upserts by primary key.

• Writes go straight to the table (no temp tables on the target), and the merge strategies insert, update, update_insert and delete_insert are applied by Sling as it streams.

• delete_missing is supported, both soft (the row stays, flagged with _sling_deleted_at) and hard (the row is deleted). See Capture Deletes.

Sling also records _sling_loaded_at on every item it writes.

Type Mapping

DynamoDB attribute	Sling type
N	decimal / number
S	text (ISO strings for dates and timestamps)
BOOL	bool
M, L	json
B, BS	binary
NS, SS, NULL	json / text

Numbers are written as N (never as S), JSON is written as a native map rather than a string, and nulls are omitted from the item.

Limitations

• No SQL engine. Views are not supported (Sling renders view statements away instead of failing), and neither are joins, expressions or order by. Reads are scans: a filter expression is applied by the scan, so the table is still read in full to match items.

• No secondary indexes. DynamoDB keys are declared when the table is created, so Sling does not create indexes (including unique indexes) on DynamoDB targets.

• Key columns only carry real typing. Attribute values keep their DynamoDB type, but only the key attributes are strongly typed by the table definition; Sling infers the rest from the data.

• Item size limit. DynamoDB caps an item at 400 KB, so wide or large rows can be rejected by the service.

If you are facing issues connecting, give sling assist a try, or reach out to us at [e-mail], on discord or open a Github Issue here.