# Introduction

An introduction to Sling

Moving data from one platform to another has long been a critical process. Sling is a modern data movement and transformation platform designed to simplify and streamline data operations. It provides both a powerful CLI tool and a comprehensive platform for managing data workflows between various sources and destinations.

## Core Features

* **Data Movement**: Transfer data between different storage systems and databases efficiently
* **Flexible Connectivity**: Support for numerous databases, data warehouses, and file storage systems
* **Transformation Capabilities**: Built-in data transformation features during transfer
* **Multiple Operation Modes**: Support for various replication modes including full-refresh, incremental, and snapshot
* **Production-Ready**: Deployable with monitoring, scheduling, and error handling

## Key Components

### Sling CLI

{% embed url="<https://f.slingdata.io/videos/sling.cli.demo.2023.10.720.mp4>" %}
Sling CLI Demo
{% endembed %}

The command-line interface provides direct access to Sling's capabilities, perfect for:

* Local development and testing
* CI/CD pipeline integration
* Automated data operations
* Quick data transfers and transformations

You can by running a command like this:

`cat my_file.csv | sling run --tgt-conn MYDB --tgt-object my_schema.my_table`

### Sling Platform

{% embed url="<https://f.slingdata.io/videos/sling.ui.demo.20241121.mp4>" %}
Sling Platform UI
{% endembed %}

{% hint style="info" %}
Want to try the Platform without signing up? Open the [live demo](https://demo.slingdata.io).
{% endhint %}

The web-based platform offers:

* Visual interface for creating and managing data workflows
* Agent based architecture for scalable execution
* Team collaboration features
* Monitoring and alerting
* Centralized connection management
* Job scheduling and orchestration

#### Sling Agents

Agents are the workers that execute your data operations:

* Run in your own infrastructure
* Secure access to your data sources
* Support for both development and production environments

## Common Use Cases

* Database replication and synchronization
* Data warehouse loading and ETL operations
* File system to database ingestion
* Cross-platform data migration
* Backup and archival operations
* Real-time data copying and transformation

## Getting Started

To begin using Sling, you can either:

1. [Install the CLI tool](/sling-cli/getting-started) for local development and testing
2. [Sign up for the Sling Platform](https://github.com/slingdata-io/sling-docs/blob/master/sling-platform/getting-started.md) for a managed experience
3. Use both in combination for a complete data operations solution

Choose the approach that best fits your needs and scale up as your requirements grow.

![](https://static.scarf.sh/a.png?x-pxid=29813085-4cb1-4636-ab5d-cce5dbafc8aa)


# Installation

An introduction to using the Sling CLI tool.

{% embed url="<https://f.slingdata.io/videos/sling.cli.demo.2023.10.720.mp4>" %}
Sling CLI Demo
{% endembed %}

## Getting Started

`sling` CLI is a free tool that allows data extraction and loading from / into many popular databases / storage platforms. Follow the instructions below to install it on your respective operating system.

### Installation

All commands below will install the latest version of sling. If you'd like to determine which version is latest/current, you can check out the [releases page](https://github.com/slingdata-io/sling-cli/releases) on github.

#### One-liner on Mac / Linux

The install script auto-detects your OS and architecture, downloads the latest binary, and adds `sling` to your `PATH`.

```shell
curl -fsSL https://slingdata.io/install.sh | bash

# You're good to go!
sling -h
```

By default sling installs to `~/.sling/bin`. Set `SLING_INSTALL` to override the prefix, or pass a version: `bash -s -- v1.5.17`.

#### One-liner on Windows (PowerShell)

```powershell
irm https://slingdata.io/install.ps1 | iex

# You're good to go!
sling -h
```

The script installs to `$HOME\.sling\bin` and updates your user `PATH`. Set `$env:SLING_INSTALL` before running to override.

#### Brew on Mac

Follow these [directions](https://brew.sh/) to install HomeBrew for mac if not already installed.

```shell
# Install with brew first
brew install slingdata-io/sling/sling

# Once, installed, `sling` should be available
sling -h
```

#### Scoop on Windows

```powershell
scoop bucket add sling https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# You're good to go!
sling -h
```

#### Manual binary download (Linux example)

```shell
# download latest binary
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

# You're good to go!
./sling -h
```

#### Docker

```shell
docker pull slingdata/sling

docker run --rm -i slingdata/sling --help
```

#### Other Binary Downloads

See [Releases](https://github.com/slingdata-io/sling-cli/releases) on Github.

### Setting up your Connections

Sling looks for credentials in several places:

* Environment Variables
* Sling Env File (located at `~/.sling/env.yaml`)
* DBT Profiles Files (located at `~/.dbt/profiles.yml`)

Please see [environment](/sling-cli/environment) for more details.

## Using Sling

`sling` CLI is designed to be easy to use. Say we want to load a CSV file into a PostgreSQL database. We could run the following command:

{% tabs %}
{% tab title="Linux" %}
{% code overflow="wrap" %}

```bash
export MY_PG='postgresql://user:mypassw@pg.host:5432/db1'

sling run --src-stream file:///path/to/myfile.csv --tgt-conn MY_PG --tgt-object public.my_new_data

# OR pipe it in
cat /path/to/myfile.csv | sling run --tgt-conn MY_PG --tgt-object public.my_new_data
```

{% endcode %}
{% endtab %}

{% tab title="Mac" %}
{% code overflow="wrap" %}

```bash
export MY_PG='postgresql://user:mypassw@pg.host:5432/db1'

sling run --src-stream file:///path/to/myfile.csv --tgt-conn MY_PG --tgt-object public.my_new_data

# OR pipe it in
cat /path/to/myfile.csv | sling run --tgt-conn MY_PG --tgt-object public.my_new_data
```

{% endcode %}
{% endtab %}

{% tab title="Windows" %}
{% code overflow="wrap" %}

```powershell
# using windows Powershell
$env:MY_PG = 'postgresql://user:mypassw@pg.host:5432/db1'
sling run --src-stream file://C:/path/to/myfile.csv --tgt-conn MY_PG --tgt-object public.my_new_data

# OR pipe it in
cat C:\path\to\myfile.csv | sling run --tgt-conn MY_PG --tgt-object public.my_new_data
```

{% endcode %}
{% endtab %}

{% tab title="Docker" %}
{% code overflow="wrap" %}

```bash
export MY_PG='postgresql://user:mypassw@pg.host:5432/db1'

docker run --rm -i -e MY_PG -v /path/to/myfile.csv slingdata/sling run --src-stream file:///path/to/myfile.csv --tgt-conn MY_PG --tgt-object public.my_new_data
```

{% endcode %}
{% endtab %}
{% endtabs %}

### With Python

If you have Python `pip` installed, you can simply run:

```bash
pip install sling
```

You call also check out the [Python wrapper](https://github.com/slingdata-io/sling-python) library on github.

```python
from sling import Replication, ReplicationStream

replication = Replication(
  source='MY_PG',
  target='MY_AWS_S3',
  steams={
    "my_table": ReplicationStream(
      sql="select * from my_table",
      object='my_folder/new_file.csv',
    ),
  }
)

replication.run()
```


# Environment

Sling looks for connection credentials in several places:

* [Sling Env File](#sling-env-file-env.yaml) (located at `~/.sling/env.yaml`)
* [Project Env File](#project-env-file-.env.sling) (`.env.sling` in the current working directory)
* [DBT Profiles Files](#dbt-profiles-dbt-profiles.yml) (located at `~/.dbt/profiles.yml`)
* [Environment Variables](#environment-variables)

One of the easiest ways is to manage your connections is to use the `sling conns` sub-command. Follow in the next section.

## Managing Connections

Sling makes it easy to **set**, **list** and **test** connections. You can even see the available streams in a connection by using the **discover** sub-command.

```bash
$ sling conns -h
conns - Manage and interact with local connections

See more details at https://docs.slingdata.io/sling-cli/

  Usage:
    conns [discover|list|set|test]

  Subcommands:
    discover   list available streams in connection
    list       list local connections detected
    test       test a local connection
    unset      remove a connection from the sling env file
    set        set a connection in the sling env file
    exec       execute a SQL query on a Database connection

  Flags:
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.
```

### Set Connections

Here we can easily set a connection with the `sling conns set` command and later refer to them by their name. This ensures credentials are not visible by other users when using process monitors, for example.

{% code overflow="wrap" %}

```bash
# set a connection by providing the key=value pairs
$ sling conns set AWS_S3 type=s3 bucket=sling-bucket access_key_id=ACCESS_KEY_ID secret_access_key="SECRET_ACCESS_KEY"

# we set a database connection with just the url
$ sling conns set MY_PG url='postgresql://postgres:myPassword@pghost:5432/postgres'
```

{% endcode %}

To see what credential keys are necessary/accepted for each type of connector, click below:

* File/Storage Connections (see [here](/connections/file-connections))
* Database Connections (see [here](/connections/database-connections))

### List Connections

Once connections are set, we can run the `sling conns list` command to list our detected connections:

```bash
$ sling conns list
+--------------------------+-----------------+-------------------+
| CONN NAME                | CONN TYPE       | SOURCE            |
+--------------------------+-----------------+-------------------+
| AWS_S3                   | FileSys - S3    | sling env yaml    |
| FINANCE_BQ               | DB - BigQuery   | sling env yaml    |
| DO_SPACES                | FileSys - S3    | sling env yaml    |
| LOCALHOST_DEV            | DB - PostgreSQL | dbt profiles yaml |
| MSSQL                    | DB - SQLServer  | sling env yaml    |
| MYSQL                    | DB - MySQL      | sling env yaml    |
| ORACLE_DB                | DB - Oracle     | env variable      |
| MY_PG                    | DB - PostgreSQL | sling env yaml    |
+--------------------------+-----------------+-------------------+
```

### Test Connections

We can also test a connection by running the `sling conns test` command:

```bash
$ sling conns test LOCALHOST_DEV
9:04AM INF success!
```

### Discover Connections

We can easily discover streams available in a connection with the `sling conns discover` command:

```bash
$ sling conns discover postgres --pattern public.work*
+---+--------+-------------------+-------+---------+
| # | SCHEMA | NAME              | TYPE  | COLUMNS |
+---+--------+-------------------+-------+---------+
| 1 | public | worker_heartbeats | table |      14 |
| 2 | public | workers           | table |      20 |
| 3 | public | workspaces        | table |       9 |
+---+--------+-------------------+-------+---------+

$ sling conns discover aws_s3
+---+------------------+-----------+---------+-------------------------------+
| # | NAME             | TYPE      | SIZE    | LAST UPDATED (UTC)            |
+---+------------------+-----------+---------+-------------------------------+
| 1 | logging/         | directory | -       | -                             |
| 2 | sling_test/      | directory | -       | -                             |
| 3 | work/            | directory | -       | -                             |
| 4 | temp/            | directory | -       | -                             |
| 5 | records.json     | file      | 442 KiB | 2022-12-07 11:05:01 (1y ago)  |
| 6 | test.sqlite.db   | file      | 4.8 MiB | 2022-12-14 21:00:48 (1y ago)  |
| 7 | test1.parquet    | file      | 48 KiB  | 2024-03-31 22:54:52 (29d ago) |
| 8 | test_1000.csv    | file      | 99 KiB  | 2024-02-23 09:53:13 (67d ago) |
+---+------------------+-----------+---------+-------------------------------+
```

Show column level information:

```bash
$ sling conns discover postgres -p public.workspaces --columns
+----------+--------+------------+----+--------------+--------------------------+--------------+
| DATABASE | SCHEMA | TABLE      | ID | COLUMN       | NATIVE TYPE              | GENERAL TYPE |
+----------+--------+------------+----+--------------+--------------------------+--------------+
| postgres | public | workspaces |  1 | id           | bigint                   | bigint       |
| postgres | public | workspaces |  2 | account_id   | bigint                   | bigint       |
| postgres | public | workspaces |  3 | name         | text                     | text         |
| postgres | public | workspaces |  4 | short_name   | varchar                  | string       |
| postgres | public | workspaces |  5 | token        | text                     | text         |
| postgres | public | workspaces |  6 | settings     | jsonb                    | json         |
| postgres | public | workspaces |  7 | created_dt   | timestamp with time zone | timestampz   |
| postgres | public | workspaces |  8 | updated_dt   | timestamp with time zone | timestampz   |
| postgres | public | workspaces |  9 | deleted_dt   | timestamp with time zone | timestampz   |
+----------+--------+------------+----+--------------+--------------------------+--------------+

$ sling conns discover aws_s3 -p test1.parquet --columns
+---------------------------------+----+------------------+----------------+--------------+
| FILE                            | ID | COLUMN           | NATIVE TYPE    | GENERAL TYPE |
+---------------------------------+----+------------------+----------------+--------------+
| s3://my-bucket/test1.parquet    |  1 | id               | INT_64         | bigint       |
| s3://my-bucket/test1.parquet    |  2 | first_name       | UTF8           | string       |
| s3://my-bucket/test1.parquet    |  3 | last_name        | UTF8           | string       |
| s3://my-bucket/test1.parquet    |  4 | email            | UTF8           | string       |
| s3://my-bucket/test1.parquet    |  5 | target           | BOOLEAN        | bool         |
| s3://my-bucket/test1.parquet    |  6 | create_dt        | Timestamp      | datetime     |
| s3://my-bucket/test1.parquet    |  7 | date             | Timestamp      | datetime     |
| s3://my-bucket/test1.parquet    |  8 | rating           | DECIMAL        | decimal      |
| s3://my-bucket/test1.parquet    |  9 | code             | DECIMAL        | decimal      |
| s3://my-bucket/test1.parquet    | 10 | json_data        | UTF8           | string       |
| s3://my-bucket/test1.parquet    | 11 | _sling_loaded_at | INT_64         | bigint       |
+---------------------------------+----+------------------+----------------+--------------+
```

## Credentials Location

### Sling Env File (`env.yaml`)

The Sling Env file is the primary way sling reads connections globally. It needs to be saved in the path `~/.sling/env.yaml` where the `~` denotes the path of the user Home folder, which can have different locations depending on the operating system (see [here for Windows](https://stackoverflow.com/a/42966089/2295355), [here for Mac](https://apple.stackexchange.com/a/51282) and [here for Linux](https://www.linuxshelltips.com/find-user-home-directory-linux/)). Sling automatically creates the `.sling` folder in the user home directory, which is typically as shown below:

* Linux: `/home/<username>/.sling`, or `/root/.sling` if user is `root`
* Mac: `/Users/<username>/.sling`
* Windows: `C:\Users\<username>\.sling`

Once in the user home directory, setting the Sling Env File (named `env.yaml`) is easy, and adheres to the structure below. Running `sling` the first time will auto-create it. You can alternatively provide the environment variable `SLING_HOME_DIR`.

To see what credential keys are necessary/accepted for each type of connector, click below:

* File/Storage Connections (see [here](/connections/file-connections))
* Database Connections (see [here](/connections/database-connections))

```yaml
# Holds all connection credentials for Extraction and Loading
connections:
  marketing_pg:
    url: 'postgres://...' 
    ssh_tunnel: 'ssh://...' # optional
  
  # or dbt profile styled
  marketing_pg:
    type: postgres        
    host: [hostname]      
    user: [username]      
    password: ${PASSWORD} # you can pass in environment variables as well
    port: [port]          
    dbname: [database name]
    schema: [dbt schema]  
    ssh_tunnel: 'ssh://...' 
  
  finance_bq:
    type: bigquery
    method: service-account
    project: [GCP project id]
    dataset: [the name of your dbt dataset]
    keyfile: [/path/to/bigquery/keyfile.json]

# Global variables for specific settings, available to all connections at runtime (Optional)
variables:
  SLING_CLI_TOKEN: xxxxxxxxxxxxxxxx  # picked up machine wide
  SLING_LOG_DIR: ~/.sling/logs       # write debug logs here
  aws_access_key: '...'
  aws_secret_key: '...'
```

### Dot-Env File (`.env.sling`)

Sling automatically loads a `.env.sling` file from the **current working directory** when it starts. This lets you define per-project connections and variables without modifying the global `env.yaml`. This applies to version *1.5.10+*.

This is useful when you have multiple projects, each with their own connections or credentials. Simply place a `.env.sling` file in the project directory, and Sling will pick it up automatically when run from that directory.

The file uses a simple `KEY=VALUE` format (one per line). Comments and blank lines are supported. Values can optionally be wrapped in single or double quotes.

```bash
# .env.sling - project-specific connections

# Database connections
MY_PG='postgresql://user:password@localhost:5432/mydb'
STAGING_PG='postgresql://user:password@staging-host:5432/mydb'

# File storage
MY_S3='{type: s3, bucket: my-project-bucket, access_key_id: AKID, secret_access_key: SECRET}'

# API Connection
SALESFORCE='{ type: api, spec: salesforce, secrets: { client_id: "xxxxxxxx", client_secret: "xxxxxxxx", instance: "mycompany.my.salesforce.com" } }'

# Other variables
SLING_LOG_DIR=/tmp/logs
SLING_LOADED_AT_COLUMN=timestamp
SLING_CLI_TOKEN=xxxxxxxxxxxxxxxx
```

{% hint style="info" %}
**Existing environment variables are not overwritten.** If a variable is already set in the shell environment, the value from `.env.sling` will be ignored. This means shell-level overrides always take precedence.
{% endhint %}

{% hint style="warning" %}
Since `.env.sling` may contain sensitive credentials, make sure to add it to your `.gitignore` to avoid accidentally committing secrets to version control.
{% endhint %}

### Environment Variables

Sling also reads environment variables. Simply export a connection URL (or YAML payload) to the current shell environment to use them.

To see examples of setting environment variables for each type of connector, click below:

* File/Storage Connections (see [here](/connections/file-connections))
* Database Connections (see [here](/connections/database-connections))

{% tabs %}
{% tab title="Mac / Linux" %}
{% code overflow="wrap" %}

```bash
$ export MY_PG='postgresql://user:mypassw@pg.host:5432/db1'
$ export MY_PG='{type: postgres, host: "pg.host", user: user, database: "db1", password: "mypassw", port: 5432}'

$ export MY_SNOWFLAKE='snowflake://user:mypassw@sf.host/db1'
$ export MY_SNOWFLAKE='{type: snowflake, host: "<host>", user: "<user>", database: "<database>", password: "<password>", role: "<role>"}'

$ export ORACLE_DB='oracle://user:mypassw@orcl.host:1521/db1'

$ export BIGQUERY_DB='{type: bigquery, dataset: public, key_file: /path/to/service.json, project: my-google-project}' # yaml or json form is also accepted

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_PG         | DB - PostgreSQL  | env variable    |
| MY_SNOWFLAKE  | DB - Snowflake   | env variable    |
| ORACLE_DB     | DB - Oracle      | env variable    |
| BIGQUERY_DB   | DB - Big Query   | env variable    |
+---------------+------------------+-----------------+
```

{% endcode %}
{% endtab %}

{% tab title="Windows" %}
{% code overflow="wrap" %}

```powershell
$ $env:MY_PG='postgresql://user:mypassw@pg.host:5432/db1'
$ $env:MY_PG='{type: postgres, host: "pg.host", user: user, database: "db1", password: "mypassw", port: 5432}'

$ $env:MY_SNOWFLAKE='snowflake://user:mypassw@sf.host/db1'
$ $env:MY_SNOWFLAKE='{type: snowflake, host: "sf.host", user: user, database: db1, password: "mypassw", role: "<role>"}'

$ $env:ORACLE_DB='oracle://user:mypassw@orcl.host:1521/db1'

$ $env:BIGQUERY_DB='{type: bigquery, dataset: public, key_file: /path/to/service.json, project: my-google-project}' # yaml or json form is also accepted

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_PG         | DB - PostgreSQL  | env variable    |
| MY_SNOWFLAKE  | DB - Snowflake   | env variable    |
| ORACLE_DB     | DB - Oracle      | env variable    |
+---------------+------------------+-----------------+
```

{% endcode %}
{% endtab %}
{% endtabs %}

### DBT Profiles (`~/dbt/profiles.yml`)

Sling also reads dbt profiles connections! If you're already set up with dbt cli locally, you don't need to create additional duplicate connections.

See [here](https://docs.getdbt.com/dbt-cli/configure-your-profile) for more details.

```bash
$ sling conns list
+------------------+------------------+-------------------+
| CONN NAME        | CONN TYPE        | SOURCE            |
+------------------+------------------+-------------------+
| SNOWCASTLE_DEV   | DB - Snowflake   | dbt profiles yaml |
| SNOWCASTLE_PROD  | DB - Snowflake   | dbt profiles yaml |
+------------------+------------------+-------------------+
```

## Location String

The location string is a way to describe where sling should look for a file object or database object. It is used in a few places, such as the [`SLING_STATE`](/sling-cli/cli-pro#file--state-based-incremental-loading) env var, as well as [Hooks](/concepts/hooks) such as [`delete`](/concepts/hooks/delete), [`copy`](/concepts/hooks/copy) and [`inspect`](/concepts/hooks/inspect).

The proper input format is `CONN_NAME/path/to/key` for storage connections, or `CONN_NAME/[database.]schema.table` for database objects.

Local location examples:

* `local/relative/path`
* `local/../parent/relative/path`
* `local//absolute/linux/path`
* `local/C:/absolute/windows/path`

For cloud or remote storage connections (with defined `AWS_S3`, `GCP`, `AZURE`, `SFTP` connections):

* `aws_s3/path/to/folder`
* `gcp/path/to/folder/file.parquet`
* `azure/path/to/folder/file.log`
* `sftp/relative/path/to/folder/file.log`
* `sftp//absolute/path/to/folder/file.log`

Database location examples (for hooks like `inspect`):

* `postgres/public.users` - PostgreSQL table in public schema
* `mysql_db/analytics.events` - MySQL table in analytics schema
* `snowflake/DATABASE.SCHEMA.TABLE` - Snowflake table with explicit database
* `bigquery/project.dataset.table_name` - BigQuery table
* `oracle_db/HR.EMPLOYEES` - Oracle table in HR schema
* `mssql/dbo.customers` - SQL Server table in dbo schema

{% hint style="warning" %}
For **file storage connections** (`local`, `ftp` and `sftp`), you can specify a relative or absolute path. For FTP connections, it will be relative to the default folder of the username connecting.

**Relative Path**: You use the typical single slash (`/`) after the connection name:

* `local/relative/path`
* `sftp/relative/path`
* `ftp/relative/path`

**Absolute Path**: You need to add 2 slashes (`//`) after the connection name:

* `local//absolute/path`
* `local/C:/absolute/path`
* `sftp//absolute/path`
* `ftp//absolute/path`

For **database connections**, use the standard database object naming convention: `connection_name/[database.]schema.table_name`
{% endhint %}


# Running Sling

The `sling run` command is the primary mechanism for executing data movement operations in Sling CLI. It provides a flexible interface for transferring data between various sources and targets, with support for different replication modes and configuration options.

There are 2 primary ways to configure and run sling, using:

* [**CLI Flags**](#cli-flags-overview): quick ad-hoc runs from your terminal shell or script.
* [**Replication**](/concepts/replication): streams defined in a YAML or JSON file.

***

Furthermore, you'll find plenty of examples on how to use Sling:

* [Database to Database](/examples/database-to-database)
* [Database to File](/examples/database-to-file)
* [File to Database](/examples/file-to-database)

## CLI Flags Overview

For quickly running ad-hoc operations from the terminal, using CLI flags is often best. Here are some examples using:

{% code overflow="wrap" %}

```bash
# Load all tables in a schema in with 3 threads
$ export SLING_THREADS=3
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.*' \
    --tgt-conn MY_TARGET_DB \
    --tgt-object 'target_schema.{stream_table}'
    --mode full-refresh
    
# Pipe in your json file and flatten the nested keys into their own columns
$ cat /tmp/my_file.json | sling run --src-options '{"flatten": "true"}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

# Read folder containing many CSV files
$ sling run \
    --src-stream 'file:///tmp/my_csv_folder/' \
    --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' \
    --mode full-refresh

# Load only latest data from one source DB to another.
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_TARGET_DB \
    --tgt-object 'target_schema.target_table' \
    --mode incremental \
    --primary-key 'id' --update-key 'last_modified_dt' 

# Export / Backup database tables to JSON files
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_S3_BUCKET \
    --tgt-object 's3://my-bucket/my_json_folder/' \
    --tgt-options '{"file_max_rows": 100000, "format": "jsonlines"}'
```

{% endcode %}

## Interface Specifications

<table data-full-width="false"><thead><tr><th width="215.4264102691841">CLI Flag</th><th>Description</th></tr></thead><tbody><tr><td><code>--src-conn</code></td><td>The source database connection (name, conn string or URL).</td></tr><tr><td><code>--tgt-conn</code></td><td>The target database connection (name, conn string or URL).</td></tr><tr><td><code>--src-stream</code></td><td>The source table (schema.table), local / cloud file path. Can also be the path of sql file or in-line text to use as query. Use <code>file://</code> for local paths.</td></tr><tr><td><code>--tgt-object</code></td><td>The target table (schema.table) or local / cloud file path. Use <code>file://</code> for local paths. See <a href="/pages/y2M4d0G7ur8y4f4KiQ0i">here</a> for details on runtime variables.</td></tr><tr><td><code>--mode</code></td><td>The target load <a href="/pages/YmhsNnZAWVIOnjeuszZx">mode</a> to use: <code>incremental</code>, <code>truncate</code>, <code>full-refresh</code>, <code>backfill</code> or <code>snapshot</code>. Default is <code>full-refresh</code>.</td></tr><tr><td><code>--primary-key</code></td><td>The column(s) to use as primary key (for <code>incremental</code> mode). If composite key, use a comma-delimited string.</td></tr><tr><td><code>--update-key</code></td><td>The column to use as update key (for <code>incremental</code> mode).</td></tr><tr><td><code>--src-options</code></td><td>In-line options to further configure source (JSON or YAML). See <a href="/pages/3BkvtN9AW4GCDlyu6jGw">here</a> for details.</td></tr><tr><td><code>--tgt-options</code></td><td>In-line options to further configure target (JSON or YAML). See <a href="/pages/dhKMJTL71CTD8kxiAgi4">here</a> for details.</td></tr><tr><td><code>--stdout</code></td><td>Output the stream to standard output (STDOUT).</td></tr><tr><td><code>--select</code></td><td>Select or exclude specific columns from the source stream. (comma separated). Use <code>-</code> prefix to exclude.</td></tr><tr><td><code>--transforms</code></td><td>An object/map, or array/list of built-in transforms to apply to records (JSON or YAML).</td></tr><tr><td><code>--columns</code></td><td>An object/map to specify the type that a column should be cast as (JSON or YAML).</td></tr><tr><td><code>--streams</code></td><td>Only run specific streams from a replication (comma separated). See <a href="/pages/pQWU3jzZUoz5vHLm4CZw">here</a> for details.</td></tr></tbody></table>

## Features

* **Flexible Data Sources**: Supports databases, files, cloud storage, and standard input
* **Multiple Load Modes**: Includes full refresh, incremental, snapshot, and truncate modes
* **Data Transformations**: Allows column selection, type casting, and custom transformations
* **Progress Tracking**: Monitors row counts, bytes transferred, and constraint violations
* **Error Handling**: Provides detailed error reporting and validation

The `sling run` command is designed to be both powerful and flexible, accommodating various data movement scenarios while maintaining ease of use through consistent parameter patterns and comprehensive documentation.


# Global Variables

Learn how to use Global Environment Variables with Sling

Sling utilizes the following global environment variables to further configure the load behavior. You can simply define them in your environment, the `env.yaml` file or the `env` section in a task or replication.

<table><thead><tr><th width="295">Variable Name</th><th>Description</th></tr></thead><tbody><tr><td><code>SLING_HOME_DIR</code></td><td>The sling home directory, which contains <code>env.yaml</code>. Will use <a href="https://github.com/slingdata-io/sling-docs/blob/master/environment.md#sling-env-file-env.yaml">default</a> if not provided.</td></tr><tr><td><code>SLING_LOADED_AT_COLUMN</code></td><td>Whether to add an audit timestamp column named <code>_sling_loaded_at</code> in target object. Accepts values <code>true</code>, <code>false</code>, <code>unix</code> (for <a href="https://www.epochconverter.com/">epoch</a> integer values) or <code>timestamp</code>. <code>true</code> defaults to <code>unix</code>.</td></tr><tr><td><code>SLING_SYNCED_AT_COLUMN</code></td><td>Whether to add sync tracking columns <code>_sling_synced_at</code> (timestamp) and <code>_sling_synced_op</code> (operation type) in target object. The <code>_sling_synced_op</code> column tracks the last operation: <code>I</code> = Insert, <code>U</code> = Update, <code>D</code> = Delete (soft delete). This is useful when streaming from staging to multiple destinations to track when records were last touched. To enable, set to <code>true</code>. When enabled, <code>_sling_synced_at</code> replaces <code>_sling_deleted_at</code> for soft deletes.</td></tr><tr><td><code>SLING_STREAM_URL_COLUMN</code></td><td>If source is file, whether to add a column <code>_sling_stream_url</code> with the source file path / url in target object. To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_TIMEOUT</code></td><td>The maximum number of minutes the sling replication should run. Once reached, it will kill the process. To enable, set a number (<code>SLING_TIMEOUT=10.5</code>)</td></tr><tr><td><code>SLING_RECURSIVE_LIMIT</code></td><td>The number limit of file names to pull, when listing from cloud file systems such as S3, GCP and Azure Storage.</td></tr><tr><td><code>SLING_ROW_ID_COLUMN</code></td><td>Whether to add a column named <code>_sling_row_id</code> in the target object, which will have a random UUIDv7 value. This will be unique. To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_ROW_NUM_COLUMN</code></td><td>If source is file, whether to add a column named <code>_sling_row_num</code> in the target object, which will be the row number of the stream (incremented by record processed). To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_EXEC_ID_COLUMN</code></td><td>Whether to add a column named <code>_sling_exec_id</code> in the target object, which will have the run / execution string (a random UUIDv7 value). This will be unique per run. To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_STATE</code></td><td>The <a href="/pages/eAdVs2BHCgdr6RS8GoJC#location-string">location</a> to read/write information such as incremental values. Proper input format is <code>CONN_NAME/key</code>. For example: <code>POSTGRES/sling_state.state_table</code> , <code>AWS_S3/my/folder</code> or <code>MY_SFTP/my/folder</code>.</td></tr><tr><td><code>SLING_ALLOW_EMPTY</code></td><td>This is useful to create tables / files using the stream columns structure, even if there is no data. To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_DIRECT_INSERT</code></td><td>Tells sling to insert directly into the final table (not create a temp table before). To enable, set to <code>true</code>.</td></tr><tr><td><code>SLING_THREADS</code></td><td>sets the maximum number of concurrent stream runs. Accepts an integer value, default is <code>1</code>.</td></tr><tr><td><code>SLING_RETRIES</code></td><td>sets the maximum number of retries for a failed stream run. Accepts an integer value, default is <code>0</code>.</td></tr><tr><td><code>SLING_KEEP_TEMP</code></td><td>Tells sling to keep any temporary files or tables created in the load process. To enable, set to <code>true</code></td></tr><tr><td><code>SLING_ENV_YAML</code></td><td>Provide the body of the <code>env.yaml</code> file as an environment variable.</td></tr><tr><td><code>SLING_DISABLE_TELEMETRY</code></td><td>this disables any anonymous usage reporting. These are used to improve sling. To disable, set this to <code>true</code>.</td></tr><tr><td><code>SLING_SHOW_PROGRESS</code></td><td>Whether the progress of the stream should be displayed (<code>true</code> or <code>false</code>).</td></tr><tr><td><code>SLING_LOGGING</code></td><td>How sling formats the log lines. Accepts values <code>JSON</code>, <code>NO_COLOR</code> or <code>CONSOLE</code> (default).</td></tr><tr><td><code>SLING_LOG_DIR</code></td><td>Directory for automatic date-based debug log files. When set, Sling creates a <code>sling_debug_YYYY_MM_DD.log</code> file in the specified directory and automatically cleans up old log files, keeping the latest 15. Supports <code>~</code> for home directory (e.g., <code>~/.sling/logs</code>).</td></tr><tr><td><code>SLING_SAMPLE_SIZE</code></td><td>The number of records to process in order to infer column types (especially for file sources). Default is <code>900</code>.</td></tr><tr><td><code>SLING_DUCKDB_COMPUTE</code></td><td>Whether to use DuckDB for writing to parquet files and partitioned parquet/CSV files. DuckDB provides optimized performance for these formats. To disable DuckDB compute, set to <code>false</code>. Default is <code>true</code>.</td></tr><tr><td><code>SLING_OTEL_ENDPOINT</code></td><td>The OpenTelemetry HTTP endpoint URL to export logs to (e.g., <code>http://otel.host.ip:4318/v1/logs</code>). When set, Sling will send structured logs to the specified OTLP endpoint with execution attributes including <code>project_id</code>, <code>job_id</code>, and <code>exec_id</code> if configured. Requires <a href="/pages/QEkscZBpnadf8fqm2m7l">CLI Pro</a>.</td></tr><tr><td><code>SLING_PROXY</code></td><td>Route database connections through a SOCKS5 proxy. Accepts a SOCKS5 URL (e.g., <code>socks5://user:pass@host:1080</code>). To proxy a specific connection instead, set <code>proxy_url</code> in its <code>env.yaml</code> connection entry. Useful for reaching databases behind VPNs or private networks (e.g., Tailscale). Available in v1.5.9+</td></tr></tbody></table>


# CLI Pro

Use CLI Pro to enable advanced features

## CLI Pro Features

Sling CLI Pro extends the core functionality with advanced features designed for production environments and complex data operations.

* ✅ [API Sources](#api-sources) (extract data from any REST API by using Specs)
* ✅ [Parallel Stream Processing](#stream-chunking-and-parallel-processing) (run streams in parallel)
* ✅ [Stream Chunking](#stream-chunking-and-parallel-processing) (split large streams into smaller ones)
* ✅ [Pipelines & Hooks](#pipelines-and-hooks) (such as `http`, `query`, `check` and more)
* ✅ [OpenTelemetry Logging](#opentelemetry-logging) (export structured logs to any OTLP endpoint)
* ✅ [Capture Deletes](#capture-deletes) (similar to CDC)
* ✅ [Staged Transforms](#staged-transforms) (advanced multi-stage transformations with expressions and functions)
* ✅ [File Target Incremental Mode](#state-based-incremental-loading)
* ✅ [State Based Incremental](#state-based-incremental-loading)
* ✅ [ODBC Connections](#odbc-connections) (connect to any database via ODBC drivers)
* ✅ Support Sling and its continuous development

**CLI Pro Max Features**

* ✅ [Change Capture (CDC)](#change-capture-cdc) (continuously replicate row-level changes from transaction logs)
* ✅ [Schema Migration](#schema-migration) (migrate primary keys, foreign keys, indexes, defaults, and more)
* ✅ Priority Support (direct access to the Sling team for faster issue resolution)

{% hint style="success" %}
You can obtain a token for free at <https://dash.slingdata.io>. There is 7-day trial (no credit card needed).

Once you have a token, just put the value into the `SLING_CLI_TOKEN` environment variable before running sling (make sure the version is 1.4+).

For Pricing details see [here](https://slingdata.io/cli-pro).
{% endhint %}

### API Sources

Extract data from any REST API with powerful YAML-based specifications called `API Specs`:

* Define authentication methods (Bearer, Basic, OAuth2)
* Configure endpoints with pagination strategies
* Process responses with JMESPath extraction
* Manage state for incremental synchronization
* Support for queues and dependent requests
* Built-in retry and error handling

In your `env.yaml`:

```yaml
connections:
  stripe_api:
    type: api
    spec: stripe  # Use official spec or custom YAML (e.g. file://path/to/stripe.spec.yaml)
    secrets:
      api_key: sk_live_xxxxxx
```

In your replication:

```yaml
source: stripe_api
target: ducklake

defaults:
  object: stripe.{stream_name}

streams:
  customers:
    mode: incremental
```

See [API Specs](/concepts/api-specs) for complete documentation and examples.

### Stream Chunking & Parallel Processing

Process large datasets efficiently with automatic chunking and parallel execution:

* Break down data into manageable chunks for various modes (`full-refresh`, `truncate`,`incremental`, `backfill`)
* Support for time-based (hours, days, months), numeric, count-based, and expression-based chunks
* Run multiple streams concurrently with automatic retry mechanisms
* Configurable concurrency and retry settings

```yaml
streams:
  my_schema.events:
    mode: full-refresh  # works with various modes
    primary_key: [id]
    update_key: event_date
    source_options:
      chunk_count: 8  # Process in 8 equal sized chunks

  my_schema.orders:
    mode: incremental  # works with various modes
    update_key: order_date
    source_options:
      chunk_size: 7d  # Process in 7-day chunks

env:
  SLING_THREADS: 3   # maximum of 3 streams concurrently
  SLING_RETRIES: 1   # maximum of 1 retry per failed stream
```

Environment variables:

* `SLING_THREADS` sets the maximum number of concurrent stream runs. Accepts an integer value, default is `1`.
* `SLING_RETRIES` sets the maximum number of retries for a failed stream run. Accepts an integer value, default is `0`.

See [Chunking](/examples/database-to-database/chunking) for detailed examples.

### Pipelines & Hooks

Extend functionality with hooks and pipelines to create complex workflows. Hooks are used within replications to execute custom logic before/after operations, while Pipelines are standalone workflows that execute multiple steps in sequence.

Available action types:

| Step Type   | Description                                                | Documentation                                   |
| ----------- | ---------------------------------------------------------- | ----------------------------------------------- |
| Check       | Validate conditions and control flow                       | [Check Hook](/concepts/hooks/check)             |
| Command     | Run any command/process                                    | [Command Hook](/concepts/hooks/command)         |
| Copy        | Transfer files between local or remote storage connections | [Copy Hook](/concepts/hooks/copy)               |
| Delete      | Remove files from local or remote storage connections      | [Delete Hook](/concepts/hooks/delete)           |
| Group       | Run sequences of steps or loop over values                 | [Group Hook](/concepts/hooks/group)             |
| HTTP        | Make HTTP requests to external services                    | [HTTP Hook](/concepts/hooks/http)               |
| Inspect     | Inspect a file or folder                                   | [Inspect Hook](/concepts/hooks/inspect)         |
| List        | List files in folder                                       | [List Hook](/concepts/hooks/list)               |
| Log         | Output custom messages and create audit trails             | [Log Hook](/concepts/hooks/log)                 |
| Query       | Execute SQL queries against any defined connection         | [Query Hook](/concepts/hooks/query)             |
| Read        | Read contents of files from storage connections            | [Read Hook](/concepts/hooks/read)               |
| Replication | Run a Replication                                          | [Replication Hook](/concepts/hooks/replication) |
| Routine     | Execute reusable step sequences from external files        | [Routine Hook](/concepts/hooks/routine)         |
| Store       | Store values for later in-process access                   | [Store Hook](/concepts/hooks/store)             |
| Write       | Write content to files in storage connections              | [Write Hook](/concepts/hooks/write)             |

See [Hooks](/concepts/hooks) and [Pipelines](/concepts/pipeline) for usage examples and patterns.

### Staged Transforms

Transform data with advanced multi-stage processing using expressions and functions:

* Apply transformations in sequential stages with cross-column references
* Create new columns dynamically without modifying source schemas
* Use 50+ built-in functions for string, numeric, date, and conditional operations
* Build complex logic with `if/then/else` conditions and record references

```yaml
streams:
  customers:
    transforms:
      # Stage 1: Clean and normalize data
      - first_name: "trim_space(value)"
        last_name: "trim_space(value)" 
        email: "lower(value)"
      
      # Stage 2: Create computed columns
      - full_name: 'record.first_name + " " + record.last_name'
        email_hash: 'hash(record.email, "md5")'
      
      # Stage 3: Add business logic
      - customer_type: 'record.total_orders >= 50 ? "vip" : "regular"'
        discount_rate: 'record.customer_type == "vip" ? 0.15 : 0.05'
```

See [Transforms](/concepts/replication/transforms) for detailed examples and [Available Functions](/concepts/functions) for all available functions.

### State Based Incremental Loading

Maintain state across file & database loads with intelligent incremental processing:

* Track and resume file processing from last successful position
* Support for incremental writes to databases and files
* Automatic file partitioning and truncation management

See [Database to Database Incremental Loading](/examples/database-to-database/incremental#using-sling_state), [Database to File Incremental Loading](/examples/database-to-file/incremental) and [File to Database Incremental Loading](/examples/file-to-database/incremental) for detailed examples.

### Capture Deletes

Track deleted records using a `_sling_deleted_at` column:

* Automatically detect and mark deleted records
* Maintain historical record states
* Support for soft deletes in target systems

See [Delete Missing Records](/examples/database-to-database/capture_deletes) for implementation details.

### OpenTelemetry Logging

Export structured logs to any OpenTelemetry-compatible endpoint for centralized logging and observability:

* Send logs to any OTLP HTTP endpoint (Grafana, Datadog, Honeycomb, etc.)
* Automatic enrichment with execution attributes (such as `exec_id`, `stream_name`, `object_name`, `row_count`, `status`, `duration`, etc.)
* Structured log records with severity levels and timestamps
* Seamless integration with existing observability infrastructure

Set the `SLING_OTEL_ENDPOINT` environment variable to enable:

```bash
export SLING_OTEL_ENDPOINT='http://otel-collector:4318/v1/logs'
```

### Schema Migration

Migrate database schema attributes along with your data to preserve structure and relationships:

* Primary keys, foreign keys, and indexes
* Auto-increment/identity columns with seed and increment values
* NOT NULL constraints and default values
* Column and table descriptions/comments
* Automatic topological sorting for foreign key dependencies

```yaml
source: mssql
target: postgres

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  dbo.categories:
  dbo.customers:
  dbo.products:    # FK to categories
  dbo.orders:      # FK to customers
  dbo.order_items: # FK to orders and products

env:
  # Enable all schema attributes
  SLING_SCHEMA_MIGRATION: all

  # Or enable specific attributes
  # SLING_SCHEMA_MIGRATION: description, primary_key, foreign_key, indexes
```

Available options: `all`, `primary_key`, `foreign_key`, `indexes`, `auto_increment`, `nullable`, `default_value`, `description`

See [Schema Migration](/examples/database-to-database/schema-migration) for detailed examples and supported databases.

### Change Capture (CDC)

Continuously replicate row-level changes (inserts, updates, deletes) from a source database's transaction log:

* Automatic initial snapshot with chunked, resumable loading
* Incremental change capture from transaction logs (binlog, WAL, etc.)
* Soft delete support to preserve deleted rows with timestamps
* Bounded runs with configurable event limits and duration
* Replay/backfill from earlier positions when needed

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    run_max_events: 10000
    run_max_duration: 10m

streams:
  my_database.customers:
  my_database.orders:
    change_capture_options:
      soft_delete: true
```

See [Change Capture (CDC)](/concepts/change-capture) for complete documentation, supported sources, and examples.

### ODBC Connections

Connect to any database using Open Database Connectivity (ODBC) drivers:

* Access databases that may not have a dedicated Sling connector
* Use standardized ODBC interface for maximum compatibility
* Support for SQL Server, PostgreSQL, MySQL, DB2, SAP HANA, Teradata, and more
* Create custom SQL templates for unsupported database dialects

```yaml
connections:
  my_odbc:
    type: odbc
    conn_string: "Driver={ODBC Driver 18 for SQL Server};Server=myserver;Database=mydb;Uid=myuser;Pwd=mypassword"
```

See [ODBC Connections](/connections/database-connections/odbc) for complete documentation, driver installation, and custom template examples.

## Frequently Asked Questions

**How are tokens validated?**

Tokens are validated through CloudFlare's global network, ensuring high reliability and fast response times worldwide. This validation occurs when the Sling CLI process initializes. If you'd like to confirm validation, run sling in debug mode (with flag `-d`), and you should see a log message: `CLI Pro token validated`.

**Can I get an offline/air-gapped token?**

For air-gapped or high-security environments, we offer offline license tokens. These require yearly renewal by default, but perpetual licenses are also available. Please contact <support@slingdata.io> to request a quote.

**How many subscriptions do I need?**

Each CLI Pro subscription includes 2 tokens:

* 1 Production token: For use in production environments
* 1 Development token: For development and testing

Each subscription is designed for **a single team** within your organization. A "team" refers to a cohesive group managing separate data pipelines, configurations, or business objectives that benefit from isolation for security, governance, or operational independence.

This per-team structure enables us to maintain predictable flat-rate pricing that's sustainable and fair for everyone, allowing us to deliver high-performance features and priority support without usage-based metering.

**What this means for you:**

* Each team or distinct project needs its own subscription
* Subscriptions are for internal use within your organization only
* You can use the production token across all your production environments (servers, containers, etc.)
* Team members can share the development token for testing and collaboration

**Company-wide licensing:** If you prefer a single license for your entire organization, perpetual licenses are available that cover company-wide usage. Contact <support@slingdata.io> to request a quote.

**Examples:**

* A data engineering team handling customer data → 1 subscription
* A separate analytics team working on reporting → 1 subscription
* Multiple independent teams in your organization → 1 subscription per team
* Consultants or freelancers serving multiple clients → 1 subscription per client

**Important Licensing Restrictions:**

{% hint style="warning" %}
**Reselling and Commercial Redistribution Prohibited**

CLI Pro subscriptions are licensed for use by the subscribing organization only. You are prohibited from:

* Reselling or redistributing access to CLI Pro features
* Acting as a service provider offering CLI Pro to third parties
* White-labeling or rebranding CLI Pro as your own service
* Providing commercial access to CLI Pro without proper licensing

**For Consultants and Service Providers:** If you wish to use CLI Pro in a consulting capacity or provide it to your clients, each client organization should have their own CLI Pro subscription. Contact us at <support@slingdata.io> to discuss partner licensing arrangements.

**For System Integrators:** We offer specific partner licensing programs for system integrators and technology partners. Contact us to discuss appropriate licensing for your use case.

Unauthorized reselling or redistribution will result in immediate termination of your subscription and may subject you to legal action.
{% endhint %}

Please use tokens responsibly and in accordance with our [Terms of Service](https://slingdata.io/terms). Each subscription is intended for use within a single organization or team, not for redistribution to external parties.


# VS Code Extension 🔌

The Sling VS Code extension provides schema validation and language server features for SlingData.io configuration files.

## Why Use It

This extension enhances your productivity when working with Sling configuration files by offering:

* **Auto-detection** of Sling configuration files (e.g., pipeline.yaml, replication.yaml, spec.yaml).
* **Schema validation** to ensure your configurations adhere to the correct structure.
* **Auto-completion** for properties, values, and expressions.
* **Hover information** providing inline documentation.
* **Diagnostics** for identifying errors, warnings, and suggestions in your configs.

These features help you write valid and efficient Sling configurations, reducing errors and speeding up development!

## How to Get It

You can get the extension here: <https://marketplace.visualstudio.com/items?itemName=sling.sling-vscode>

Or simply search `Sling` in the Extensions Market place panel in VS Code.

![Install via Marketplace](/files/rAyLtmihnTVFncvXKA7i)

If you're using a forked VS Code editor (such as Cursor or VSCodium), you can install directly from the VSIX package ([download it here](https://marketplace.visualstudio.com/_apis/public/gallery/publishers/sling/vsextensions/sling-vscode/latest/vspackage)).

![Install via VSIX](/files/wWyVjP41rRGswFaokbkf)

## Working with Pipelines & Hooks

This VSCode extension helps tremendously with developing Sling Hooks & Pipelines. The extension provides:

* Schema validation for pipeline YAML files
* Auto-completion for step types, properties, and expression (such as functions)
* Hover documentation for pipeline steps and variables

This makes authoring and maintaining complex pipelines much easier and error-free. For more on pipelines, see the [Pipelines](/concepts/pipeline), [Hooks](/concepts/hooks) and [Functions](/concepts/functions) documentation documentation.

## Screenshots

![Input Suggestions](/files/3cBhLyhMwAt008gbHI4k)

![Function Suggestions](/files/PmCLS6u5V8EhQjypcZTQ)


# Working with AI 🤖

Using Sling CLI with AI coding assistants like Claude Code

Sling CLI integrates seamlessly with AI coding assistants through the **slingdata-ai** plugin, enabling you to research APIs, design data integrations, and debug configurations using natural language—all within your AI assistant. See github repo <https://github.com/slingdata-io/slingdata-ai> for resources.

## Overview

The slingdata-ai plugin provides:

* **Slash commands** for direct execution of replications and pipelines
* **Specialized agents** for complex workflows (API research, spec building, replication design)
* **Skills** that load contextual documentation automatically
* **MCP integration** that connects your AI assistant to Sling CLI tools

**Supported AI Assistants:**

* [Claude Code](https://claude.ai/code) (primary)
* [GitHub Copilot](https://github.com/features/copilot)
* [Google Gemini](https://gemini.google.com/)
* [Cline](https://github.com/cline/cline)
* Other [MCP-compatible assistants](https://modelcontextprotocol.io/)

## Installation

### From Marketplace (Recommended)

If using Claude Code with marketplace support:

```bash
/plugin marketplace add slingdata-io/slingdata-ai
/plugin install sling@slingdata-ai
```

### Local Development

For contributing or local development:

```bash
claude --plugin-dir /path/to/slingdata-ai
```

### Prerequisites

* **Sling CLI v1.5.0+** must be installed
* Connections configured in `~/.sling/env.yaml`
* AI assistant with MCP support

Install Sling CLI:

{% tabs %}
{% tab title="macOS" %}

```bash
curl -fsSL https://slingdata.io/install.sh | bash
```

{% endtab %}

{% tab title="Linux" %}

```bash
curl -fsSL https://slingdata.io/install.sh | bash
```

{% endtab %}

{% tab title="Windows" %}

```powershell
irm https://slingdata.io/install.ps1 | iex
```

{% endtab %}
{% endtabs %}

## Quick Start

### 1. Configure a Connection

See the [Environment documentation](/sling-cli/environment) for details on managing connections.

```bash
sling conns set MY_POSTGRES type=postgres host=localhost user=postgres database=mydb
```

### 2. Use Commands in AI Assistant (Claude Code)

List connections:

```
/sling:conns list
```

Test a connection:

```
/sling:conns test MY_POSTGRES
```

Discover tables:

```
/sling:conns discover MY_POSTGRES public.*
```

### 3. Build with Natural Language

Instead of writing YAML manually, describe what you want:

**Example: API Integration**

```
"Research the Stripe API and create a specification for extracting customers and invoices"
```

The AI will:

1. Use the `api-researcher` agent to analyze Stripe's API documentation
2. Invoke `api-spec-builder` to create the YAML [API specification](/concepts/api-specs)
3. Use `api-spec-tester` to validate and debug
4. Generate a [replication](/concepts/replication) config to sync data to your database

**Example: Database Replication**

```
"Design a replication from MY_POSTGRES to MY_SNOWFLAKE that syncs the public.users and public.orders tables incrementally"
```

The AI will use the `replication-builder` agent to create a proper [replication](/concepts/replication) YAML with [incremental settings](/concepts/replication/modes).

## Slash Commands (Claude Code)

Slash commands provide direct access to Sling CLI operations:

### /sling:run

Execute a [replication](/concepts/replication) or [pipeline](/concepts/pipeline) file with validation.

**Usage:**

```
/sling:run /path/to/replication.yaml
/sling:run /path/to/pipeline.yaml
```

The command will:

* Parse and validate the YAML file
* Check connection availability
* Execute the replication or pipeline
* Display progress and results

### /sling:conns

Manage connections interactively.

**List all connections:**

```
/sling:conns list
```

**Test a connection:**

```
/sling:conns test MY_POSTGRES
```

**Discover streams (tables/files/endpoints):**

```
/sling:conns discover MY_POSTGRES
/sling:conns discover MY_POSTGRES public.*
/sling:conns discover MY_S3 data/*.csv
```

## Specialized Agents

Agents are AI sub-processes that handle complex, multi-step tasks autonomously. They have access to specific tools and are optimized for particular workflows.

### API Specification Agents

Build custom REST API connectors through a research → build → test workflow.

#### api-researcher

Research REST API documentation to gather authentication methods, endpoints, pagination patterns, and rate limits.

**Example:**

```
"Use the api-researcher agent to analyze the Shopify Admin API documentation"
```

**Outputs:**

* Authentication type and credentials needed
* Available endpoints and their purposes
* Pagination strategy (cursor, offset, page-based)
* Rate limit information

#### api-spec-builder

Create Sling [API specification](/concepts/api-specs) YAML files from research findings.

**Example:**

```
"Build an API spec for Shopify with orders, customers, and products endpoints"
```

**Outputs:**

* Complete `shopify.yaml` specification
* [Authentication](/concepts/api-specs/authentication) configuration
* Endpoint definitions with [pagination](/concepts/api-specs/request)
* [Response processors](/concepts/api-specs/response)

### Data Integration Agents

#### replication-builder

Design replication configurations for moving data between databases, files, and APIs.

**Example:**

```
"Create a replication from MY_POSTGRES to MY_SNOWFLAKE for the sales schema"
```

**Outputs:**

* Replication YAML with proper modes (full-refresh vs incremental)
* Stream selection and transformations
* Performance optimizations

#### pipeline-builder

Design multi-step data [workflows](/concepts/pipeline) with validation, transformations, and notifications.

**Example:**

```
"Create a pipeline that:
1. Syncs data from Stripe to Postgres
2. Runs a dbt transformation
3. Sends a Slack notification on completion"
```

**Outputs:**

* [Pipeline](/concepts/pipeline) YAML with sequential steps
* [Hook](/concepts/hooks) configurations for notifications
* Error handling and retries

## Skills

Skills are topic-specific documentation modules that load automatically based on keywords in your conversation. They provide contextual guidance without requiring explicit invocation.

### Available Skills

| Skill                   | Triggers                      | Purpose                                              |
| ----------------------- | ----------------------------- | ---------------------------------------------------- |
| `sling`                 | "sling", "data integration"   | Platform overview and tools                          |
| `sling-connections`     | "connection", "connect to"    | [Connection management](/sling-cli/environment)      |
| `sling-replications`    | "replication", "sync", "copy" | [Data movement configs](/concepts/replication)       |
| `sling-pipelines`       | "pipeline", "workflow"        | [Multi-step orchestration](/concepts/pipeline)       |
| `sling-transforms`      | "transform", "convert"        | [Data transformation functions](/concepts/functions) |
| `sling-hooks`           | "hook", "before", "after"     | [Pre/post actions](/concepts/hooks)                  |
| `sling-troubleshooting` | "error", "debug", "fix"       | Error diagnosis                                      |
| `sling-api-specs`       | "api spec", "rest api"        | [API specification building](/concepts/api-specs)    |

**Example:**

When you ask:

```
"How do I add a webhook notification after my replication completes?"
```

The `sling-hooks` skill automatically loads, providing context about:

* [Hook types](/concepts/hooks) (http, sql, check, command)
* Placement options (pre/post replication)
* Configuration examples
* [Variable access](/concepts/replication/runtime-variables) in hooks

## Common Workflows

### Building a Custom API Connector

**Goal:** Extract data from a REST API that Sling doesn't natively support. See the [API Specifications concept guide](/concepts/api-specs) for detailed information.

**Steps:**

1. **Research the API**

```
"Research the Zendesk API focusing on tickets, users, and organizations endpoints"
```

→ Uses `api-researcher` agent

2. **Build the specification**

```
"Create a Zendesk API spec with OAuth2 authentication and the endpoints we researched"
```

→ Uses `api-spec-builder` agent

3. **Test the spec**

```
/sling:conns set MY_ZENDESK type=api spec=zendesk secrets='{ client_id: xxx, client_secret: xxx }'
/sling:conns test MY_ZENDESK
```

→ Uses `api-spec-tester` agent if issues arise

4. **Create replication**

```
"Create a replication from MY_ZENDESK to MY_POSTGRES"
```

→ Uses `replication-builder` agent

5. **Execute**

```
/sling:run zendesk-replication.yaml
```

### Database Migration

**Goal:** Migrate a database from [Postgres](/connections/database-connections/postgres) to [Snowflake](/connections/database-connections/snowflake). See [database-to-database examples](/examples/database-to-database).

**Steps:**

1. **Set up connections**

```
/sling:conns test MY_POSTGRES
/sling:conns test MY_SNOWFLAKE
```

2. **Design replication**

```
"Create a replication from MY_POSTGRES to MY_SNOWFLAKE for all tables in the public schema with incremental sync where possible"
```

See [replication modes](/concepts/replication/modes) and [incremental examples](/examples/database-to-database/incremental).

3. **Review and adjust** The AI generates a replication YAML. You can refine:

```
"Exclude the public.logs table and use full-refresh for public.dim_products"
```

4. **Execute**

```
/sling:run pg-to-snowflake.yaml
```

### ETL Pipeline with Validation

**Goal:** Build a [pipeline](/concepts/pipeline) that syncs data, validates it, and sends notifications. See [pipeline examples](/concepts/pipeline/examples).

**Steps:**

1. **Describe the pipeline**

```
"Create a pipeline that:
- Syncs Stripe charges to MY_POSTGRES
- Runs a SQL check that revenue is > 0
- Sends a Slack webhook on success
- Sends an email alert on failure"
```

2. **Review generated YAML** The `pipeline-builder` agent creates a pipeline with:

* [Replication step](/concepts/hooks/replication)
* [SQL check hook](/concepts/hooks/check)
* [HTTP webhook](/concepts/hooks/http) for Slack
* [HTTP webhook](/concepts/hooks/http) for email alerts

3. **Execute**

```
/sling:run stripe-validation-pipeline.yaml
```

## MCP Tools Reference

The plugin communicates with Sling CLI through [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) tools. These are invoked automatically by the AI but understanding them helps debug issues.

For detailed information about the MCP server, configuration options, and using Sling MCP with other AI assistants ([Claude Desktop](https://claude.ai/download), [VSCode Copilot](https://code.visualstudio.com/docs/copilot/overview), ChatGPT), see the [MCP Server documentation](/sling-cli/ai/mcp).

### connection

Manage [connections](/sling-cli/environment) (list, test, discover, set). See [database connections](/connections/database-connections), [storage connections](/connections/file-connections), and [API connections](/connections/api-connections).

**Actions:**

* `list` - Show all connections
* `test` - Test connection validity
* `discover` - List streams (tables/files/endpoints)
* `set` - Create or update connection

### database

Query databases and retrieve metadata.

**Actions:**

* `query` - Execute SELECT queries
* `get_schemata` - Get schema/table/column metadata
* `get_columns` - Get column details

### file\_system

List, copy, and inspect files.

**Actions:**

* `list` - List files/directories
* `copy` - Copy between [storage connections](/connections/file-connections)
* `inspect` - Get file metadata

### replication

Manage [replications](/concepts/replication).

**Actions:**

* `parse` - Validate YAML syntax
* `compile` - Full validation
* `run` - Execute replication

### pipeline

Manage [pipelines](/concepts/pipeline).

**Actions:**

* `parse` - Validate YAML syntax
* `run` - Execute pipeline

### api\_spec

Work with [API specifications](/concepts/api-specs).

**Actions:**

* `parse` - Load and validate spec
* `test` - Test API endpoints

## Troubleshooting

### Plugin Not Loading

**Check installation:**

```bash
claude plugin list
```

You should see `sling@slingdata-ai` in the list.

**Verify Sling CLI:**

```bash
sling --version
```

Should return v1.5.0 or higher.

### MCP Connection Issues

**Check MCP server status:** The plugin uses `sling mcp` as the MCP server. Test it:

```bash
sling mcp
```

Should start an MCP server on stdio.

**Configuration location:** `.mcp.json` in the plugin directory configures the server.

**For more MCP troubleshooting:** See the [MCP Server troubleshooting section](/sling-cli/ai/mcp#troubleshooting) for log locations, debug output, and detailed diagnostics.

### Commands Not Working

**Check syntax:**

```
# Correct
/sling:run /path/to/file.yaml

# Incorrect
/sling run /path/to/file.yaml  (missing colon)
```

**Check file paths:** Use absolute paths or paths relative to current working directory.

### Agent Failures

**Enable debug mode:** When testing connections or API specs, add debug flag:

```
/sling:conns test MY_API --debug
```

**Check credentials:** Verify secrets in `~/.sling/env.yaml`:

```yaml
connections:
  MY_API:
    type: api
    spec: my_spec
    secrets:
      api_key: ${API_KEY}  # Must be set in environment
```

**Review agent output:** Agents provide detailed error messages. Look for:

* Authentication failures (401/403)
* Rate limit errors (429)
* Invalid endpoints (404)
* Malformed requests (400)

## Best Practices

### Use Natural Language First

Instead of manually writing YAML, describe what you want:

```
"Sync PostgreSQL users table to Snowflake incrementally using updated_at column"
```

The AI will generate proper configuration and explain design decisions.

### Iterate with Agents

For complex tasks, work step-by-step with agents:

1. Research (api-researcher)
2. Cross-reference (api-cross-referencer)
3. Build (api-spec-builder)
4. Test (api-spec-tester)

Each agent focuses on its specialty, improving quality.

### Use Skills for Learning

Skills auto-load based on keywords. To learn about a topic, ask questions:

```
"What transformation functions are available?"  → loads sling-transforms skill (see [Functions](../concepts/functions.md))
"How do I handle errors in pipelines?"          → loads sling-hooks skill (see [Hooks](../concepts/hooks.md))
"How do I set up OAuth2 for an API?"            → loads sling-api-specs skill (see [Authentication](../concepts/api/authentication.md))
```

## What's Next?

* [MCP Server](/sling-cli/ai/mcp) - Use Sling with [Claude Desktop](https://claude.ai/download), [VSCode Copilot](https://code.visualstudio.com/docs/copilot/overview), and other AI assistants
* [Environment Setup](/sling-cli/environment) - Configure connections and credentials
* [Replications](/concepts/replication) - Deep dive into replication configs
* [Pipelines](/concepts/pipeline) - Build multi-step workflows
* [API Specifications](/concepts/api-specs) - Create custom API connectors
* [Hooks](/concepts/hooks) - Add pre/post actions


# MCP Server

Use Sling CLI as an MCP (Model Context Protocol) server to enable AI assistants to interact with databases, files, and APIs

## Overview

The Sling CLI includes a built-in MCP (Model Context Protocol) server that enables AI assistants like Claude, ChatGPT, and GitHub Copilot to interact with your data infrastructure through a standardized interface. MCP is an open protocol that allows AI models to connect to external tools and data sources safely and efficiently.

By running `sling mcp`, you expose Sling's powerful data movement and transformation capabilities to AI assistants, enabling them to:

* Query and explore databases across 30+ database systems
* Manage files across cloud storage providers (S3, Azure, GCS, etc.)
* Execute data replications and pipelines
* Create and test API specifications
* Discover schemas, tables, and columns

{% embed url="<https://f.slingdata.io/videos/mcp.demo.20251201.mp4>" %}
Sling MCP Demo
{% endembed %}

## Core Capabilities

The Sling MCP server exposes six main tools that AI assistants can use:

### 1. Connection Tool

Manages connections to databases, file systems, and APIs.

**Actions:**

* `list` - List all configured connections
* `discover` - Discover tables, files, or endpoints in a connection
* `test` - Test connection validity
* `set` - Create or update a connection
* `docs` - Fetch connection documentation

### 2. Database Tool

Provides database-specific operations for querying and schema exploration.

**Actions:**

* `docs` - Fetch database documentation
* `query` - Execute SQL queries (read-only by default)
* `get_schemata` - Get detailed schema information (databases, schemas, tables, columns)
* `get_schemas` - List available schemas
* `get_columns` - Get column metadata for specific tables

### 3. File System Tool

Manages files across local and cloud storage systems.

**Actions:**

* `list` - List files and directories
* `copy` - Copy files between connections
* `inspect` - Get file metadata and statistics
* `docs` - Fetch file system documentation

### 4. API Spec Tool

Creates and manages API specifications for REST APIs.

**Actions:**

* `parse` - Parse and validate API specification files
* `test` - Test API endpoints defined in specifications
* `docs` - Fetch API specification documentation

### 5. Replication Tool

Executes data replication configurations.

**Actions:**

* `parse` - Parse replication YAML files
* `compile` - Compile and validate replications
* `run` - Execute replications
* `docs` - Fetch replication documentation

### 6. Pipeline Tool

Manages and executes data pipelines.

**Actions:**

* `parse` - Parse pipeline configurations
* `run` - Execute pipelines
* `docs` - Fetch pipeline documentation

## Installation

### Prerequisites

1. **Install Sling CLI**: Follow the [installation guide](/sling-cli/getting-started)
2. **Verify installation**: Run `sling --version`
3. **Set up connections**: Configure your database and storage connections using [environment variables](/sling-cli/environment)

### VSCode with GitHub Copilot

GitHub Copilot in VSCode supports MCP servers through workspace or user configuration:

#### Workspace Configuration

Create `.vscode/mcp.json` in your project root:

```json
{
  "servers": {
    "sling": {
      "type": "stdio",
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "your-token-here"
      }
    }
  }
}
```

#### User Configuration (Global)

1. Open Command Palette (`Cmd+Shift+P` or `Ctrl+Shift+P`)
2. Run `MCP: Add Server`
3. Select "Global"
4. Enter configuration:

```json
{
  "name": "sling",
  "type": "stdio",
  "command": "sling",
  "args": ["mcp"],
  "env": {
    "SLING_CLI_TOKEN": "your-token-here"
  }
}
```

#### Using with Copilot

1. Open Chat view (`Ctrl+Alt+I` or `Cmd+Alt+I`)
2. Select "Agent mode" from the dropdown
3. Click "Tools" button to see available Sling tools
4. Start using Sling commands in your prompts

![Sling MCP on VSCode](/files/RJ7E383gmukDIyGnGKvi)

![Sling MCP on VSCode](/files/MjXOPp9QGe5RMIe0xQV6)

![Sling MCP on VSCode](/files/tCFF364dN99QVhgvQgT1)

### Claude Desktop

Claude Desktop supports MCP servers through a configuration file. Here's how to set up Sling:

{% tabs %}
{% tab title="macOS" %}

1. Open the configuration file:

```bash
open ~/Library/Application\ Support/Claude/claude_desktop_config.json
```

2. Add the Sling MCP server configuration:

```json
{
  "mcpServers": {
    "sling": {
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "your-token-here"
      }
    }
  }
}
```

3. Restart Claude Desktop
4. Look for the MCP indicator (🔌) in the bottom-right corner of the chat input
   {% endtab %}

{% tab title="Windows" %}

1. Open the configuration file at:

```
%APPDATA%\Claude\claude_desktop_config.json
```

2. Add the Sling MCP server configuration:

```json
{
  "mcpServers": {
    "sling": {
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "your-token-here"
      }
    }
  }
}
```

3. Restart Claude Desktop
4. Look for the MCP indicator (🔌) in the bottom-right corner of the chat input
   {% endtab %}
   {% endtabs %}

![Sling MCP on Claude Desktop](/files/cg2mrr6jfJIaZnwOzgkB)

![Sling MCP on Claude Desktop](/files/1zgz6LhJMr2GAatu7LR9)

![Sling MCP on Claude Desktop](/files/Hnh3uGXvpD8N7kRBnBIG)

### Claude Code

Claude Code supports MCP servers at three configuration scopes:

#### Local Scope (Project-specific)

```bash
# Add for current project only
claude mcp add sling --args "mcp" --env SLING_CLI_TOKEN=your-token-here
```

#### Project Scope (Shared with team)

Create `.mcp.json` in your project root:

```json
{
  "servers": {
    "sling": {
      "type": "stdio",
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "${SLING_CLI_TOKEN}"
      }
    }
  }
}
```

#### User Scope (Global)

```bash
# Add globally for all projects
claude mcp add sling --scope user --args "mcp" --env SLING_CLI_TOKEN=your-token-here
```

Alternatively, edit `~/.claude.json` directly:

```json
{
  "mcpServers": {
    "sling": {
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "your-token-here"
      }
    }
  }
}
```

### ChatGPT Desktop

{% hint style="warning" %}
**Note:** As of 2025, OpenAI has announced plans to add native MCP support to ChatGPT Desktop, but implementation is pending. Check the [OpenAI Developer Community](https://community.openai.com/) for updates.
{% endhint %}

For now, you can use bridge solutions like the [chatgpt-mcp server](https://github.com/xncbf/chatgpt-mcp) that enables MCP interaction through the ChatGPT macOS app:

```json
{
  "mcpServers": {
    "chatgpt-sling-bridge": {
      "command": "uvx",
      "args": ["chatgpt-mcp"],
      "env": {
        "SLING_MCP_COMMAND": "sling mcp",
        "SLING_CLI_TOKEN": "your-token-here"
      }
    }
  }
}
```

## Usage Examples

### Querying a Database

**Simple Analysis Prompt to AI Assistant:**

{% code overflow="wrap" %}

```
Use sling connection `postgres_prod` to query the sales table in my warehouse connection and show me the top 10 revenue generating products this month
```

{% endcode %}

The assistant will construct and execute:

{% code overflow="wrap" %}

```json
{
  "action": "query",
  "input": {
    "connection": "postgres_prod",
    "query": "SELECT product_id, SUM(revenue) as total_revenue FROM sales WHERE date >= '2025-01-01' GROUP BY product_id ORDER BY total_revenue DESC LIMIT 10"
  }
}

```

{% endcode %}

**Table Comparison Prompt to AI Assistant:**

{% code overflow="wrap" %}

```
Use sling in connection `snowflake_dw` to compare the tables: dbt_dev.core_transactions (dev table) and finance.core_transactions (prod table). Compare the counts and null counts as well as distinct counts.
```

{% endcode %}

The assistant will construct and execute multiple queries and return a summary.

### Discovering Database Tables

**Prompt to AI Assistant:**

{% code overflow="wrap" %}

```
Using Sling, show me all tables in my `postgres_rds` connection that start with "customer_"
```

{% endcode %}

The assistant will use:

```json
{
  "action": "discover",
  "input": {
    "connection": "postgres_rds",
    "pattern": "*.customer_*"
  }
}
```

### Copying Files Between Storage Systems

**Prompt to AI Assistant:**

{% code overflow="wrap" %}

```
Use sling to copy all CSV files from connection `aws_s3` folder "raw/2025/" to connection `AZURE_PROD` "processed/" folder
```

{% endcode %}

The assistant will execute:

```json
{
  "action": "copy",
  "input": {
    "source_location": "aws_s3/raw/2025/*.csv",
    "target_location": "azure_prod/processed/",
    "recursive": true
  }
}
```

## MCP Prompts

The Sling MCP server provides specialized prompts that guide AI assistants through complex API specification workflows. These prompts are pre-built conversation templates that help with creating, extending, and debugging API integrations.

### api\_spec\_create\_spec

Creates a complete Sling API specification from scratch by analyzing API documentation and building endpoints with authentication, pagination, and data extraction configuration.

**Arguments:**

| Argument          | Required | Description                                       |
| ----------------- | -------- | ------------------------------------------------- |
| `spec_name`       | Yes      | Name for the API specification (used as filename) |
| `spec_file_path`  | No       | Full file path for the spec file                  |
| `connection_name` | Yes      | Name for the API connection to create and test    |
| `api_docs_url`    | Yes      | URL to the API documentation website              |
| `endpoint_names`  | Yes      | Comma-separated list of endpoint names to include |
| `additional_info` | No       | Additional instructions or requirements           |

**Workflow:**

1. Fetches Sling API spec documentation
2. Analyzes the target API documentation (using browser if available)
3. Creates the specification file
4. Creates or uses existing connection
5. Tests and iterates until endpoints work correctly

### api\_spec\_add\_endpoint

Adds a new endpoint to an existing Sling API specification by analyzing endpoint documentation and implementing proper configuration.

**Arguments:**

| Argument            | Required | Description                                    |
| ------------------- | -------- | ---------------------------------------------- |
| `spec_file_path`    | Yes      | Full file path to the existing spec file       |
| `endpoint_name`     | Yes      | Name of the new endpoint to add                |
| `endpoint_docs_url` | No       | URL to the specific endpoint documentation     |
| `additional_info`   | No       | Additional instructions for the implementation |

**Workflow:**

1. Fetches Sling API spec documentation
2. Loads and parses the existing specification
3. Analyzes endpoint documentation
4. Implements the new endpoint following existing patterns
5. Tests until the endpoint returns data successfully

### api\_spec\_debug\_endpoint

Debugs and fixes issues with an existing endpoint in a Sling API specification by analyzing errors and adjusting configuration.

**Arguments:**

| Argument          | Required | Description                           |
| ----------------- | -------- | ------------------------------------- |
| `spec_file_path`  | Yes      | Full file path to the spec file       |
| `endpoint_name`   | Yes      | Name of the endpoint to debug and fix |
| `additional_info` | No       | Additional context about the issues   |

**Workflow:**

1. Fetches Sling API spec documentation
2. Loads and examines the current specification
3. Analyzes API documentation for verification
4. Tests and diagnoses issues
5. Fixes configuration and iterates until successful

**Common issues checked:**

* Authentication (token format, headers, auth type)
* URL construction (base URLs, parameter encoding)
* Data extraction (JMESPath expressions)
* Pagination (next page logic, stop conditions)
* Rate limiting (request rates, backoff strategies)

## Troubleshooting

### Log Locations

* **Claude Desktop**: `~/Library/Logs/Claude/` (macOS) or `%APPDATA%\Claude\logs\` (Windows)
* **Claude Code**: View logs with `claude mcp logs sling`
* **VSCode**: Use Command Palette > "MCP: Show Logs"

### Debug Output

Enable trace logging for detailed debugging:

```json
{
  "mcpServers": {
    "sling": {
      "command": "sling",
      "args": ["mcp"],
      "env": {
        "SLING_CLI_TOKEN": "your-token-here",
        "DEBUG": "TRACE"
      }
    }
  }
}
```

### Resources

* [Model Context Protocol Documentation](https://modelcontextprotocol.io/)
* [Sling Environment Configuration](/sling-cli/environment)
* [Sling Replications Guide](/concepts/replication)
* [Sling Pipelines Guide](/concepts/pipeline)

{% hint style="success" %}
**Pro Tip:** Start with simple operations like listing connections and discovering tables before moving to complex replications and pipelines. This helps you understand how the AI assistant interacts with your data infrastructure.
{% endhint %}


# Sling Platform

{% hint style="info" %}
Want to try the Platform without signing up? Open the [live demo](https://demo.slingdata.io).
{% endhint %}

{% embed url="<https://f.slingdata.io/videos/sling.ui.demo.20241121.mp4>" %}
Sling Platform UI
{% endembed %}

## Why Sling Platform

Sling Platform provides a comprehensive web-based interface for managing data operations at scale. Key benefits include:

* Visual interface for creating and managing data workflows
* Agent-based architecture for secure and scalable execution
* Team collaboration features
* Centralized connection management
* Built-in monitoring and alerting
* Job scheduling and orchestration

## Platform Features

The paid platform plan includes exclusive capabilities:

* ✅ [Smart Editor (IDE)](/sling-platform/platform/editor) - Live compilation, auto-complete, syntax highlighting, and validation for replications
* ✅ [Alerting](/sling-platform/platform/editor#managing-jobs) - Email, Slack, and MS Teams notifications for job status (error, warning, success)
* ✅ [Job Scheduling](/sling-platform/platform/editor#managing-jobs) - Schedule and orchestrate data jobs with cron expressions
* ✅ [Self-Hosted Agents](/sling-platform/platform/agents) - Deploy agents in your infrastructure for secure data access
* ✅ [REST API](/sling-platform/platform/api) - Programmatic access to connections, jobs, executions, and git sync
* ✅ [Job Run History](/sling-platform/platform/editor#job-run-history) - View detailed history and logs for all job executions

### CLI Pro Features

The paid platform plan also includes all of the [CLI Pro](/sling-cli/cli-pro) features:

* ✅ [API Sources](/sling-cli/cli-pro#api-sources) (extract data from any REST API by using Specs)
* ✅ [Parallel Stream Processing](/sling-cli/cli-pro#stream-chunking-and-parallel-processing) (run streams in parallel with automatic retries)
* ✅ [Stream Chunking](/sling-cli/cli-pro#stream-chunking-and-parallel-processing) (split large streams into smaller chunks)
* ✅ [Pipelines & Hooks](/sling-cli/cli-pro#pipelines-and-hooks) (such as `http`, `query`, `check`, `copy`, `command` and more)
* ✅ [OpenTelemetry Logging](/sling-cli/cli-pro#opentelemetry-logging) (export structured logs to any OTLP endpoint)
* ✅ [Capture Deletes](/sling-cli/cli-pro#capture-deletes-cdc) (similar to CDC)
* ✅ [Staged Transforms](/sling-cli/cli-pro#staged-transforms) (advanced multi-stage transformations with expressions and functions)
* ✅ [State Based Incremental](/sling-cli/cli-pro#state-based-incremental-loading) (file target incremental mode)
* ✅ [Schema Migration](/sling-cli/cli-pro#schema-migration) (migrate primary keys, foreign keys, indexes, defaults, and more)
* ✅ [ODBC Connections](/sling-cli/cli-pro#odbc-connections) (connect to any database via ODBC drivers)

### Advanced Plan Features

The following features are available with the [Advanced plan](https://slingdata.io/platform/#pricing):

* ✅ [Platform Self-Hosting](/sling-platform/platform/self-hosting) - Host the entire platform in your private network for full control and security
* ✅ [Git Integration](/sling-platform/platform/api#git-sync) - Connect to GitHub, GitLab, or Bitbucket for version control and CI/CD workflows
* ✅ User Roles - Define granular access controls with custom user roles and permissions
* ✅ Audit Logs - Comprehensive logging and tracking of all platform activities for compliance

![Sling Platform UI](/files/33ReA95FxIhUTrOLvgSy)

## Sign Up

1. Visit [platform.slingdata.io](https://platform.slingdata.io) to create your account
2. Choose your preferred authentication method
3. Create a Project
4. Get Slinging

<div align="center"><img src="/files/Wvr69UMaO9dGZYoRiuEP" alt="Sling Platform Sign Up" width="500"></div>

## Creating a Project

1. After logging in, You will need to create a project
2. Enter a project name, and click "Create Project"

<div align="center"><img src="/files/v1wDjvAh3xMn5UjNbLsX" alt="Sling Platform Create Project" width="500"></div>

Once your project is created, you can:

* Set up connections to your data sources and targets
* Deploy agents to securely access your data
* Create and schedule replications
* Monitor your data operations
* Invite team members to collaborate

{% hint style="info" %}
Each project is isolated from other projects, meaning that you can have multiple projects, each with their own connections, agents, replications, etc.
{% endhint %}

For detailed instructions on specific features, please refer to the relevant sections in the documentation.


# Architecture

The Sling Data Platform consists of three main components that work together to provide a scalable and secure data movement solution:

1. Control Server & Database
2. Agents (Self-hosted)
3. Frontend UI

## System Overview

![Sling Platform Architecture](/files/9pJdPa1D5YGfszHAbnj8)

### Control Plane

The Control Plane is the central coordination point of the Sling Platform that:

* Manages user authentication and authorization
* Stores configuration data (connections, replications, etc.)
* Coordinates job scheduling and execution
* Maintains system state and job history
* Communicates with agents via secure NATS websockets
* Provides REST API endpoints for the frontend UI

### Agents

Agents are distributed workers that:

* Run the Sling CLI to execute data jobs
* Operate within your infrastructure for secure data access
* Connect to the Control Server via secure NATS websockets
* Can be deployed across multiple environments (dev, prod, etc.)
* Scale horizontally to handle increased workload
* Provide secure access to data sources without exposing credentials

### Frontend UI

The Frontend UI provides:

* Web-based interface for managing Sling operations
* Secure communication with Control Server via HTTPS
* Visual workflow creation and management
* Real-time monitoring and alerting
* Team collaboration features
* Connection management interface

## Communication Flow

1. Users interact with the Frontend UI over HTTPS
2. Frontend UI communicates with Control Server via REST API
3. Control Server coordinates with Agents using NATS secure websockets
4. Agents execute jobs and report status back to Control Server
5. Control Server updates Frontend UI with job status and results

## Security

* All communication between components uses encryption (HTTPS, WSS)
* Agents run in your infrastructure, keeping sensitive data and credentials secure. No ports need to be opened.
* Authentication and authorization managed by Control Server
* Credentials stored securely in Control Server database, or on your self-hosted agent.


# Agents

## What is an agent?

To put it simply, an agent is a container for your data pipelines. It runs in the background and waits for data jobs (such as replications). You can run an agent using Docker or by using the Sling CLI. An agent can run on any machine (Windows, Mac, Linux) and connects to the control server via a secure WebSocket connection (TLS). There is no need to open ports or configure firewalls.

<div align="center"><img src="/files/V6gOPJxoKGbFyRU5BAHX" alt="Sling Platform Agent" width="500"></div>

## How does the agent work?

Upon starting, the agent will connect to the control server, using a provided API Key, and will wait for jobs to be assigned to it. It will then execute the job (data will flow only through it) and then send the results back to the control server (such as logs, errors, number of rows processed, etc). The agent will continue to run and wait for new jobs to be assigned to it.

## Running an Agent

You can simply use the `sling agent` command from the CLI or Docker.

```bash
$ sling agent
agent - Manage the local sling agent

See more details at https://docs.slingdata.io/sling-cli/

  Usage:
    agent [run]

  Subcommands: 
    run   run the sling agent

  Flags: 
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.
```

Before running it, you need to obtain the `SLING_AGENT_KEY` from the Sling Platform UI.

Once you have the key, you can start the agent by running `sling agent run`.

{% tabs %}
{% tab title="Linux" %}
{% code overflow="wrap" %}

```bash
export SLING_AGENT_KEY='....'

sling agent run
```

{% endcode %}
{% endtab %}

{% tab title="Mac" %}
{% code overflow="wrap" %}

```bash
export SLING_AGENT_KEY='....'

sling agent run
```

{% endcode %}
{% endtab %}

{% tab title="Windows" %}
{% code overflow="wrap" %}

```powershell
# using windows Powershell
$env:SLING_AGENT_KEY='....'

sling agent run
```

{% endcode %}
{% endtab %}

{% tab title="Docker CLI" %}
{% code overflow="wrap" %}

```bash
export SLING_AGENT_KEY='....'

docker run -d -e SLING_AGENT_KEY slingdata/sling agent run
```

{% endcode %}
{% endtab %}

{% tab title="Docker Compose" %}
{% code overflow="wrap" %}

```yaml
services:
  sling-agent:
    image: slingdata/sling
    command: agent run
    container_name: sling-agent
    restart: unless-stopped
    environment:
      SLING_AGENT_KEY: '....'
    volumes:
      - agent_data:/home/sling

volumes:
  agent_data:
```

{% endcode %}
{% endtab %}
{% endtabs %}

## How do development and production agents differ?

A development agent is a special agent, with defined limits, that is used for development purposes. It is not billed, nor intended for production use, and therefore will not be selected for scheduled jobs. All plans include one development agent. On the other hand, a production agent will be selected for scheduled jobs, and requires a paid plan.

### What about my Connection Credentials?

If you choose to self-host your Sling Agent, you can define a local env.yaml file to store your connection credentials (on the machine). Using this method, the sensitive credentials will never leave the agent. You can also store your credentials in the Sling Control Server, per project, which is securely transmitted over TLS and encrypted at rest.

## Custom Agent Docker Image

If you need to build your own custom Sling agent Docker image (for example, to include additional dependencies or configurations), you can use the official Dockerfiles as a starting point:

* **AMD64 Architecture**: [Dockerfile](https://github.com/slingdata-io/sling-cli/blob/main/cmd/sling/Dockerfile)
* **ARM64 Architecture**: [Dockerfile.arm64](https://github.com/slingdata-io/sling-cli/blob/main/cmd/sling/Dockerfile.arm64)

### Building a Custom Image

Here's an example of how to build a custom agent image with additional Python packages:

```dockerfile
# Start from the official Sling image
FROM slingdata/sling:latest
# FROM slingdata/sling:latest-arm64 # for ARM64

# Switch to root to install additional packages
USER root

# Install additional system dependencies if needed
RUN apt-get update && apt-get install -y \
    your-package-here \
    && rm -rf /var/lib/apt/lists/*

# Install additional Python packages
RUN pip install --no-cache-dir \
    pandas \
    sqlalchemy \
    your-custom-package

# Switch back to sling user
USER sling

# The default command is already set to run the agent
```


# Connections

Connections are used to connect to external systems. For example, they are used to read and write data from databases, files, and other systems when running replications.

![Sling Platform Connections](/files/qcGOuM1sfm23CkH0gocn)

## Project env.yaml vs Agent env.yaml

Each project has an internal `env.yaml` file, which is used to store the connection credentials. This file is used to authenticate the agent when it needs to access the external system. When using the project env.yaml file, the credentials are securely transmitted over TLS and encrypted at rest.

You can also define a local `env.yaml` file (saved on the agent machine), which can be used to authenticate the agent when it needs to access the external system. When using the local env.yaml file, the credentials are never transmitted outside the agent machine. Only the name and type of the connection are sent to the control server for displaying on the UI.

### Project env.yaml

To add a connection to the project env.yaml file, you can click on the `New` button on the connections page and then select the type of connection you want to add.

![Sling Platform Connections New](/files/1tF5FapE8XqEfAMia0pm)

You can then fill in the connection details and click on the `Save` button to add the connection to the project env.yaml file. You can also click on the `Test` button to test the connection before saving it.

![Sling Platform Connections Save](/files/0XJOg9iBRrXPyMLcuJ1d)

## Explore Connections

You can explore the files or tables in a connection by clicking one on the left sidebar. This allows you to see which objects are available, as well as previewing the data inside. This is useful for getting familiar with the data, and for finding the correct object names for your replications.&#x20;

![Sling Platform Connections Explore](/files/aw8iML5WBOetOom3YJvg)

{% hint style="warning" %}
Note that objects are accessed in a read-only fashion, therefore no changes can be made to the data.
{% endhint %}


# Editor

The Sling Data Platform Editor is a powerful tool for creating and editing replications. It allows you to visually build replications by incorporating an IDE like experience:

* Live compilation of replications
* Interactive visual editor for building replications
* Auto-complete for source and target objects
* Syntax highlighting for SQL and Sling
* Validation for your replications

![Sling Platform Editor](/files/kfYR7zjjghKXNBeqXq1I)

## Referring to Query Files

In order to refer to query files (to avoid putting large SQL text in your replication YAML), you can specify query files that exist in your project. Always use the relative path to the file from the root of your project. For example, the file `my_long_query.sql` below is in folder `queries` in the root:

```yaml
source: MY_SOURCE
target: MY_TARGET

streams:
  my_custom_stream:
    sql: 'file://queries/my_long_query.sql'
```

## Managing Jobs

You can manage your jobs by clicking on the `Jobs` button on the bottom bar. This allows you to see the status of your jobs, as well as creating new jobs.

<div align="center"><img src="/files/7SWxWFMe0qftSXhLeQrS" alt="Sling Platform Editor Jobs" width="500"></div>

### Using a Specific Sling CLI Version

By default, any job will use the latest Sling CLI version to run the jobs. If you'd like to specify a specific version to use, you can do so with:

<div align="center"><img src="/files/GoWl9udIJZ7V2jsbQCVi" alt="Specific version"></div>

### Using the Dev Build

You can set your job to use the latest dev build (which is a preview of the upcoming release).

<div align="center"><img src="/files/kM8aiJCDpt5buuLmipea" alt="Setting the env var"></div>

## Job Run History

You can view the history of your jobs by clicking on the `History` button on the bottom bar. This allows you to see the status of your job runs, as well as details about each run.

![Sling Platform Job Run Details](/files/r8hm3AL1LQQWbRVq11Ut)


# API

Base URL: `https://api.slingdata.io`

## Authentication & Headers

All API requests require authentication using a Sling Project Token in the header:

```
Authorization: Sling-Project-Token xxxxxxxxxxxx
Content-Type: application/json
```

Project tokens can be created and managed through the Sling Data Platform, in the `Settings > API Tokens` section. Each token is associated with a specific project.

## Connections

### List Connections

```
GET /connection/list
```

Returns a list of all connections configured for the project.

**Response**

```json
{
  "connections": [
    {
      "name": "string",
      "type": "string"
    }
  ]
}
```

### Test Connection

```
POST /connection/test
```

Tests if a connection is valid and accessible.

**Request Body**

```json
{
  "name": "string",  // Connection name
}
```

**Response**

```json
{
  "valid": true|false,
  "error": "string"  // Present if valid is false
}
```

## Files

### Get File

```
GET /project/file/get
```

Retrieves contents of a specific project file.

**Query Parameters**

* `name`: File path relative to project root

**Response**

```json
{
  "name": "string",
  "body": "string",
  "is_dir": boolean,
  "updated": "datetime"
}
```

### List Files

```
POST /project/file/list
```

Lists all files in the project.

**Response**

```json
{
  "files": [
    {
      "name": "string",
      "body": "string",
      "is_dir": boolean,
      "updated": "datetime"
    }
  ]
}
```

### Save File

```
POST /project/file/save
```

Creates or updates a project file. If the file is a valid Sling job file (replication, pipeline, monitor, or query), it will be parsed and validated before saving. A default job will be automatically created for new Sling job files.

**Request Body**

```json
{
  "file": {
    "name": "string",  // File path relative to project root
    "body": "string",  // File contents
    "is_dir": false     // Set to true to create a directory (body is ignored)
  }
}
```

**Response**

```json
{
  "file": {
    "name": "string",
    "body": "string",
    "is_dir": false,
    "size": 123,
    "default_job_id": "string",
    "updated": "datetime"
  }
}
```

## Jobs

### List Jobs

```
POST /project/job/list
```

Lists all jobs in the project.

**Request Body**

```json
{
  "name": "string",  // Optional job name
  "file_name": "string",  // Optional file path relative to project root
  "type": "string"  // Optional job type filter (replication, pipeline, monitor)
}
```

**Response**

```json
{
  "jobs": [
    {
      "id": "string",
      "name": "string",
      "type": "string",
      "status": "string",
      "file_name": "string"
    }
  ]
}
```

### Get Job

```
GET /project/job/get
```

Gets details of a specific job.

**Query Parameters**

* `job_id`: ID of the job

**Response**

```json
{
  "id": "string",
  "name": "string",
  "type": "string",
  "status": "string",
  "file_name": "string",
  "config": {}
}
```

### Save Job

```
POST /project/job/save
```

Creates or updates a job configuration. The job must reference an existing project file.

**Request Body**

```json
{
  "data": {
    "id": "string",         // Optional — omit to create a new job
    "name": "string",       // Job name
    "type": "string",       // Job type: replication, pipeline, monitor, or query
    "file_name": "string",  // File path relative to project root
    "active": false,        // Whether the job schedule is active
    "schedules": [],        // Array of cron expressions
    "timezone": "string",   // Optional timezone for schedules (e.g. "America/New_York")
    "streams": [],          // Optional array of stream names (empty means all)
    "tags": [],             // Optional array of tags
    "group": "string",      // Optional job group for concurrency limiting
    "config": {             // Optional job configuration
      "mode": "string",           // Sync mode (full-refresh, incremental, etc.)
      "threads": 1,               // Number of parallel threads
      "retries": 0,               // Number of retries on failure
      "timeout": 0,               // Timeout in seconds
      "variables": []             // Array of variable maps
    }
  }
}
```

**Response**

```json
{
  "job": {
    "id": "string",
    "name": "string",
    "type": "string",
    "status": "string",
    "file_name": "string",
    "active": false,
    "schedules": [],
    "scheduled": "datetime",
    "config": {}
  }
}
```

### Run Job

```
POST /project/job/run
```

Triggers execution of a job.

**Request Body**

```json
{
  "job_id": "string"
}
```

**Response**

```json
{
  "exec_id": "string"
}
```

## Executions

### Cancel Execution

```
POST /execution/cancel
```

Cancels a running job run / execution.

**Request Body**

```json
{
  "exec_id": "string"
}
```

### List Executions

```
GET /execution/list
```

Returns a list of recent job runs / executions.

**Query Parameters**

* `status` (optional): Filter by execution status: `running` | `success` | `error` | `warning` | `skipped`
* `limit` (optional): Number of records to return (max 100)

**Response**

```json
{
  "executions": [
    {
      "exec_id": "string",
      "status": "string",
      "start_time": "datetime",
      "end_time": "datetime",
      "error": "string",
      ...
    }
  ]
}
```

### Get Execution

```
GET /execution/list
```

Fetches details for a single execution by passing an `exec_id` filter to the list endpoint. The first matching record is the execution.

**Query Parameters**

* `filters` (required): URL-encoded JSON object containing `exec_id`. Example: `filters=%7B%22exec_id%22%3A%22exc_abc123%22%7D` (decodes to `{"exec_id":"exc_abc123"}`)

**Response**

```json
{
  "executions": [
    {
      "exec_id": "string",
      "job_id": "string",
      "status": "string",
      "start_time": "datetime",
      "end_time": "datetime",
      "rows": 0,
      "bytes": 0,
      "error": "string",
      ...
    }
  ]
}
```

If no execution matches the given `exec_id`, the `executions` array will be empty.

**Example**

```bash
curl -G https://api.slingdata.io/execution/list \
  -H "Authorization: Sling-Project-Token $SLING_PROJECT_TOKEN" \
  --data-urlencode 'filters={"exec_id":"exc_abc123"}'
```

### Get Execution Logs (Replication Tasks)

```
POST /execution/replication-tasks
```

Returns per-task records for a replication execution, including the full log output for each stream task. Use this to retrieve logs for a completed or running replication execution.

**Request Body**

```json
{
  "exec_id": "string",        // Required — execution ID to fetch tasks for
  "stream_id": "string",      // Optional — filter to a specific stream name
  "data": {
    "status": "string",       // Optional — filter by task status (running, success, error, etc.)
    "exclude_output": false   // Optional — set to true to omit the heavy log `output` column
  }
}
```

Set `exclude_output` to `false` (or omit it) to include log output in the response. Set it to `true` for lightweight task listings without the log payload.

**Response**

```json
{
  "tasks": [
    {
      "exec_id": "string",
      "stream_id": "string",
      "status": "string",
      "start_time": "datetime",
      "end_time": "datetime",
      "rows": 0,
      "bytes": 0,
      "output": "string",     // Full log output for the task (present when exclude_output is false)
      "error": "string",
      ...
    }
  ]
}
```

**Example**

```bash
curl -X POST https://api.slingdata.io/execution/replication-tasks \
  -H "Authorization: Sling-Project-Token $SLING_PROJECT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "exec_id": "exc_abc123",
    "data": {
      "exclude_output": false
    }
  }'
```

## Git Sync

### Pull from Git

```
POST /project/git/pull
```

Pulls changes from the configured Git repository and syncs them to the project. This is useful for triggering a sync when changes are pushed to the repository (e.g., via a CI/CD webhook) instead of waiting for the automatic polling interval.

**Prerequisites**

* Git integration must be configured and enabled for the project
* The configured branch must exist in the remote repository

**Response**

```json
{
  "result": {
    "pulled": true,
    "created": 2,
    "updated": 1,
    "deleted": 0,
    "sha": "abc123...",
    "created_files": [...],
    "updated_files": [...],
    "deleted_file_names": [...]
  }
}
```

**Response Fields**

* `pulled`: Whether files were pulled from the repository
* `created`: Number of new files created
* `updated`: Number of existing files updated
* `deleted`: Number of files deleted
* `sha`: The commit SHA that was synced
* `created_files`: Array of newly created file objects
* `updated_files`: Array of updated file objects
* `deleted_file_names`: Array of deleted file names

**Example: GitHub Actions Webhook**

You can trigger a git pull when changes are pushed to your repository using GitHub Actions:

```yaml
name: Sync to Sling Platform

on:
  push:
    branches:
      - main  # or your configured branch

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Sling Git Pull
        run: |
          curl -X POST https://api.slingdata.io/project/git/pull \
            -H "Authorization: Sling-Project-Token ${{ secrets.SLING_PROJECT_TOKEN }}" \
            -H "Content-Type: application/json"
```


# Self-Hosting

Deploy and manage Sling Platform in your own infrastructure

{% hint style="success" %}
Self-hosting is available with the [**Advanced plan**](https://slingdata.io/platform/#pricing). This feature allows you to deploy the entire Sling Platform within your private network for full control and security. **Start your 10-day trial at** [**https://dash.slingdata.io/self-platform**](https://dash.slingdata.io/self-platform)**, no credit card needed!**
{% endhint %}

## Overview

Self-hosting Sling Platform enables organizations to:

* Maintain complete control over data and infrastructure
* Meet strict compliance and security requirements
* Deploy in air-gapped or private environments
* Customize deployment configurations

The `slingdata/sling-platform` image provides a self-hosted Sling data platform for managing data pipelines, connections, and agents.

## Quick Start

Run below and access on <http://localhost:7878>:

```bash
docker run -d \
  -p 7878:7878 \
  -p 4443:4443 \
  -e SLING_PLATFORM_LICENSE="your-license-key" \
  -e SLING_PLATFORM_ENCRYPTION_KEY="your-32-char-encryption-key" \
  -e SLING_PLATFORM_ADMIN_CREDENTIALS='{"email":"admin@example.com","password":"your-secure-password"}' \
  -v sling-platform-data:/home/sling \
  slingdata/sling-platform:latest
```

## Required Environment Variables

* **`SLING_PLATFORM_LICENSE`** - Your Sling Platform license key. This can be either a 36-character UUID for cloud-verified licenses or an offline license token for air-gapped environments. Obtain a trial key from <https://dash.slingdata.io/self-platform> (no credit card needed).
* **`SLING_PLATFORM_ENCRYPTION_KEY`** - A 32-character encryption key used to encrypt sensitive data like connection credentials. Generate a secure random string (`openssl rand -hex 16`).
* **`SLING_PLATFORM_ADMIN_CREDENTIALS`** - JSON payload containing the initial admin user credentials. need `email` and `password` keys.

{% hint style="success" %}
**Air-Gapped/Offline License**: By default, license tokens are validated through CloudFlare's global network for high reliability and fast response times. If you require an air-gapped or high-security environment, we offer offline license tokens that don't require internet connectivity for validation. These require yearly renewal by default, but perpetual licenses are also available. Please contact <support@slingdata.io> to request a quote.
{% endhint %}

## Optional Environment Variables

* **`SLING_PLATFORM_HOST`** - The external URL where your platform will be accessible. This is used for agent connections and web UI access. The default is `http://localhost:7878`
* **`SLING_PLATFORM_SMTP_CREDENTIALS`** - SMTP configuration for email notifications. If your SMTP service requires SSL, specify the optional `security` key with the value `SSL`. If not provided, email notifications will be disabled. For example: `{"host":"smtp.gmail.com","port":587,"user":"your-email@gmail.com","password":"your-app-password","from":"Sling Platform <noreply@yourdomain.com>", "security": "SSL"}`
* **SLING\_LOGGING** - The default logger uses ANSI colors. You can change this if you'd like, accepts values: `NO_COLOR` or `JSON`.

## Connecting Agents

To connect Sling agents to your platform, use the `slingdata/sling` image with these environment variables:

```bash
docker run -d \
  -e SLING_PLATFORM_HOST="http://your-platform-host:7878" \
  -e SLING_AGENT_KEY="your-agent-key-from-platform-ui" \
  -v sling-agent-data:/home/sling \
  slingdata/sling agent run
```

### Agent Environment Variables

* **`SLING_AGENT_KEY`** - Agent authentication key obtained from the platform UI under Agents section
* **`SLING_PLATFORM_HOST`** - Same URL as your platform instance
* **`SLING_PLATFORM_HOST_WS`** - Optional, the web-socket host to use for NATS connection, e.g. `ws://your-platform-host:4443`. Default uses the same host as `SLING_PLATFORM_HOST`.

## Docker Compose Example

```yaml
services:
  sling-platform:
    image: slingdata/sling-platform
    ports:
      - "7878:7878"  # HTTP port for UI and API
      - "4443:4443"  # NATS WebSocket port for agent communication
    expose:
      - "7878"
      - "9876" # Postgres port accessible with user `sling`
      - "4443"
    environment:
      SLING_PLATFORM_LICENSE: "your-license-key"
      SLING_PLATFORM_ENCRYPTION_KEY: "your-32-char-encryption-key"
      SLING_PLATFORM_ADMIN_CREDENTIALS: '{"email":"admin@example.com","password":"secure-password"}'
      SLING_PLATFORM_HOST: "http://external.ip:7878"  # External URL for web UI access
      SLING_PLATFORM_SMTP_CREDENTIALS: '{"host":"smtp.gmail.com","port":587,"user":"your-email","password":"your-password","from":"Sling <noreply@yourdomain.com>"}'
    volumes:
      - sling-platform-data:/home/sling
    restart: unless-stopped
    
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7878/status/get"]
      interval: 5s
      timeout: 2s
      retries: 3

  # Add Agent(s) as needed
  # sling-agent:
  #   image: slingdata/sling:latest
  #   environment:
  #     SLING_PLATFORM_HOST: "http://sling-platform:7878"  # Internal container communication
  #     SLING_AGENT_KEY: "your-agent-key"
  #   command: agent run
  #   volumes:
  #     - sling-agent-data:/home/sling
  #   depends_on:
  #     - sling-platform
  #   restart: unless-stopped

volumes:
  sling-platform-data:
  sling-agent-data:
```

## Ports

* **7878** - Main HTTP port for web UI and API access
* **4443** - Web-Socket port for agent communication (via NATS)

## Volumes

* `/home/sling` - Platform data directory.

{% hint style="info" %}
**Embedded Database**: The sling-platform container includes an embedded PostgreSQL database. Database files are stored under `/home/sling/platform/postgres` within the container. This database is accessible on port 9876 with username `sling` (read-only) and password `pgpass.$PREFIX`, where `$PREFIX` is the first 6 characters of `SLING_PLATFORM_ENCRYPTION_KEY`. The database name is `sling_platform`.

Here is the env var to have Sling read from the Platform database (replace `$PREFIX` and `$SLING_PLATFORM_HOST`): `SLING_PLATFORM_DB='postgresql://sling:pgpass.$PREFIX@$SLING_PLATFORM_HOST:9876/sling_platform?sslmode=disable'`
{% endhint %}

## Custom Agent Docker Image

If you need to build your own custom Sling agent Docker image (for example, to include additional dependencies or configurations), you can use the official Dockerfiles as a starting point:

* **AMD64 Architecture**: [Dockerfile](https://github.com/slingdata-io/sling-cli/blob/main/cmd/sling/Dockerfile)
* **ARM64 Architecture**: [Dockerfile.arm64](https://github.com/slingdata-io/sling-cli/blob/main/cmd/sling/Dockerfile.arm64)

See [here](/sling-platform/platform/agents#custom-agent-docker-image) for an example.

## Getting Started

1. Start the platform container with required environment variables
2. Access the web UI at your configured `SLING_PLATFORM_HOST`
3. Log in with your admin credentials
4. Create agent keys in the Agents section
5. Deploy agents using the `slingdata/sling` image
6. Start creating connections and data pipelines

## Security Notes

* Use strong, unique values for `SLING_PLATFORM_ENCRYPTION_KEY` (32 characters)
* Store sensitive environment variables securely
* Use HTTPS in production environments

## Support

For support, reach out to <support@slingdata.io>.

## Kubernetes

```yaml
# Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: sling-platform
---
# ConfigMap for environment variables
apiVersion: v1
kind: ConfigMap
metadata:
  name: sling-platform-config
  namespace: sling-platform
data:
  SLING_PLATFORM_HOST: "http://external.ip:8080"  # Update with your actual external IP/domain
---
# Secret for sensitive environment variables
apiVersion: v1
kind: Secret
metadata:
  name: sling-platform-secrets
  namespace: sling-platform
type: Opaque
stringData:
  SLING_PLATFORM_LICENSE: "your-license-key"
  SLING_PLATFORM_ENCRYPTION_KEY: "your-32-char-encryption-key"
  SLING_PLATFORM_ADMIN_CREDENTIALS: '{"email":"admin@example.com","password":"secure-password"}'
  SLING_PLATFORM_SMTP_CREDENTIALS: '{"host":"smtp.gmail.com","port":587,"user":"your-email","password":"your-password","from":"Sling <noreply@yourdomain.com>"}'
  SLING_AGENT_KEY: "your-agent-key"
---
# PersistentVolumeClaim for platform data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sling-platform-data
  namespace: sling-platform
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  # storageClassName: your-storage-class  # Uncomment and specify if needed
---
# PersistentVolumeClaim for agent data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sling-agent-data
  namespace: sling-platform
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  # storageClassName: your-storage-class  # Uncomment and specify if needed
---
# Sling Platform Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sling-platform
  namespace: sling-platform
  labels:
    app: sling-platform
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sling-platform
  template:
    metadata:
      labels:
        app: sling-platform
    spec:
      securityContext:
        fsGroup: 999
        runAsUser: 999
      containers:
      - name: sling-platform
        image: slingdata/sling-platform:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 7878
          name: http
          protocol: TCP
        - containerPort: 4443
          name: nats
          protocol: TCP
        - containerPort: 9876
          name: postgres
          protocol: TCP
        env:
        - name: SLING_PLATFORM_HOST
          valueFrom:
            configMapKeyRef:
              name: sling-platform-config
              key: SLING_PLATFORM_HOST
        - name: SLING_PLATFORM_LICENSE
          valueFrom:
            secretKeyRef:
              name: sling-platform-secrets
              key: SLING_PLATFORM_LICENSE
        - name: SLING_PLATFORM_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: sling-platform-secrets
              key: SLING_PLATFORM_ENCRYPTION_KEY
        - name: SLING_PLATFORM_ADMIN_CREDENTIALS
          valueFrom:
            secretKeyRef:
              name: sling-platform-secrets
              key: SLING_PLATFORM_ADMIN_CREDENTIALS
        - name: SLING_PLATFORM_SMTP_CREDENTIALS
          valueFrom:
            secretKeyRef:
              name: sling-platform-secrets
              key: SLING_PLATFORM_SMTP_CREDENTIALS
              optional: true
        volumeMounts:
        - name: platform-data
          mountPath: /home/sling
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "4Gi"
            cpu: "2"
      volumes:
      - name: platform-data
        persistentVolumeClaim:
          claimName: sling-platform-data
      restartPolicy: Always
---
# Sling Platform Service
apiVersion: v1
kind: Service
metadata:
  name: sling-platform
  namespace: sling-platform
  labels:
    app: sling-platform
spec:
  type: ClusterIP
  ports:
  - port: 7878
    targetPort: 7878
    protocol: TCP
    name: http
  - port: 4443
    targetPort: 4443
    protocol: TCP
    name: nats
  - port: 9876
    targetPort: 9876
    protocol: TCP
    name: postgres
  selector:
    app: sling-platform
---
# Sling Agent Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sling-agent
  namespace: sling-platform
  labels:
    app: sling-agent
spec:
  # Each agent requires a unique SLING_AGENT_KEY, so this deployment must have replicas: 1
  # To deploy additional agents, duplicate this deployment with a new name and a different SLING_AGENT_KEY
  replicas: 1  
  selector:
    matchLabels:
      app: sling-agent
  template:
    metadata:
      labels:
        app: sling-agent
    spec:
      containers:
      - name: sling-agent
        image: slingdata/sling:latest
        imagePullPolicy: Always
        args: ["agent", "run"]
        env:
        - name: SLING_PLATFORM_HOST
          value: "http://sling-platform:7878"
        - name: SLING_AGENT_KEY
          valueFrom:
            secretKeyRef:
              name: sling-platform-secrets
              key: SLING_AGENT_KEY
        volumeMounts:
        - name: agent-data
          mountPath: /home/sling
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "8Gi"
            cpu: "4"
      volumes:
      - name: agent-data
        persistentVolumeClaim:
          claimName: sling-agent-data
      restartPolicy: Always
```

## Troubleshooting

### Common Issues

1. **Platform won't start**
   * Check container logs: `docker logs sling-platform`
   * Verify environment variables are set correctly
   * Ensure data volume has sufficient permissions
2. **Agents not connecting**
   * Verify network connectivity between agent and platform
   * Check agent key is valid and properly set
   * Ensure platform URL is accessible from agent
   * Review agent logs for error messages
3. **Performance issues**
   * Increase resource limits (CPU/memory)
   * Scale agents horizontally for more concurrent jobs
   * Monitor disk space for data volume

### Getting Help

If you are facing issues setting up, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).

{% hint style="warning" %}
Self-hosting requires careful consideration of security, backup, and maintenance procedures. Ensure your team has the necessary expertise to manage the deployment.
{% endhint %}


# Deploy from CLI

The `sling project` command helps you manage projects on the Sling Data Platform. It provides several subcommands for project management and job operations.

```bash
$ sling project
project - Manage a project on the Sling Data Platform

See more details at https://docs.slingdata.io/

  Usage:
    project [init|status|jobs|sync]

  Subcommands: 
    init     create or link to a sling project
    status   see project status
    jobs     manage project jobs
    sync     Push local file changes

  Flags: 
    --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.
```

## Prerequisites

* A valid Sling Project Token (`SLING_PROJECT_TOKEN` environment variable, obtained from the `Settings` section)
* For self-hosted platform setups, the `SLING_PLATFORM_HOST` environment variable pointing to your platform server (e.g. `https://sling.mycompany.com`)

## Commands

### Initialize Project

Links your local directory to a Sling project:

```bash
# navigate to the root folder of your project first
$ sling project init
```

This command will:

* Create or link to an existing Sling project
* Generate a `.sling.json` file in your project directory
* Configure project settings like paths and ID

#### Project File (.sling.json)

The project configuration is stored in `.sling.json`:

```json
{
  "id": "project_id",
  "paths": ["subfolder1", "subfolder2"]
}
```

* `id`: The Sling project ID
* `paths`: Array of subfolders to monitor for files (relative to project root). This is optional. If not provided, the root directory will be used.

### Check Project Status

View the current project status and configuration:

```bash
$ sling project status
```

This displays:

* Project ID and name
* Organization details
* Owner information
* Project folder location
* Configured project paths
* Additional project status details

### Manage Jobs

#### List Jobs

View all jobs in the project:

```bash
$ sling project jobs list
```

Shows a table with:

* Job ID
* Name
* File Name
* Status
* Active status
* Schedule information

#### View Job History

See run history for project jobs:

```bash
$ sling project jobs history
```

#### Trigger Job

Run a specific job by its ID:

```bash
$ sling project jobs trigger job_123456
```

### Sync Files

Push local file changes to the Sling platform (one way sync):

```bash
$ sling project sync [--force]
```

The sync command:

* Compares local files with remote versions
* Identifies new and modified files
* Prompts for confirmation before pushing changes (unless --force is used)
* Shows a summary of synced files

#### Options

* `--force`: Skip confirmation prompt and push changes immediately

## Examples

```bash
# Initialize a new project
$ sling project init

# Check project status
$ sling project status

# List all jobs
$ sling project jobs list

# Trigger a specific job
$ sling project jobs trigger job_abc123

# Sync local changes
$ sling project sync

# Force sync without confirmation
$ sling project sync --force
```

## Environment Variables

| Variable              | Description                                                                            | Required                    |
| --------------------- | -------------------------------------------------------------------------------------- | --------------------------- |
| `SLING_PROJECT_TOKEN` | Your project token, obtained from the **Settings** section of the Sling Platform       | Yes                         |
| `SLING_PLATFORM_HOST` | The URL of your self-hosted Sling Platform server (e.g. `https://sling.mycompany.com`) | Only for self-hosted setups |

### Cloud-Hosted (Default)

For Sling's cloud platform, you only need to set the project token:

```bash
export SLING_PROJECT_TOKEN="your_project_token_here"
```

### Self-Hosted Platform

If you are running a self-hosted Sling Platform, you must also set `SLING_PLATFORM_HOST` so the CLI knows where to validate your project token and communicate with the platform API:

```bash
export SLING_PLATFORM_HOST="https://sling.mycompany.com"
export SLING_PROJECT_TOKEN="your_project_token_here"
```

When `SLING_PLATFORM_HOST` is set, all project commands (init, status, jobs, sync) will communicate with your self-hosted server instead of the Sling cloud API.

## Notes

* The project token must be valid and have appropriate permissions
* File syncing only pushes changes from local to remote (not from remote to local)
* Jobs can be managed through both CLI and the Sling Platform interface
* Project paths are relative to the project root directory


# Monitors

Monitor your data for schema changes, freshness, anomalies, and column-level statistics

Monitors in Sling observe your data assets without moving data. They track schema drift, data freshness, row counts, column-level statistics, and anomalies — giving you continuous visibility into the health and quality of your database tables and views.

![monitor-charts](/files/0ueIG4gWEufUMqQtvPTc)

## What Can Monitors Do?

* **Schema change detection** — Track when tables, columns, or types are added, dropped, or altered
* **Data freshness** — Alert when data falls behind its expected update schedule
* **Row count tracking** — Monitor table sizes and detect unexpected growth or shrinkage
* **Column statistics** — Collect min/max/mean, percentiles, null counts, cardinality, and more
* **Value validation** — Enforce regex patterns, accepted values, and rejected values on columns
* **Anomaly detection** — Automatically detect spikes, drops, and pattern changes using z-score analysis

## Quick Start

Create a YAML file that specifies which connection to monitor and what to track:

```yaml
connection: MY_POSTGRES

defaults:
  metadata: true
  row_count: true

objects:
  # monitor all objects in schemas
  marketing.*:
  salesforce.*:

  public.users:
    freshness_column: updated_at
    freshness_threshold: "24h"
    columns:
      email:
        count_distinct: true

  public.orders:
    freshness_column: created_at
    freshness_threshold: "12h"
```

On the Sling Platform, save this file in the `monitors/` directory of your project and schedule it as a job.

{% hint style="success" %}
Monitors are a [Sling Platform](/sling-platform/platform) feature available on the **Advanced plan**. See [Pricing](https://slingdata.io/platform/) for details.
{% endhint %}

## Configuration Overview

A monitor configuration has five top-level keys:

| Key          | Required | Description                                                                                                                      |
| ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `connection` | Yes      | The connection to monitor (must match a connection in `env.yaml`)                                                                |
| `defaults`   | No       | Default settings applied to all objects. See [Structure](/sling-platform/platform/monitors/structure).                           |
| `objects`    | Yes      | Map of object names or patterns to their monitoring configuration. See [Structure](/sling-platform/platform/monitors/structure). |
| `schemata`   | No       | Schema change detection settings. See [Schema Changes](/sling-platform/platform/monitors/schema-changes).                        |

For the complete YAML reference, see [Structure](/sling-platform/platform/monitors/structure).

## Running Monitors

Monitor files are stored in the `monitors/` directory of your project on the Sling Platform. Once created, you can schedule them as jobs with a cron expression for automated execution. Monitor runs process objects in parallel (default 2 threads) and retry transient errors up to 3 times.

## Supported Connections

### Databases

| Database        | Table Size | Percentiles  | Freshness |
| --------------- | ---------- | ------------ | --------- |
| PostgreSQL      | Yes        | Yes          | Yes       |
| MySQL / MariaDB | Yes        | —            | Yes       |
| Snowflake       | Yes        | Yes          | Yes       |
| BigQuery        | Yes        | Yes (approx) | Yes       |
| ClickHouse      | Yes        | Yes          | Yes       |
| DuckDB          | Yes        | Yes          | Yes       |
| SQL Server      | Yes        | Yes          | Yes       |
| Oracle          | Yes        | Yes          | Yes       |
| Redshift        | Yes        | Yes          | Yes       |
| Databricks      | Yes        | Yes          | Yes       |
| Trino / Athena  | Yes        | Yes          | Yes       |
| SQLite / D1     | —          | —            | Yes       |

## Notifications

{% hint style="info" %}
Notification settings are a Sling Platform feature, configured per monitor job.
{% endhint %}

![monitor-notification](/files/bvpPYxFM9JGp1F57TzO9)

Each monitor job supports four notification triggers:

| Setting              | Fires When                                                   |
| -------------------- | ------------------------------------------------------------ |
| **On Schema Change** | Tables, columns, or types are added, dropped, or altered     |
| **On Freshness**     | Data exceeds its freshness threshold                         |
| **On Anomaly**       | Metric values deviate significantly from historical baseline |
| **On Failure**       | Monitor execution fails                                      |

Notifications can be sent via **email**, **Slack**, **Discord**, or **Microsoft Teams**.

## Learn More

Detailed documentation for each aspect of monitors:

* [Structure](/sling-platform/platform/monitors/structure) — Complete YAML reference, wildcards, defaults, and definition order
* [Object Metrics & Freshness](/sling-platform/platform/monitors/object-metrics) — Object-level metrics and data staleness detection
* [Column Metrics & Validation](/sling-platform/platform/monitors/column-metrics) — Column statistics and value validation rules
* [Anomaly Detection](/sling-platform/platform/monitors/anomaly-detection) — Z-score analysis, configuration, and event types
* [Schema Changes](/sling-platform/platform/monitors/schema-changes) — Schema drift detection and change events


# Structure

Complete YAML structure reference for Sling monitor configuration files

Below is the structure of the monitor configuration file.

## Root Level

At the root level, a monitor configuration accepts the following keys:

```yaml
# 'connection' and 'objects' keys are required
connection: <connection name>

defaults: <monitor object config>

objects:
  <object name or pattern>: <monitor object config>

schemata:
  enabled: true | false
  exclude: [<glob patterns>]
```

## Object Configuration

The `<object name or pattern>` identifies the table or view to monitor. This can be a fully qualified name (e.g., `public.users`), or a wildcard pattern using `*` and `?` (e.g., `public.*`, `analytics.fact_*`).

The `<monitor object config>` accepts the following keys:

```yaml
# Object-level metrics
disabled: true | false
metadata: true | false
row_count: true | false
body_md5: true | false

# Freshness
freshness_threshold: <duration string>
freshness_column: <column name>

# Anomaly detection tuning
anomaly_detection:
  z_score_threshold: <float>
  min_history_points: <int>
  min_history_days: <int>
  history_days: <int>

# Alert triggers
alert_on_change:
  - name | type | timestamp | size | body | count

# Column-level monitoring
columns:
  <column name or "*">: <monitor column config>
```

## Column Configuration

The `<monitor column config>` accepts the following keys:

```yaml
# Statistics
count: true | false
null_count: true | false
count_distinct: true | false
unique_count: true | false
size: true | false
min_max_mean: true | false
min_max_len: true | false
percentile: true | false

# Validation
regex_match:
  - <regex pattern>
regex_not_match:
  - <regex pattern>
accepted_values:
  - <value>
rejected_values:
  - <value>

# Alert triggers
alert_on_change:
  - name | type | timestamp | size | body | count
```

## Schemata Configuration

The `schemata` block controls schema change detection across the monitored connection:

```yaml
schemata:
  enabled: true
  exclude:
    - "temp_schema.*"
    - "*.staging_*"
```

| Key       | Type      | Default       | Description                                               |
| --------- | --------- | ------------- | --------------------------------------------------------- |
| `enabled` | bool      | `false`       | Enable schema change detection                            |
| `exclude` | string\[] | `["*.*_tmp"]` | Glob patterns for objects to exclude from schema tracking |

{% hint style="info" %}
When `exclude` is not set, the default pattern `["*.*_tmp"]` is applied automatically. Set `exclude: []` (empty array) to disable all exclusions.
{% endhint %}

## Configuration Reference

### Object-Level Keys

| Key                   | Type      | Default | Description                                                                |
| --------------------- | --------- | ------- | -------------------------------------------------------------------------- |
| `disabled`            | bool      | `false` | Exclude this object from monitoring                                        |
| `metadata`            | bool      | `false` | Collect schema info (columns, types). Required for schema change detection |
| `row_count`           | bool      | `false` | Count total rows                                                           |
| `body_md5`            | bool      | `false` | Track MD5 hash of view/procedure definitions                               |
| `freshness_threshold` | string    | —       | Maximum data age before staleness alert (e.g., `"24h"`, `"7d"`)            |
| `freshness_column`    | string    | —       | Column to query `MAX()` for data age                                       |
| `anomaly_detection`   | object    | —       | Override anomaly detection parameters                                      |
| `alert_on_change`     | string\[] | —       | Change types that trigger alerts                                           |
| `columns`             | map       | —       | Column-level monitoring configuration                                      |

### Column-Level Keys

| Key               | Type      | Default | Description                                                       |
| ----------------- | --------- | ------- | ----------------------------------------------------------------- |
| `count`           | bool      | `false` | Non-null and null value counts                                    |
| `null_count`      | bool      | `false` | Null value count                                                  |
| `count_distinct`  | bool      | `false` | Unique value count (cardinality)                                  |
| `unique_count`    | bool      | `false` | Unique value count                                                |
| `size`            | bool      | `false` | Total size in bytes                                               |
| `min_max_mean`    | bool      | `false` | Minimum, maximum, and mean for numeric columns                    |
| `min_max_len`     | bool      | `false` | Minimum and maximum string length for text columns                |
| `percentile`      | bool      | `false` | Percentile statistics (p50, p90, p95, p99) and standard deviation |
| `regex_match`     | string\[] | —       | Patterns that values should match                                 |
| `regex_not_match` | string\[] | —       | Patterns that values should NOT match                             |
| `accepted_values` | string\[] | —       | Valid values (anything else is a violation)                       |
| `rejected_values` | string\[] | —       | Values that should not appear                                     |
| `alert_on_change` | string\[] | —       | Change types that trigger alerts                                  |

### Anomaly Detection Keys

| Key                  | Type  | Default | Description                                          |
| -------------------- | ----- | ------- | ---------------------------------------------------- |
| `z_score_threshold`  | float | `3.0`   | Z-score threshold for anomaly detection              |
| `min_history_points` | int   | `7`     | Minimum data points required before detection begins |
| `min_history_days`   | int   | `7`     | Minimum days of history required                     |
| `history_days`       | int   | `30`    | Lookback window in days for baseline calculation     |

## Wildcards & Patterns

Use `*` and `?` wildcards in object names to monitor multiple objects with the same configuration:

* `*` matches any sequence of characters
* `?` matches a single character

```yaml
objects:
  # All tables in public schema
  public.*:
    metadata: true
    row_count: true

  # Fact tables in analytics schema
  analytics.fact_*:
    row_count: true
    freshness_threshold: "6h"

  # Single character wildcard
  staging.tmp_?:
    metadata: true
```

Use `"*"` as a column name in `defaults` to apply column metrics to all columns:

```yaml
defaults:
  columns:
    "*":
      null_count: true
      count_distinct: true
```

### Definition Order

Objects are processed in **definition order**. When a wildcard expands to include a table and a later entry targets that same table, the later entry's configuration wins entirely.

**Exclude specific tables from a wildcard:**

```yaml
objects:
  # Monitor all public tables...
  public.*:
    metadata: true
    row_count: true

  # ...except these
  public.sensitive_table:
    disabled: true
  public.audit_logs:
    disabled: true
```

**Disable all, then re-enable specific tables:**

```yaml
objects:
  public.*:
    disabled: true

  public.users:
    metadata: true
    row_count: true
  public.orders:
    row_count: true
    freshness_threshold: "12h"
```

{% hint style="info" %}
Definition order matters: when a wildcard expands to include a table, and a later entry targets that same table, the later entry's configuration wins entirely.
{% endhint %}

## Defaults Inheritance

Settings defined under `defaults` are applied to all objects. Individual objects can override any default value.

```yaml
connection: MY_POSTGRES

defaults:
  metadata: true
  row_count: true
  freshness_threshold: "24h"
  columns:
    "*":
      null_count: true

objects:
  public.users: {}              # inherits all defaults
  public.orders:
    freshness_threshold: "6h"   # overrides default threshold
    columns:
      total:
        min_max_mean: true      # adds to inherited column config
  public.staging:
    disabled: true              # excluded entirely
```


# Object Metrics & Freshness

Object-level monitoring metrics and data freshness tracking

Object-level metrics collect information about tables and views as a whole — row counts, schema metadata, definition hashes, and data freshness.

![monitor-chart-object](/files/xChSoWT9ET7m1CGtdpbY)

## Object-Level Metrics

| Key         | Type | Description                                                                                                                                            |
| ----------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `metadata`  | bool | Collect schema information — column names, types, and table type. Enables [schema change detection](/sling-platform/platform/monitors/schema-changes). |
| `row_count` | bool | Count the total number of rows via `COUNT(*)`. Also used as a baseline for [anomaly detection](/sling-platform/platform/monitors/anomaly-detection).   |
| `body_md5`  | bool | Track the MD5 hash of view or stored procedure definitions to detect code changes. Not applicable to regular tables.                                   |

```yaml
objects:
  public.users:
    metadata: true
    row_count: true

  public.my_view:
    metadata: true
    body_md5: true
```

## Alert on Change

Use `alert_on_change` to specify which types of changes should trigger alerts:

| Change Type | Description                          |
| ----------- | ------------------------------------ |
| `name`      | Object was renamed                   |
| `type`      | Object type changed                  |
| `timestamp` | Last modified timestamp changed      |
| `size`      | Object size changed                  |
| `body`      | View or procedure definition changed |
| `count`     | Row count changed                    |

```yaml
objects:
  public.users:
    metadata: true
    row_count: true
    alert_on_change:
      - count
      - size
      - timestamp
```

## Data Freshness

Freshness monitoring detects when data becomes stale. Combine `freshness_column` with `freshness_threshold` to track data age and receive alerts when tables fall behind their expected update schedule.

### How It Works

When `freshness_column` is set, Sling queries `MAX(freshness_column)` to determine the timestamp of the most recent data. If the result is older than `freshness_threshold`, a **data\_stale** event is fired.

If only `freshness_threshold` is set without `freshness_column`, Sling falls back to the object's metadata timestamp (last DDL or modification time). This is less precise for tables that receive `INSERT` operations without schema changes.

### Duration Format

Thresholds use duration strings composed of one or more units:

| Unit | Meaning | Example            |
| ---- | ------- | ------------------ |
| `d`  | Days    | `7d` = 7 days      |
| `h`  | Hours   | `24h` = 24 hours   |
| `m`  | Minutes | `30m` = 30 minutes |
| `s`  | Seconds | `60s` = 60 seconds |

Units can be combined: `"1d12h"` = 1 day and 12 hours, `"1h30m"` = 1 hour and 30 minutes.

### Basic Freshness Check

```yaml
objects:
  public.orders:
    freshness_column: created_at
    freshness_threshold: "12h"

  public.users:
    freshness_column: updated_at
    freshness_threshold: "24h"
```

### Freshness in Defaults

Apply a default freshness configuration to all objects, then override per table as needed:

```yaml
defaults:
  freshness_column: updated_at
  freshness_threshold: "24h"

objects:
  public.users: {}               # uses default (24h on updated_at)
  public.orders:
    freshness_column: created_at  # different column
    freshness_threshold: "6h"     # tighter SLA
```

{% hint style="warning" %}
For accurate freshness detection, set `freshness_column` to a timestamp column that is updated when new data arrives. Without it, Sling relies on metadata timestamps which may not reflect actual data changes.
{% endhint %}

## Complete Example

```yaml
connection: MY_POSTGRES

defaults:
  metadata: true
  row_count: true

objects:
  public.users:
    freshness_column: updated_at
    freshness_threshold: "24h"
    alert_on_change:
      - count
      - timestamp

  public.orders:
    freshness_column: created_at
    freshness_threshold: "6h"
    alert_on_change:
      - count

  public.products:
    row_count: true
    body_md5: false

  public.daily_report_view:
    metadata: true
    body_md5: true
    alert_on_change:
      - body
```


# Column Metrics & Validation

Column-level statistics, validation rules, and value monitoring

Column-level monitoring lets you collect detailed statistics and apply validation rules to individual columns. These metrics feed into [anomaly detection](/sling-platform/platform/monitors/anomaly-detection) over time, alerting you when column values deviate from historical patterns.

## Column Statistics

| Key              | Type | Description                                                       |
| ---------------- | ---- | ----------------------------------------------------------------- |
| `count`          | bool | Total non-null and null value counts                              |
| `null_count`     | bool | Number of null values                                             |
| `count_distinct` | bool | Number of unique values (cardinality)                             |
| `unique_count`   | bool | Unique value count                                                |
| `size`           | bool | Total size in bytes of column values                              |
| `min_max_mean`   | bool | Minimum, maximum, and mean values for numeric columns             |
| `min_max_len`    | bool | Minimum and maximum string length for text columns                |
| `percentile`     | bool | Percentile statistics (p50, p90, p95, p99) and standard deviation |

```yaml
objects:
  public.orders:
    columns:
      revenue:
        count: true
        min_max_mean: true
        percentile: true

      customer_id:
        count_distinct: true
        null_count: true

      description:
        min_max_len: true
```

## Column Validation

Validation rules check column values against defined patterns or value lists. Violations are reported as anomaly events.

### Regex Patterns

Use `regex_match` to define patterns that values **should** match, and `regex_not_match` for patterns that values should **not** match:

```yaml
objects:
  public.users:
    columns:
      email:
        regex_match:
          - "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"

      phone:
        regex_not_match:
          - "^000"        # flag placeholder numbers
          - "^555"        # flag test numbers
```

### Value Lists

Use `accepted_values` to define valid values (anything else is a violation), and `rejected_values` to define values that should not appear:

```yaml
objects:
  public.orders:
    columns:
      status:
        accepted_values:
          - pending
          - processing
          - shipped
          - delivered
          - cancelled

      source:
        rejected_values:
          - test
          - unknown
          - "null"
          - ""
```

{% hint style="info" %}
Validation results include match counts, violation counts, and a `valid` boolean. These are tracked over time and can trigger anomaly alerts when violation rates change.
{% endhint %}

## Column Wildcard

Use `"*"` as a column name in `defaults` to apply metrics to all columns across all monitored objects:

```yaml
defaults:
  columns:
    "*":
      null_count: true
      count_distinct: true

objects:
  public.users: {}          # all columns get null_count + count_distinct
  public.orders:
    columns:
      revenue:
        min_max_mean: true   # adds to the inherited wildcard metrics
```

## Complete Example

```yaml
connection: MY_POSTGRES

defaults:
  metadata: true
  row_count: true
  columns:
    "*":
      null_count: true

objects:
  public.users:
    columns:
      email:
        count_distinct: true
        regex_match:
          - "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
      status:
        accepted_values:
          - active
          - inactive
          - pending
      name:
        min_max_len: true

  public.orders:
    freshness_threshold: "6h"
    freshness_column: created_at
    columns:
      total:
        min_max_mean: true
        percentile: true
      quantity:
        min_max_mean: true
      discount_code:
        count_distinct: true
        rejected_values:
          - test
          - "EXPIRED_2023"

  public.products:
    columns:
      price:
        min_max_mean: true
        percentile: true
      category:
        count_distinct: true
        accepted_values:
          - electronics
          - clothing
          - food
          - home
```


# Anomaly Detection

Automatic z-score anomaly detection for data quality monitoring

Sling automatically detects anomalies in your monitored metrics using **z-score analysis** over historical data. No extra configuration is needed — just enable the metrics you care about and Sling will alert you when values deviate significantly from their historical baseline.

## How It Works

After collecting sufficient historical data, Sling calculates a **baseline** (mean and standard deviation) from recent metric values. Each new value is compared against this baseline by computing a z-score:

```
z-score = |current_value - mean| / standard_deviation
```

When the z-score exceeds the configured threshold (default: 3.0), an anomaly event is fired — either a **spike** (value significantly higher than expected) or a **drop** (value significantly lower than expected).

## Requirements

Anomaly detection activates automatically once sufficient history is available:

| Parameter            | Default | Description                                      |
| -------------------- | ------- | ------------------------------------------------ |
| Minimum data points  | 7       | Number of historical measurements needed         |
| Minimum history days | 7       | Days of data collection before detection begins  |
| Lookback window      | 30 days | How far back to look when computing the baseline |

All parameters are configurable via the `anomaly_detection` block.

{% hint style="info" %}
Anomaly detection begins automatically after collecting at least 7 data points over at least 7 days. Until then, metrics are tracked but no anomaly events are fired.
{% endhint %}

## What Gets Checked

**Object-level:** Row count changes are checked automatically when `row_count: true` is set.

**Column-level:** Any column metric you enable is tracked for anomalies over time. This includes `count`, `null_count`, `count_distinct`, `min_max_mean`, `percentile`, `size`, and others.

**Validation rules:** Violations from `accepted_values`, `rejected_values`, `regex_match`, and `regex_not_match` are also reported as anomaly events.

```yaml
objects:
  public.orders:
    row_count: true              # anomaly detection on row count
    columns:
      revenue:
        min_max_mean: true       # detect revenue spikes/drops
        null_count: true         # detect null count changes
      status:
        count_distinct: true     # detect cardinality shifts
```

## Configuration

Use `anomaly_detection` in `defaults` or per object to tune detection sensitivity:

| Key                  | Type  | Default | Description                                         |
| -------------------- | ----- | ------- | --------------------------------------------------- |
| `z_score_threshold`  | float | `3.0`   | Z-score threshold — lower values are more sensitive |
| `min_history_points` | int   | `7`     | Minimum data points before detection activates      |
| `min_history_days`   | int   | `7`     | Minimum days of history required                    |
| `history_days`       | int   | `30`    | Lookback window for baseline calculation            |

Per-object settings override individual fields from defaults; unset fields inherit from defaults.

## Event Types

| Event                        | Description                                 |
| ---------------------------- | ------------------------------------------- |
| `anomaly_spike`              | Value is significantly higher than expected |
| `anomaly_drop`               | Value is significantly lower than expected  |
| `anomaly_pattern_change`     | Overall pattern shift detected              |
| `anomaly_validation_failure` | Validation rule violations detected         |

## Severity Levels

Anomaly severity is determined by the z-score magnitude:

| Severity | Z-Score Range             |
| -------- | ------------------------- |
| Low      | ≥ 2.0                     |
| Medium   | ≥ 3.0 (default threshold) |
| High     | ≥ 4.0                     |
| Critical | ≥ 5.0                     |

## Examples

### Default Detection

Enable metrics and let Sling handle anomaly detection with default settings:

```yaml
connection: MY_POSTGRES

defaults:
  row_count: true

objects:
  public.orders:
    columns:
      revenue:
        min_max_mean: true
      quantity:
        null_count: true
  public.users:
    columns:
      email:
        count_distinct: true
```

### Custom Thresholds

Set a lower z-score threshold for sensitive tables:

```yaml
connection: MY_POSTGRES

defaults:
  row_count: true
  anomaly_detection:
    z_score_threshold: 2.5       # more sensitive globally
    min_history_points: 10       # require more history

objects:
  public.orders:
    anomaly_detection:
      z_score_threshold: 2.0     # even more sensitive for orders
    columns:
      revenue:
        min_max_mean: true

  public.users:
    row_count: true              # inherits defaults (2.5 / 10)
```

### Longer History Window

For data with seasonal patterns, use a longer lookback window:

```yaml
connection: MY_POSTGRES

defaults:
  anomaly_detection:
    history_days: 90             # 3-month window for seasonal data
    min_history_points: 14       # require 2 weeks of data

objects:
  public.monthly_sales:
    row_count: true
    columns:
      total_revenue:
        min_max_mean: true
```

## Notifications

Enable **On Anomaly** in your monitor job's notification settings to receive alerts when anomalies are detected. Notifications can be sent via email, Slack, Discord, or Microsoft Teams.

{% hint style="info" %}
Notification settings are configured per monitor job on the Sling Platform. See the [Platform documentation](/sling-platform/platform) for details on setting up notification channels.
{% endhint %}


# Schema Changes

Schema change detection for monitoring database structural drift

Monitors can track structural changes to your database schemas, alerting you when tables, columns, or types are added, dropped, or altered.

![monitor-notification](/files/bvpPYxFM9JGp1F57TzO9)

## Schema Change Detection

Schema change detection captures the full column-level state of your database on each monitor run and compares it against the previous state. Any differences — new tables, dropped columns, type changes — are recorded as events.

### Configuration

Enable schema change detection with the `schemata` block:

```yaml
connection: MY_POSTGRES

schemata:
  enabled: true

objects:
  "public.*":
    metadata: true
```

| Key       | Type      | Default       | Description                                               |
| --------- | --------- | ------------- | --------------------------------------------------------- |
| `enabled` | bool      | `false`       | Enable schema change detection                            |
| `exclude` | string\[] | `["*.*_tmp"]` | Glob patterns for objects to exclude from schema tracking |

{% hint style="warning" %}
Schema change detection requires `metadata: true` on the monitored objects. Without metadata collection, Sling cannot compare schema states between runs.
{% endhint %}

### Exclude Patterns

Use `exclude` to skip objects that change frequently or are not relevant:

```yaml
schemata:
  enabled: true
  exclude:
    - "temp_schema.*"
    - "*.staging_*"
    - "*.*_backup"
```

{% hint style="info" %}
When `exclude` is not set, the default pattern `["*.*_tmp"]` excludes temporary tables automatically. Set `exclude: []` (empty array) to disable all exclusions.
{% endhint %}

### Detected Events

| Event                 | Description                                 |
| --------------------- | ------------------------------------------- |
| `schema_added`        | A new schema was detected                   |
| `schema_dropped`      | A schema was removed                        |
| `table_added`         | A new table or view was detected            |
| `table_dropped`       | A table or view was removed                 |
| `table_recreated`     | A table was dropped and re-created          |
| `column_added`        | A new column was added to an existing table |
| `column_dropped`      | A column was removed from an existing table |
| `column_type_altered` | A column's data type was changed            |

Each event includes the database name, schema name, object name, and object type. For `column_type_altered` events, both the old and new data types are recorded.

### Example

Monitor all tables in multiple schemas for structural changes:

```yaml
connection: MY_POSTGRES

schemata:
  enabled: true
  exclude:
    - "*.tmp_*"

objects:
  "public.*":
    metadata: true
  "analytics.*":
    metadata: true
  "staging.*":
    metadata: true
```

### Notifications

Enable **On Schema Change** in your monitor job's notification settings to receive alerts when schema drift is detected.


# Replications

Multiple streams in a YAML or JSON file. Best way to scale Sling.

## Overview

Replications are the best way to use sling in a reusable manner. The `defaults` key allows reusing your inputs with the ability to override any of them in a particular stream. Both YAML or JSON files are accepted. When you run a replication, internally, Sling auto-generates many tasks (one per stream) and runs them in order.

See these pages for more details:

* [Structure](/concepts/replication/structure)
* [Modes](/concepts/replication/modes)
* [Source Options](/concepts/replication/source-options)
* [Target Options](/concepts/replication/target-options)
* [Columns & Constraints](/concepts/replication/columns)
* [Transformations](/concepts/replication/transforms)
* [Hooks](/concepts/hooks)

Here is a basic example, where all PostgreSQL tables in the schema `my_schema` will be loaded into Snowflake. The `my_schema.*` notation as the stream name is a feature possible **only in Replications**. Also notice how `defaults.object` uses [runtime variables](/concepts/replication/runtime-variables).

{% code title="replication.yaml" %}

```yaml
source: MY_POSTGRES
target: MY_SNOWFLAKE

# default config options which apply to all streams
defaults:
  mode: full-refresh
  object: new_schema.{stream_schema}_{stream_table}

streams:
  my_schema.*:

env:
  SLING_THREADS: 3
```

{% endcode %}

Another example:

{% code title="replication.yaml" %}

```yaml
source: MY_MYSQL
target: MY_BIGQUERY

defaults:
  mode: incremental
  object: '{target_schema}.{stream_schema}_{stream_table}'
  primary_key: [id]
  
  source_options:
    empty_as_null: false
    
  target_options:
    column_casing: snake

streams:
  finance.accounts:
  finance.users:
    disabled: true
  
  finance.departments:
    object: '{target_schema}.finance_departments_old' # overwrite default object
    source_options:
      empty_as_null: false

  finance."Transactions":
    mode: incremental # overwrite default mode
    primary_key: [other_id]
    update_key: last_updated_at
  
  finance.all_users.custom:
    sql: |
      select col1, col2
      from finance."all_Users"
    object: finance.all_users # need to add 'object' key for custom SQL

env:
  # adds the _sling_loaded_at timestamp column
  SLING_LOADED_AT_COLUMN: true 
  
  # if source is file, adds a _sling_stream_url column with file path / url
  SLING_STREAM_URL_COLUMN: true

  # parallel stream runs
  SLING_THREADS: 3

  # retry failing stream runs
  SLING_RETRIES: 1
```

{% endcode %}

We can use a replication config with: `sling run -r /path/to/replication.yaml`


# Structure

Below is the structure of the replication configuration file.

## Root Level

At the root level, we have the following keys:

```yaml
# 'source', 'target' and 'streams' keys are required
source: <connection name>
target: <connection name>

defaults: <replication stream map>

hooks: <replication level hooks map>

streams:
  <stream name>: <replication stream map>

env:
  <variable name>: <variable value>

```

## Stream Level

The `<stream name>` identifies the stream to replicate. This can be either a source table name, a file path, or a wildcard pattern using `*`. Wildcards allow matching multiple tables within a schema or multiple files within a directory. For example, `my_schema.*` matches all tables in `my_schema`, while `data/*.csv` matches all CSV files in the `data` directory. See [Tags & Wildcards](/concepts/replication/tags-wildcards) for more details.

The `<replication stream map>` is a map object which accepts the following keys:

```yaml
object: <target table or file name>
mode: full-refresh | incremental | truncate | snapshot | backfill
description: <stream description>
disabled: true | false

primary_key: [<array of column names to use as primary key>]
update_key: <column name to use as incremental key>

columns: {<map of column name to data type>}
select: [<array of column names to include or exclude>]
files: [<array of file paths to include or exclude>]
where: <SQL where clause. Also accepts placeholders update_key, incremental_value, and incremental_where_cond>
single: true | false
sql: <source custom SQL query>
transforms: [<array of transforms or map of column name to array of transforms>]
hooks: <stream level hooks map>

source_options: <source options map>
target_options: <target options map>
```

## Hooks

The `<replication level hooks map>` and `<stream level hooks map>` accepts the keys below. See [Hooks](/concepts/hooks) for more details.

```yaml
# replication level, at start and end of replication
start: [<array of hooks>]
end: [<array of hooks>]

# stream level, before and after a stream run
pre: [<array of hooks>]
post: [<array of hooks>]
pre_merge: [<array of hooks>]   # since v1.4.24
post_merge: [<array of hooks>]  # since v1.4.24
```

## Source Options

The `<source options map>` accepts the keys below. See [Source Options](/concepts/replication/source-options) for more details.

```yaml
compression: auto | none | zip | gzip | snappy | zstd
chunk_size: <backfill chunk size>
datetime_format: auto | <ISO 8601 date format>
delimiter: <character to use as flat file delimiter>
encoding: latin1 | latin5 | latin9 | utf8 | utf8_bom | utf16 | windows1250 | windows1252
empty_as_null: true | false
escape: <character to use as flat file quote escape>
flatten: true | false
format: csv | xml | xlsx | json | parquet | avro | sas7bdat | jsonlines | arrow | delta | raw | geojson
header: true | false
jmespath: <JMESPath expression>
jq: <JQ expression>
limit: <integer>
null_if: <null_if expression>
range: <backfill range expression>
sheet: <excel sheet/range expression>
skip_blank_lines: true | false
```

## Target Options

The `<target options map>` accepts the keys below. See [Target Options](/concepts/replication/target-options) for more details.

```yaml
add_new_columns: true | false
adjust_column_type: true | false
batch_limit: <integer>
column_casing: source | target | snake | upper | lower
column_typing: {map of column type generation configuration}
compression: auto | none | gzip | snappy | zstd
datetime_format: auto | <ISO 8601 date format>
delimiter: <character to use as flat file delimiter>
delete_missing: hard | soft
direct_insert: true | false
encoding: latin1 | latin5 | latin9 | utf8 | utf8_bom | utf16 | windows1250 | windows1252
isolation_level: default | read_uncommitted | read_committed | write_committed | repeatable_read | snapshot | serializable | linearizable
file_max_bytes: <integer>
file_max_rows: <integer>
format: csv | xlsx | json | parquet | raw
header: true | false
ignore_existing: true | false
merge_strategy: update_insert | delete_insert | insert | update
table_ddl: <ddl sql query>
table_keys: {map of table key type to array of column names}
table_tmp: <name of table>
use_bulk: true | false
```

## Replication Specification

Here we have the definitions for the accepted keys.

<table data-full-width="false"><thead><tr><th width="328.950030469226">Replication Config Key</th><th>Description</th></tr></thead><tbody><tr><td><code>source</code></td><td>The source database connection (name, conn string or URL).</td></tr><tr><td><code>target</code></td><td>The target database connection (name, conn string or URL).</td></tr><tr><td><code>hooks</code></td><td>The replication level hooks to apply (at start &#x26; end of replication). See <a href="/pages/mXqUY9i8SyvKZkHpTdN8">here</a> for details.</td></tr><tr><td><code>streams.&#x3C;key></code></td><td>The source table (schema.table), local / cloud file path. Use <code>file://</code> for local paths.</td></tr><tr><td><p><code>streams.&#x3C;key>.object</code></p><p>or <code>defaults.object</code></p></td><td>The target table (schema.table) or local / cloud file path. Use <code>file://</code> for local paths.</td></tr><tr><td><p><code>streams.&#x3C;key>.columns</code></p><p>or <code>defaults.columns</code></p></td><td>The columns types map. See <a href="/pages/NstgDMeKK2JX3ydHtU8i">here</a> for details.</td></tr><tr><td><p><code>streams.&#x3C;key>.transforms</code></p><p>or <code>defaults.transforms</code></p></td><td>The transforms to apply. See <a href="/pages/qQxzTipx9JKhQcaiNNi4">here</a> for details.</td></tr><tr><td><p><code>streams.&#x3C;key>.hooks</code></p><p>or <code>defaults.hooks</code></p></td><td>The stream level hooks to apply (pre- &#x26; post-stream run). See <a href="/pages/mXqUY9i8SyvKZkHpTdN8">here</a> for details.</td></tr><tr><td><p><code>streams.&#x3C;key>.mode</code></p><p>or <code>defaults.mode</code></p></td><td>The target load <a href="/pages/YmhsNnZAWVIOnjeuszZx">mode</a> to use: <code>incremental</code>, <code>truncate</code>, <code>full-refresh</code>, <code>backfill</code> or <code>snapshot</code>. Default is <code>full-refresh</code>.</td></tr><tr><td><code>streams.&#x3C;key>.select</code> or <code>defaults.select</code></td><td>Select or exclude specific columns from the source stream. Use <code>-</code> prefix to exclude.</td></tr><tr><td><code>streams.&#x3C;key>.single</code> or <code>defaults.single</code></td><td>When using a wildcard (<code>*</code>) in the stream name, consider as a single stream (don't expand into many streams).</td></tr><tr><td><code>streams.&#x3C;key>.sql</code> or <code>defaults.sql</code></td><td>The custom SQL query to use. Accepts <code>file://path/to.query.sql</code> as well.</td></tr><tr><td><p><code>streams.&#x3C;key>.primary_key</code></p><p>or <code>defaults.primary_key</code></p></td><td>The column(s) to use as primary key. If composite key, use array.</td></tr><tr><td><p><code>streams.&#x3C;key>.update_key</code></p><p>or <code>defaults.update_key</code></p></td><td>The column to use as update key (for <code>incremental</code> mode).</td></tr><tr><td><p><code>streams.&#x3C;key>.source_options</code></p><p>or <code>defaults.source_options</code></p></td><td>Options to further configure source. See <a href="/pages/3BkvtN9AW4GCDlyu6jGw">here</a> for details.</td></tr><tr><td><p><code>streams.&#x3C;key>.target_options</code></p><p>or <code>defaults.target_options</code></p></td><td>Options to further configure target. See <a href="/pages/dhKMJTL71CTD8kxiAgi4">here</a> for details.</td></tr><tr><td><code>env</code></td><td>Environment variables to use for replication. See <a href="/pages/xaHAsi2Yf0s9DhcNxxh2">here</a> for details.</td></tr></tbody></table>


# Modes

Here are the various loading modes available. All modes load into a new temporary table prior to final load.

<table data-full-width="false"><thead><tr><th width="212.60270475141198">Mode</th><th>Description</th></tr></thead><tbody><tr><td><code>full-refresh</code></td><td>This is the default mode. The target table will be dropped and recreated with the source data.</td></tr><tr><td><code>incremental</code></td><td>The source data will be merged or appended into the target table. If the table does not exist, it will be created. See below for more details.</td></tr><tr><td><code>truncate</code></td><td>Similar to <code>full-refresh</code>, except that the target table is truncated instead of dropped. This keeps any special DDL / GRANT applied.</td></tr><tr><td><code>snapshot</code></td><td>Appends the full dataset with an added timestamp column. If the target table exists, Sling will insert into / append data with a <code>_sling_loaded_at</code> column. If it does not, the table will be created.</td></tr><tr><td><code>backfill</code></td><td>Similar to <code>incremental</code>, but takes a <code>range</code> input to backfill a specific <code>update_key</code> range, such as dates or numbers.</td></tr><tr><td><code>definition-only</code></td><td>Creates the target table or file structure without transferring any data. For database targets, the table is created with the inferred schema but 0 rows. For file targets, only Parquet and Arrow formats are supported, creating a file with proper column types but no data rows.</td></tr><tr><td><code>change-capture</code></td><td>Captures row-level changes (inserts, updates, deletes) from the source database's transaction log. Performs an automatic initial snapshot on the first run, then reads incremental changes on subsequent runs. Requires a <code>primary_key</code>. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/replication/change-capture.md">Change Capture (CDC)</a> for details.</td></tr></tbody></table>

### Incremental Mode Strategies

<table data-full-width="false"><thead><tr><th width="215.18644986449863">Load Strategy</th><th width="145" align="center">Primary Key</th><th width="160" align="center">Update Key</th><th>Stream Strategy</th></tr></thead><tbody><tr><td>New Data Upsert (update/insert)</td><td align="center"><code>yes</code></td><td align="center"><code>yes</code></td><td>Only new records after <code>max(update_key)</code></td></tr><tr><td>Full Data Upsert (update/insert)</td><td align="center"><code>yes</code></td><td align="center">no</td><td>Full data</td></tr><tr><td>Append Only (insert, no update)</td><td align="center">no</td><td align="center"><code>yes</code></td><td>Only new records after <code>max(update_key)</code></td></tr></tbody></table>

### Incremental or Backfill Mode With Custom SQL

When using `incremental` or `backfill` mode with a custom SQL stream, Sling provides two placeholder options to handle incremental loading:

* `{incremental_where_cond}`: Injects a complete WHERE condition
* `{incremental_value}`: Injects only the value, allowing for custom condition logic

#### Using {incremental\_where\_cond}

This placeholder injects a complete WHERE condition based on the `update_key`:

```sql
SELECT * FROM my_schema.my_table 
WHERE {incremental_where_cond}
```

Sling will replace the placeholder as follows:

* First run (table doesn't exist): `WHERE 1=1`
* Subsequent runs: `WHERE my_update_key > '[incremental_value]'`

For example, if the last maximum value was '2001-01-01 01:01:01', the query becomes:

```sql
SELECT * FROM my_schema.my_table 
WHERE my_update_key > '2001-01-01 01:01:01'
```

#### Using {incremental\_value}

This placeholder gives you more control over the WHERE condition by injecting only the value:

```sql
SELECT * FROM my_schema.my_table 
WHERE my_int_key > coalesce({incremental_value}, 0)  -- For numeric columns
-- or
WHERE my_timestamp > coalesce({incremental_value}, '2001-01-01')  -- For timestamp columns
```

Sling will replace the placeholder as follows:

* First run (table doesn't exist): `{incremental_value}` becomes `null`
* Subsequent runs: `{incremental_value}` becomes the last maximum value

For example, if the last maximum value was 99, the query becomes:

```sql
SELECT * FROM my_schema.my_table 
WHERE my_int_key > coalesce(99, 0)
```


# Source Options

## Specification

Here we have various keys accepted for source options:

<table data-full-width="false"><thead><tr><th width="226.55394360395405">Key</th><th>Description</th></tr></thead><tbody><tr><td><code>compression</code></td><td><p>(<em>Only for file source</em>)</p><p>The type of compression to use when reading files. Valid inputs are <code>none</code>, <code>auto</code> and <code>gzip</code>, <code>zstd</code>, <code>snappy</code>. Default is <code>auto</code>.</p></td></tr><tr><td><code>chunk_size</code></td><td><p>(<em>Only for database source</em>)</p><p>The chunk size for backfill processing. This tells Sling to split a stream into many. Accepts values such as <code>12h</code>, <code>7d</code> or 1m. See <a href="/pages/rrJhyNHQKXbWhW8Nb0nL#chunked-backfill">here</a> for more details.</p></td></tr><tr><td><code>datetime_format</code></td><td>The <a href="https://www.w3.org/TR/NOTE-datetime">ISO 8601</a> date format to use when reading date values. Default is <code>auto</code></td></tr><tr><td><code>delimiter</code></td><td><p>(<em>Only for file source</em>)</p><p>The delimiter to use when parsing tabular files. Default is <code>auto</code>.</p></td></tr><tr><td><code>encoding</code></td><td><p>(<em>Only for file source</em>)</p><p>The text encoding to use when reading files. This is essential for correctly reading files that contain special characters or were created with non-UTF-8 encodings. Options are: <code>latin1</code>, <code>latin5</code>, <code>latin9</code>, <code>utf8</code>, <code>utf8_bom</code>, <code>utf16</code>, <code>windows1250</code>, <code>windows1252</code>. Default is <code>utf8</code>.</p></td></tr><tr><td><code>escape</code></td><td><p>(<em>Only for file source - since v1.2.4</em>)</p><p>The escape character to use when parsing tabular files. Default is <code>"</code></p></td></tr><tr><td><code>empty_as_null</code></td><td>Whether empty fields should be treated as <code>NULL</code>. Default is <code>false</code> starting in <em>v1.4.5</em>. Prior, default depends on the kind of the source connection: <code>false</code> for <a href="https://docs.slingdata.io/connections/database-connections">database connections</a>, <code>true</code> for <a href="https://docs.slingdata.io/connections/file-connections">storage connections</a>.</td></tr><tr><td><code>flatten</code></td><td>Whether to flatten a semi-structure source format (JSON, XML). Accepts <code>true</code> or <code>false</code> boolean values. Since <em>v1.4.5</em>, also accepts an integer, representing the maximum flattening depth. <code>0</code> means infinite depth.</td></tr><tr><td><code>format</code></td><td><p>(<em>Only for file source</em>)</p><p>The format of the file(s). Options are: <code>csv</code>, <code>parquet</code>, <code>xlsx</code>, <code>avro</code>, <code>json</code>, <code>jsonlines</code>, <code>sas7bdat</code> and <code>xml</code>. The <code>xlsx</code> format also reads macro-enabled <code>.xlsm</code> Excel files.</p></td></tr><tr><td><code>header</code></td><td>(<em>Only for file source</em>) Whether to consider the first line as header. Default is <code>true</code>.</td></tr><tr><td><code>jmespath</code></td><td><p>(<em>Only for file and NoSQL database source</em>)</p><p>Specify a JMESPath expression to use to filter / extract nested JSON data. See <a href="https://jmespath.org/">https://jmespath.org/</a> for more</p></td></tr><tr><td><code>limit</code></td><td>The maximum number of rows to pull from the source</td></tr><tr><td><code>null_if</code></td><td>Whether this case-sensitive string value should be treated as a database <em>null</em> value when encountered. Default is <code>NULL</code>.</td></tr><tr><td><code>sheet</code></td><td>(<em>Only for Excel source files</em>) The name of the sheet to use as a data source, for example <code>Sheet1</code>. Default is the first sheet. You can also specify the range (<code>Sheet2!B:H</code>, <code>Sheet3!B1:H70</code>).</td></tr><tr><td><code>range</code></td><td>The range to use for <code>backfill</code> mode, separated by a single comma. Example: <code>2021-01-01,2021-02-01</code> or <code>1,10000</code></td></tr><tr><td><code>skip_blank_lines</code></td><td>Whether blank lines should be skipped when encountered. Default is <code>false</code>.</td></tr></tbody></table>

## Encoding Options

When working with files that contain special characters (accented letters, non-English text, etc.), it's crucial to specify the correct encoding to ensure data integrity. Without the proper encoding, special characters may appear garbled or corrupted.

### Supported Encodings

* **`latin1` (ISO-8859-1)**: Western European languages
* **`latin5` (ISO-8859-5)**: Cyrillic alphabet (Russian, Bulgarian, etc.)
* **`latin9` (ISO-8859-15)**: Western European with Euro symbol
* **`utf8`**: Unicode UTF-8 (default, most common)
* **`utf8_bom`**: UTF-8 with Byte Order Mark
* **`utf16`**: Unicode UTF-16
* **`windows1250`**: Central European languages (Windows)
* **`windows1252`**: Western European languages (Windows)

### Examples

```yaml
# Reading a Latin-1 encoded CSV file with French characters
streams:
  "file://data/customers_french.csv":
    object: public.customers
    source_options:
      encoding: latin1
      header: true

# Reading a Windows-1252 encoded file with special quotes and em-dashes
streams:
  "file://data/documents_windows.csv":
    object: public.documents
    source_options:
      encoding: windows1252
      header: true

# Reading a UTF-8 file with BOM (Byte Order Mark)
streams:
  "file://data/international.csv":
    object: public.international_data
    source_options:
      encoding: utf8_bom
      header: true
```

### CLI Usage

```bash
# Specify encoding when reading files with special characters
sling run \
  --src-stream "file://./data/latin1_file.csv" \
  --tgt-conn POSTGRES \
  --tgt-object public.my_table \
  --src-options '{"encoding": "latin1", "header": true}'
```

**Important**: If you don't specify the correct encoding, files with special characters may appear corrupted or cause processing errors. Always verify the encoding of your source files, especially when working with data from different regions or legacy systems.


# Target Options

## Specification

Here we have various keys accepted for target options:

<table data-full-width="false"><thead><tr><th width="230.92613233622984">Key</th><th>Description</th></tr></thead><tbody><tr><td><code>add_new_columns</code></td><td><p>(<em>Only for database target</em>)</p><p>Whether to add new columns from stream not found in target table (when mode is not <code>full-refresh</code>). Default is <code>true</code>.</p></td></tr><tr><td><code>adjust_column_type</code></td><td><p>(<em>Only for database target</em>)</p><p>Whether to adjust the column type when needed. Default is <code>false</code>.</p></td></tr><tr><td><code>batch_limit</code></td><td><p>(<em>Only for database target</em> — <em>since v1.2.11</em>)</p><p>The maximum number of records per transaction batch.</p></td></tr><tr><td><code>column_casing</code></td><td>Whether to convert the column name casing. This facilitates querying tables in target databases without using quotes. Accepts <code>normalize</code> (normalizes to target casing, unless column has varied casings, since <em>v1.4.5</em>), <code>source</code> (keep original casing), <code>target</code> (converts casing according to target database), <code>snake</code> (converts snake casing according to target database), <code>upper</code> or <code>lower</code>. Default is <code>normalize</code> (before <em>v1.4.5</em>, default is <code>source</code>). See <a href="/pages/NstgDMeKK2JX3ydHtU8i#column-casing">here</a> for details.</td></tr><tr><td><code>column_typing</code></td><td>This allows precise control over how column types are generated, for example, manipulating string column lengths when transferring between databases with different character encoding requirements. See <a href="/pages/NstgDMeKK2JX3ydHtU8i#column-typing">here</a> for details.</td></tr><tr><td><code>compression</code></td><td><p>(<em>Only for file target</em>)</p><p>The type of compression to use when writing files. Valid inputs are <code>none</code>, <code>auto</code> and <code>gzip</code>, <code>zstd</code>, <code>snappy</code>. Default is <code>auto</code>.</p></td></tr><tr><td><code>datetime_format</code></td><td><p>(<em>Only for file target</em>)</p><p>The <a href="https://www.w3.org/TR/NOTE-datetime">ISO 8601</a> date format to use when writing date values. Default is <code>auto</code></p></td></tr><tr><td><code>delete_missing</code></td><td><p>(<em>Only for database target</em>)</p><p>Whether to soft/hard delete missing primary keys records in the target tables. Accepts <code>soft</code> or <code>hard</code> values. See <a href="/pages/QyAQvN7igrxHXCNmtiF5#delete-missing-records-soft--hard">here</a> for more details</p></td></tr><tr><td><code>direct_insert</code></td><td><p>(<em>Only for database target</em>)</p><p>Whether to bypass temporary table and insert directly into the target table. This can improve performance but has limitations: incremental/backfill modes with primary keys are not supported as they require merge/upsert operations. Default is <code>false</code>.</p></td></tr><tr><td><code>encoding</code></td><td><p>(<em>Only for file target</em>)</p><p>The text encoding to use when writing files. Allows writing files in different encodings for compatibility with legacy systems or international data requirements. Options include: <code>latin1</code>, <code>latin5</code>, <code>latin9</code>, <code>utf8</code>, <code>utf8_bom</code>, <code>utf16</code>, <code>windows1250</code>, <code>windows1252</code>. Default is <code>utf8</code>.</p></td></tr><tr><td><code>isolation_level</code></td><td><p>(<em>Only for database target</em>)</p><p>What transaction isolation level to set when insert/merging into final table. Accepts: <code>default</code>, <code>read_uncommitted</code>, <code>read_committed</code>, <code>write_committed</code>, <code>repeatable_read</code>, <code>snapshot</code>, <code>serializable</code>, <code>linearizable</code></p></td></tr><tr><td><code>merge_strategy</code></td><td><p>(<em>Only for database target</em> — <em>since v1.5.5</em>)</p><p>The strategy to use when merging data in <code>incremental</code> or <code>backfill</code> mode with a primary key. Accepts <code>update_insert</code> (upsert), <code>delete_insert</code>, <code>insert</code> (append-only), or <code>update</code> (update-only). Default depends on database. See <a href="/pages/JPRkvRH5g93quIQD7HqQ">Merge Strategy</a> for details.</p></td></tr><tr><td><code>delimiter</code></td><td><p>(<em>Only for file target</em>)</p><p>The delimiter to use when writing tabular files. Default is <code>,</code>.</p></td></tr><tr><td><code>file_max_bytes</code></td><td><p>(<em>For file target or temp files for DB target</em>)</p><p>The maximum number of bytes to write to a file. <code>0</code> means infinite number of bytes. When a value greater than <code>0</code> is specified, the output location will be a folder with many parts in it. Default is <code>50000000</code>. Does not work with <code>parquet</code> file format (use <code>file_max_rows</code> instead).</p></td></tr><tr><td><code>file_max_rows</code></td><td><p>(<em>For file target or temp files for DB target</em>)</p><p>The maximum number of rows (usually lines) to write to a file. <code>0</code> means infinite number of rows. When a value greater than <code>0</code> is specified, the output location will be a folder with many parts in it. Default is <code>500000</code>.</p></td></tr><tr><td><code>format</code></td><td><p>(<em>Only for file target</em>)</p><p>The format of the file(s). Options are: <code>csv</code>, <code>parquet</code>, <code>xlsx</code>, <code>json</code> and <code>jsonlines</code>.</p></td></tr><tr><td><code>ignore_existing</code></td><td>Ignore existing target file/table if it exists (do not overwrite/modify). Default is <code>false</code>.</td></tr><tr><td><code>header</code></td><td><p>(<em>Only for file target</em>)</p><p>Whether to write the first line as header. Default is <code>true</code>.</p></td></tr><tr><td><code>table_ddl</code></td><td><p>(<em>Only for database target</em>)</p><p>The table DDL to use when writing to a database. Default is auto-generated by Sling. Accepts the <code>{col_types}</code> runtime variable. See <a href="#table-ddl">here</a> for more details</p></td></tr><tr><td><code>table_keys</code></td><td><p>(<em>Only for database target</em>)</p><p>The table keys to define when creating a table, such as <code>partition</code>, <code>cluster</code>, <code>sort</code>, etc. Each entry defines an array of column names, or expression. See <a href="#table-keys">here</a> for more details.</p></td></tr><tr><td><code>table_tmp</code></td><td><p>(<em>Only for database target</em>)</p><p>The temporary table name that should be used when loading into a database. Default is auto-generated by Sling.</p></td></tr><tr><td><code>use_bulk</code></td><td><p>(<em>Only for database target</em>)</p><p>Whether to use external bulk loading tools, if installed and available in the <code>PATH</code> environment variable. Sling looks to use <a href="https://docs.microsoft.com/en-us/sql/tools/bcp-utility">bcp</a> for SQL Server, <a href="https://blogs.oracle.com/opal/oracle-instant-client-122-now-has-sqlloader-and-data-pump">sqlldr</a> for Oracle and <a href="https://www.google.com/search?q=mysql-client-core-5.7">mysql</a> for MySQL. If <code>false</code>, traditional batch INSERT loading will be used. Default is <code>true</code>.</p></td></tr></tbody></table>

## Encoding Options

When writing files for different systems or regions, you may need to specify a particular text encoding. This is essential for maintaining data integrity when working with international characters or legacy systems that expect specific encodings.

### Supported Encodings

* **`latin1` (ISO-8859-1)**: Western European languages
* **`latin5` (ISO-8859-5)**: Cyrillic alphabet (Russian, Bulgarian, etc.)
* **`latin9` (ISO-8859-15)**: Western European with Euro symbol
* **`utf8`**: Unicode UTF-8 (default, most common)
* **`utf8_bom`**: UTF-8 with Byte Order Mark
* **`utf16`**: Unicode UTF-16
* **`windows1250`**: Central European languages (Windows)
* **`windows1252`**: Western European languages (Windows)

### Examples

```yaml
# Writing data to a Latin-1 encoded CSV file for a legacy French system
streams:
  public.customers:
    object: "file://output/customers_french.csv"
    target_options:
      format: csv
      encoding: latin1
      header: true

# Writing data to a Windows-1252 encoded file for compatibility with older Windows systems
streams:
  public.documents:
    object: "file://output/documents_legacy.csv"
    target_options:
      format: csv
      encoding: windows1252
      header: true

# Writing data to a UTF-8 file with BOM for systems that require it
streams:
  public.international_data:
    object: "file://output/international.csv"
    target_options:
      format: csv
      encoding: utf8_bom
      header: true
```

### CLI Usage

```bash
# Write data to a Latin-1 encoded CSV file
sling run \
  --src-conn POSTGRES \
  --src-stream public.customers \
  --tgt-object "file://./output/customers_latin1.csv" \
  --tgt-options '{"format": "csv", "encoding": "latin1", "header": true}'

# Write data to a Windows-1252 encoded file for legacy compatibility
sling run \
  --src-conn MYSQL \
  --src-stream inventory.products \
  --tgt-object "file://./exports/products_windows.csv" \
  --tgt-options '{"format": "csv", "encoding": "windows1252", "header": true}'
```

**Important**: Choose the appropriate encoding based on your target system's requirements. Using the wrong encoding may cause special characters to appear corrupted or unreadable. UTF-8 is recommended for most modern systems and international data.

## Table DDL

The `table_ddl` option allows you to define the table DDL to use when creating a table. This is useful when you want to define a table with specific properties. Sling also allows the injection of runtime variables, such as `{object_name}`, which will be replaced with the actual stream table name. See [Runtime Variables](/concepts/replication/runtime-variables) for more information. The special `{col_types}` variable will be replaced with the actual column types of the stream table.

Here is an example of using the `table_ddl` option:

```yaml
source: my_source_db
target: clickhouse

# apply to all streams
defaults:
  target_options:
    table_ddl: create table {object_name} ({col_types}) engine=MergeTree

streams:
  # use default target_options
  my_schema.my_table:

  # override default for this stream
  my_schema.another_table:
    target_options:
      table_ddl: create table {object_name} ({col_types}) engine=AggregatingMergeTree
```

## Table Keys

Table keys are used to define the keys of the target table. They are useful to preset keys such as indexes, primary keys, etc.

Here are the accepted keys:

* `cluster` (used in BigQuery and Snowflake)
* `index` (used in PostgreSQL, MySQL, MariaDB, Oracle, SQL Server, SQLite, DuckDB and MotherDuck)
* `partition` (used in PostgreSQL, BigQuery, ClickHouse)
* `primary` (used in PostgreSQL, MySQL, Oracle, SQL Server, SQLite)
* `sort` (used in Redshift)
* `unique` (used in PostgreSQL, MySQL, Oracle, SQL Server, SQLite and DuckDB)
* `aggregate`, `duplicate`, `distribution` and `hash` (used in StarRocks)

Here is an example of using the `table_keys` option:

```yaml
source: my_source_db
target: my_target_db

# apply to all streams
defaults:
  target_options:
    table_keys:
      primary: [id]

streams:
  # use default target_options
  my_schema.my_table:

  # override default for this stream
  my_schema.another_table:
    target_options:
      table_keys:
        index: [col1, col2] # creates two separate indexes: one on col1, one on col2
```

{% hint style="warning" %}
A flat list under `index` creates **one index per entry**, not a single composite index. The example above (`index: [col1, col2]`) produces **two** single-column indexes. To create a single composite index over multiple columns, nest the columns in a list (e.g. `index: [[col1, col2]]`). See [Indexes](#indexes) below for all supported shapes.
{% endhint %}

### Indexes

The `index` key is a list where **each entry describes one index**. A bare column name (or a flat list of bare column names) therefore creates one single-column index per name — `index: [col1, col2]` creates two separate indexes. To create a composite index over multiple columns, the entry itself must be a list (i.e. nest it): `index: [[col1, col2]]`.

From *v1.5.20+* each entry can take any of these shapes:

```yaml
target_options:
  table_keys:
    index:
      # a single column -> one index on that column
      - search_col

      # an expression column -> one index on that column
      - LOWER(email)

      # a list -> one composite index over those columns
      - [org_id, project_id]

      # a custom named index
      - idx_lookup: [org_id, project_id]
```

### Primary & Unique Keys

The `primary` and `unique` keys each define **one** key over the listed columns (a composite key when more than one column is given). Unlike `index`, a flat list is *not* split into multiple keys — `primary: [user_id, brand_id]` creates a single composite primary key over both columns.

From *v1.5.21+* the nested (wrapped-list) form is also accepted and is treated identically to the flat form, so you can use the same `[[...]]` shape as `index` for consistency:

```yaml
target_options:
  table_keys:
    # both of these create a single composite PRIMARY KEY (user_id, brand_id)
    primary: [user_id, brand_id]
    # primary: [[user_id, brand_id]]

    # both of these create a single composite UNIQUE key (org_id, project_id)
    unique: [org_id, project_id]
    # unique: [[org_id, project_id]]
```


# Columns

## Selecting Columns

Use the `select` option to narrow a stream down to the columns you actually need. Replication is read-once, write-once, so dropping unused columns shrinks the bytes transferred, the storage footprint at the target, and the time spent inferring/casting types.

From *v1.5.19+*,`select` works on every source: databases (where it becomes `SELECT col1, col2, ...`), file sources (CSV, JSON, JSONL, Parquet), and API sources. Three things make up the grammar:

* **Exact names** to include (`id`, `email`)
* **Glob patterns** to include or exclude (`user_*`, `*_at`, `-internal_*`)
* **Renames** with the `as` keyword (`id as user_id`)

### Using CLI Flags

```bash
# include specific columns
sling run --select 'id,email,created_at'

# include everything except a few (minus prefix is the exclude marker)
sling run --select '-password,-ssn'

# include with a glob; here, only columns named like `event_*`
sling run --select 'event_*'

# rename inline
sling run --select 'id as user_id,name as full_name'
```

### Using YAML

```yaml
source: source_name
target: target_name

defaults:
  # narrow every stream to these columns
  select: [id, email, created_at]

streams:
  my_stream:
    # this stream: drop a few columns, keep the rest
    select: [-password, -ssn]

  my_other_stream:
    # this stream: include + rename
    select: [id, name as full_name, email]
```

### Picking a Subset

The most common use of `select` is to whittle a wide source table down to the columns the target actually needs. Two shapes show up most often:

**1. Include exactly what you want.** Everything not listed is dropped.

```yaml
streams:
  public.users:
    object: public.users_slim
    select: [id, email, created_at]
```

**2. Include everything except a few.** Use `-` prefixes. When every item is an exclusion, Sling treats it as "select all *except* these."

```yaml
streams:
  public.users:
    object: public.users_no_pii
    select: [-password, -ssn, -date_of_birth]
```

You cannot mix exact include names with `-` exclusions in the same list — pick one shape per stream.

### Globs

Globs let one pattern match many columns. They work in both include lists and exclude lists.

```yaml
streams:
  # include: keep only the audit timestamps
  public.events:
    object: public.events_audit
    select: ['*_at']            # created_at, updated_at, deleted_at, ...

  # include: keep all the user_* columns
  public.profile_join:
    select: ['user_*']           # user_id, user_name, user_role, ...

  # exclude: drop everything starting with `internal_`
  public.audit_log:
    object: public.audit_clean
    select: ['-internal_*']

  # exclude: drop trailing-`_internal` columns plus one specific field
  public.audit_log_v2:
    object: public.audit_v2_clean
    select: ['-*_internal', '-debug_info']
```

Supported patterns: prefix (`prefix_*`), suffix (`*_suffix`), contains (`*middle*`), and exact names. Matching is case-insensitive.

### Renaming

The `as` keyword renames columns on the way out. For database sources this generates `SELECT column AS alias` directly; for file and API sources Sling rewrites the column header after read. Renames sit alongside includes — list anything you don't rename as-is. Requires *v1.5.5+*.

```yaml
streams:
  public.users:
    object: public.customers
    select:
      - 'id as user_id'
      - 'first_name as name'
      - 'email as contact_email'
```

When using custom SQL with the `{fields}` placeholder, the renamed columns are substituted automatically:

```yaml
streams:
  my_stream:
    sql: SELECT {fields} FROM public.users WHERE active = true
    object: public.active_customers
    select:
      - 'id'
      - 'first_name as customer_name'
      - 'score as customer_score'
```

### Controlling Column Order

The order of items in the `select` list is the order columns appear in the target. This matters when you're writing to a file (CSV, Parquet, JSON) where readers care about column order, or when you want a stable layout regardless of how the source happens to list its columns.

You can pin specific columns to the front, let the rest follow in source order via `*`, and pin others to the back:

```yaml
streams:
  public.users:
    object: public.users_reordered
    select:
      - id              # front pin
      - email           # front pin
      - '*'             # everything else, in source order
      - created_at      # back pin
      - updated_at      # back pin
```

`*` expands to whatever's left after the pins are accounted for, in the order the source declared the columns. Globs can also be used as pins (`'user_*'` between two exact pins drops every matching column there in source order).

If you don't need reordering, just leave `*` out — Sling preserves source order by default.

### Reusing `columns` for Order

When you've already listed your columns under [`columns`](#casting-columns) to pin their types, you don't have to repeat every name under `select` just to control order. From *v1.5.20+*, the `@columns` token expands to the names declared in `columns`, **in that declared order**.

`@columns` must be the **first** item in the `select` list. Use it alone to emit exactly the declared columns, or follow it with `*` to pin the declared columns first and let the rest follow.

```yaml
streams:
  res_partner:
    columns:
      id: integer
      name: string
      email: string
      # ... more type-pinned columns, in your preferred order ...

    # emit exactly the columns above, in the order they're declared
    select: ['@columns']
```

```yaml
streams:
  res_partner:
    columns:
      id: integer
      name: string
      email: string

    # pin the declared columns first (in order), then everything else
    select: ['@columns', '*']
```

Notes:

* `@columns` is only honored as the first item — using it anywhere else is an error.
* It requires a `columns` block on the stream (after [merging with defaults](#merging-with-defaults)); using it with no columns defined is an error.
* Names that would be duplicated by the expansion are kept once (first occurrence wins), so `['@columns', 'id']` won't list `id` twice.

## Casting Columns

When running a replication, you can specify the column types to tell Sling to cast the data to the correct type. It is not necessary to include all columns. Sling will automatically detect the types for any unspecified columns. See [here](https://github.com/slingdata-io/sling-cli/blob/main/core/dbio/templates/types_general_to_native.tsv) and [here](https://github.com/slingdata-io/sling-cli/blob/main/core/dbio/templates/types_native_to_general.tsv) for various type mappings between native and generic types for all types of databases.

Acceptable data types are:

* `bigint`
* `bool`
* `datetime`
* `decimal` or `decimal(precision, scale)` (such as `decimal(10, 2)`)
* `integer`
* `json`
* `string` or `string(length)` (such as `string(100)`)
* `text` or `text(length)` (such as `text(3000)`)
* `geometry`

Sling allows you the ability to apply constraints, such as `value > 0`. See [Constraints](/concepts/data-quality/constraints) for details.

### Column Modifiers

From *v1.5.20+*, the type slot accepts space-separated **modifiers** after the data type. These shape the DDL Sling generates when it creates the target table — adding `NOT NULL`, primary keys, unique constraints, column descriptions, and indexes — without having to hand-write `table_ddl`. They apply on a fresh `CREATE TABLE` regardless of any schema-migration setting.

```yaml
streams:
  public.users:
    object: public.users_dim
    columns:
      id:      bigint not_null primary_key
      email:   text not_null unique
      region:  text index
      slug:    text unique_index
      notes:   text description('free-form user notes')
```

The first token is always the type; everything after it is a modifier. The supported modifiers are:

| Modifier                             | Effect                                                                                                                                                                                        |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `not_null`                           | Column is `NOT NULL`.                                                                                                                                                                         |
| `nullable`                           | Column is explicitly nullable (the default; useful to override a source-derived `not_null`).                                                                                                  |
| `primary_key`                        | Column is part of the table's `PRIMARY KEY` (DDL only — does not set the incremental-mode primary key; use [`primary_key`](/concepts/replication/structure) / `table_keys.primary` for that). |
| `unique`                             | Adds a `UNIQUE` constraint on the column.                                                                                                                                                     |
| `index`                              | Creates a plain index on the column.                                                                                                                                                          |
| `index(...)`                         | Creates a plain index with options (see kwargs below).                                                                                                                                        |
| `unique_index` / `unique_index(...)` | Creates a unique index on the column.                                                                                                                                                         |
| `description('<text>')`              | Sets the column comment/description. A user-supplied description wins over one inferred from the source.                                                                                      |

The `index(...)` / `unique_index(...)` forms accept keyword arguments to compose multi-column or partial indexes across several columns. Give the same `name` to two columns to build a composite index, and use `priority` to order the members:

```yaml
streams:
  public.events:
    object: public.events_fact
    columns:
      org_id:     bigint not_null index(name=idx_org_created, priority=1)
      created_at: datetime index(name=idx_org_created, priority=2, sort=desc)
      status:     text index(name=idx_active, where='deleted_at IS NULL')
```

Supported kwargs: `name`, `priority`, `sort` (`asc`/`desc`), `where`, `type`, `include`. See also [`table_keys.index`](/concepts/replication/target-options#table-keys) options for defining composite indexes.

{% hint style="info" %}
Whether each modifier renders depends on the target engine. For example, ClickHouse honors `not_null` (columns are *not* wrapped in `Nullable(...)`) and renders indexes inline, while engines without secondary indexes (Snowflake, Redshift) silently ignore `index`. Index and description statements are only applied to the final table, never the transient staging table.
{% endhint %}

The runtime [constraint](/concepts/data-quality/constraints) slot (after `|`) is unaffected and can be combined with modifiers:

```yaml
columns:
  amount: integer not_null | value >= 0
```

### Using CLI Flags

```bash
# template
sling run --columns '{ "<column_name>": "<data_type>" }'

# template with constraint
sling run --columns '{ "<column_name>": "<data_type> | <constraint>" }'

# cast column to bigint
sling run --columns '{ "my_column": "bigint" }'

# cast column to decimal(10, 2)
sling run --columns '{ "my_column": "decimal(10, 2)" }'

# cast multiple columns
sling run --columns '{ "my_column": "string(150)", "my_other_column": "decimal(10, 2)" }'

# cast all columns to string
sling run --columns '{ "*": "string" }'

# apply constraint, value should be greater than 0
sling run --columns '{ "my_column": "integert | value > 0" }'

```

### Using YAML

Using the `defaults` and `streams` keys, you can specify different columns for each stream.

```yaml
source: source_name
target: target_name

defaults:
  # apply to all streams by default
  columns:
    id: bigint

streams:
  # inherit defaults
  my_stream:

  my_other_stream:
    # apply to this stream only (replaces defaults)
    columns:
      id: bigint
      my_column: string(150)
      my_other_column: decimal(10, 2)
      my_int: int | value > 0   # apply constraint, value should be greater than 0

  # cast all columns to string
  another_stream:
    columns:
      "*": string
```

#### Merging with Defaults

By default, stream-level `columns` **replace** `defaults.columns` entirely. To **merge** stream columns with defaults instead, prefix column names with `+`. This inherits all defaults and lets you add or override specific columns. Requires *v1.5.12+*.

```yaml
source: oracle
target: postgres

defaults:
  columns:
    note_id: bigint
    source_id: bigint
    trip_header_id: bigint
    note: text

streams:
  # inherits all defaults
  table_a:

  # "+" prefix: merges with defaults
  # inherits note_id, source_id, trip_header_id, note from defaults
  # adds trip_id: bigint
  table_b:
    columns:
      +trip_id: bigint

  # without "+": replaces defaults entirely (legacy behavior)
  # only trip_id: bigint is applied, no defaults inherited
  table_c:
    columns:
      trip_id: bigint
```

{% hint style="warning" %}
You cannot mix `+` prefixed and non-prefixed column names in the same `columns` block. Either all columns use the `+` prefix (merge mode) or none do (replace mode).
{% endhint %}

#### Unsetting a Default

When using merge mode (`+` prefix), you can remove a default column type for a specific stream by setting it to `null` using YAML's `~`. This reverts that column to the auto-detected type from the source.

```yaml
defaults:
  columns:
    id: bigint
    name: text

streams:
  my_stream:
    columns:
      +id: ~      # unset: uses the auto-inferred type from the source
                   # name: text is still inherited from defaults
```

## Column Casing

The `column_casing` [target option](/concepts/replication/target-options) allows you to control how column names are formatted when creating tables in the target database. This is useful for ensuring consistent naming conventions and avoiding the need to use quotes when querying tables in databases with case-sensitive identifiers.

Starting in `v1.4.5`, the default is `normalize`. Before this version, the default was `source`. See note [here](https://github.com/slingdata-io/sling-cli/issues/538#issuecomment-2799022785) for details.

### Available Casing Options

* `normalize` - Normalize column names to target database's default casing (upper or lower case), but preserve mixed-case column names. This helps with querying tables without needing quotes for standard column names.
* `source` - Keep the original casing from the source data.
* `target` - Convert all column names according to the target database's default casing (upper case for Oracle, lower case for PostgreSQL, etc.).
* `snake` - Convert `camelCase` and other formats to `snake_case`, then apply the target database's default casing.
* `upper` - Convert all column names to UPPER CASE.
* `lower` - Convert all column names to lower case.

### Examples

Assuming source column names: `customerId`, `first_name`, `LAST_NAME`, `email-address`

#### Using CLI Flag

```bash
# Keep source casing
sling run --tgt-options '{"column_casing": "source"}'

# Use target database casing
sling run --tgt-options '{"column_casing": "target"}'

# Convert to snake case with target database casing
sling run --tgt-options '{"column_casing": "snake"}'

# Convert to upper case
sling run --tgt-options '{"column_casing": "upper"}'

# Convert to lower case
sling run --tgt-options '{"column_casing": "lower"}'
```

#### Using YAML

```yaml
source: mysql.customer_data
target: postgres.public.customers

defaults:
  mode: full-refresh
  target_options:
    column_casing: snake
```

#### Example Results

Let's see how each option transforms our sample column names for a PostgreSQL, DuckDB or MySQL target (which defaults to lowercase):

| Original Column | source        | normalize     | target         | snake          | upper          | lower          |
| --------------- | ------------- | ------------- | -------------- | -------------- | -------------- | -------------- |
| customerId      | customerId    | customerId    | customerid     | customer\_id   | CUSTOMERID     | customerid     |
| first\_name     | first\_name   | first\_name   | first\_name    | first\_name    | FIRST\_NAME    | first\_name    |
| LAST\_NAME      | LAST\_NAME    | last\_name    | last\_name     | last\_name     | LAST\_NAME     | last\_name     |
| email-address   | email-address | email-address | email\_address | email\_address | EMAIL\_ADDRESS | email\_address |

For an Oracle or Snowflake target (which defaults to uppercase):

| Original Column | source        | normalize     | target         | snake          | upper          | lower          |
| --------------- | ------------- | ------------- | -------------- | -------------- | -------------- | -------------- |
| customerId      | customerId    | customerId    | CUSTOMERID     | CUSTOMER\_ID   | CUSTOMERID     | customerid     |
| first\_name     | first\_name   | FIRST\_NAME   | FIRST\_NAME    | FIRST\_NAME    | FIRST\_NAME    | first\_name    |
| LAST\_NAME      | LAST\_NAME    | LAST\_NAME    | LAST\_NAME     | LAST\_NAME     | LAST\_NAME     | last\_name     |
| email-address   | email-address | email-address | EMAIL\_ADDRESS | EMAIL\_ADDRESS | EMAIL\_ADDRESS | email\_address |

This functionality makes it easier to work with column names when moving data between systems with different naming conventions or case sensitivity requirements.

## Column Typing

Starting in `v1.4.5`, the `column_typing` [target option](/concepts/replication/target-options) allows you to configure how Sling generates column types when creating tables in the target database. This is particularly useful when you need to ensure string columns have sufficient length to accommodate all possible values, especially when dealing with different database systems or character encodings.

### Structure

The `column_typing` configuration has the following structure:

```yaml
target_options:
  column_typing:
    string:
      length_factor: <int>
      min_length: <int>
      max_length: <int>
      use_max: <bool>

    decimal:
      min_precision: <int>
      max_precision: <int>
      min_scale: <int>
      max_scale: <int>
      cast_as: <string>  # "float" or "string"

    json:
      as_text: <bool>

    boolean:
      cast_as: <string>  # "integer" or "string"
```

Where:

* `string`: Settings for string type columns
  * `length_factor`: A multiplier applied to the detected length of string columns (default: 1)
  * `min_length`: The minimum length to use for string columns (if specified)
  * `max_length`: The maximum length to use for string columns (if specified)
  * `use_max`: Whether to always use the max\_length value instead of calculated lengths (default: false)
* `decimal`: Settings for decimal type columns
  * `min_precision`: The minimum total number of digits (precision) for decimal columns.
  * `max_precision`: The maximum total number of digits (precision) for decimal columns.
  * `min_scale`: The minimum number of digits after the decimal point (scale).
  * `max_scale`: The maximum number of digits after the decimal point (scale).
  * `cast_as`: Force decimal columns to be cast as a specific type (available in `v1.4.27`+):
    * `"float"`: Convert decimal columns to floating-point type (e.g., `DOUBLE PRECISION` in PostgreSQL)
    * `"string"`: Convert decimal columns to string/text type (e.g., `VARCHAR` in PostgreSQL)
* `json`: Settings for JSON type columns
  * `as_text`: When set to `true`, JSON columns are stored as text/string type instead of native JSON type (default: false). This is useful when the target database doesn't support native JSON types, or when you need to store JSON data as plain text for compatibility.
* `boolean`: Settings for boolean type columns
  * `cast_as`: Force boolean columns to be cast as a specific type:
    * `"integer"`: Convert boolean columns to integer type (1 for true, 0 for false)
    * `"string"`: Convert boolean columns to string type ("true" or "false")

### How It Works

When Sling creates tables in the target database, it analyzes the source data to determine appropriate column types.

For string columns:

1. Sling determines the maximum string length from the source data
2. If `length_factor` is specified, this value is multiplied by the factor
3. If `max_length` is specified and `use_max` is false, the length is capped at this value
4. If `use_max` is true, `max_length` is used regardless of the calculated length
5. If `min_length` is specified, the length will be at least that number

For decimal columns:

1. Sling determines the required precision and scale based on the source data.
2. If `cast_as` is specified:
   * `"float"`: Decimal columns are converted to floating-point type, ignoring precision/scale settings
   * `"string"`: Decimal columns are converted to string/text type, preserving exact decimal representation
3. Otherwise, if `column_typing.decimal` settings are provided, Sling adjusts the calculated precision and scale based on the `min_precision`, `max_precision`, `min_scale`, and `max_scale` values.
4. The final precision and scale are used to generate the `decimal(precision, scale)` type in the target database DDL.

This helps prevent truncation issues when moving data between systems with different character encoding requirements or different decimal precision/scale needs.

### Examples

#### Double String Column Lengths

```yaml
source: mssql
target: redshift

defaults:
  mode: truncate
  object: public.{stream_table}
  
streams:
  dbo.test_sling_unicode:
    target_options:
      column_typing:
        string:
          length_factor: 2
```

In this example, all string columns in the `dbo.test_sling_unicode` table will have their length doubled when created in the PostgreSQL target. This is useful when moving from a database that uses single-byte encoding to one that uses multi-byte encoding (like UTF-8).

#### Set Maximum String Length

```yaml
source: mysql
target: oracle

defaults:
  target_options:
    column_typing:
      string:
        max_length: 8000
```

This example sets a maximum length of 8000 characters for all string columns across all streams, which is useful for databases with column size limitations.

#### Use Fixed String Length

```yaml
streams:
  sales.customers:
    target_options:
      column_typing:
        string:
          max_length: 1000
          use_max: true
```

This configuration forces all string columns in the `sales.customers` table to use a fixed length of 1000, regardless of the actual data length.

#### Cast Decimals as Float

```yaml
source: mssql
target: postgres

streams:
  dbo.financial_data:
    target_options:
      column_typing:
        decimal:
          cast_as: float
```

This example converts all decimal columns to floating-point type (`DOUBLE PRECISION` in PostgreSQL). This is useful when you need better performance for numeric operations and can accept the loss of exact decimal precision.

#### Cast Decimals as String

```yaml
source: mssql
target: postgres

streams:
  dbo.accounting_records:
    target_options:
      column_typing:
        decimal:
          cast_as: string
```

This example converts all decimal columns to string type (`VARCHAR` in PostgreSQL). This is useful when you need to preserve exact decimal representation, including trailing zeros, or when the target database doesn't support the required decimal precision.

#### Store JSON as Text

```yaml
source: postgres
target: mysql

streams:
  public.events:
    target_options:
      column_typing:
        json:
          as_text: true
```

This example stores JSON columns as text instead of native JSON type. This is useful when replicating to databases that don't have native JSON support, or when you want to store JSON data as plain text for compatibility with older systems or specific application requirements.

#### Cast Booleans as Integer

```yaml
source: postgres
target: oracle

streams:
  public.feature_flags:
    target_options:
      column_typing:
        boolean:
          cast_as: integer
```

This example converts boolean columns to integer type (1 for true, 0 for false). This is useful when replicating to databases that don't have native boolean support, such as Oracle, or when integrating with legacy systems that expect numeric flags. Available in v1.5.3+.

#### Cast Booleans as String

```yaml
source: mysql
target: bigquery

streams:
  app.settings:
    target_options:
      column_typing:
        boolean:
          cast_as: string
```

This example converts boolean columns to string type ("true" or "false"). This can be useful for data warehousing scenarios where you want boolean values stored as human-readable text, or when the target system expects string representations of boolean values.


# Transforms

Using Sling transforms

Sling provides powerful data transformation capabilities that allow you to clean, modify, and enhance your data during the replication process. Transform data inline without needing separate ETL tools or complex SQL queries.

Starting with **v1.4.17**, Sling introduced **Functions and Staged Transforms** - a major enhancement that provides access to 50+ built-in functions with flexible expression-based transformations. Before this version, only a limited set of legacy transforms were available.

## Transform Input Structures

Sling supports three different input structures for transforms, each offering varying degrees of flexibility and simplicity:

### Array of Strings - Simple Global Transforms

Apply transform functions in sequence to **all column values**. Works with functions that accept 1 parameter.

```yaml
# Apply to all columns globally
transforms: ["trim_space", "remove_diacritics"]
```

```bash
# CLI usage
sling run --transforms '["trim_space", "remove_diacritics"]' ...
```

**Use case**: When you need the same transformation applied to every column in your dataset.

### Map/Object - Column-Specific Transforms

Apply transform functions in sequence to **specific columns**. Works with functions that accept 1 parameter.

```yaml
# Apply to specific columns
transforms:
  name: ["trim_space", "upper"]
  email: ["lower"]
  customer_id: ["hash"]
  target: ["bool_parse"]
  text: ["replace_non_printable"]
```

```bash
# CLI usage
sling run --transforms '{"name": ["trim_space", "upper"], "email": ["lower"]}' ...
```

**Use case**: When you need different transformations for different columns, or want to combine global and column-specific transforms.

### Array of Objects - Staged Transforms

**Multi-stage transformations** with expressions and functions, evaluated in order. Each stage can modify existing columns or create new ones.

```yaml
transforms:
  # Stage 1: Clean text fields
  - text_field: "trim_space(value)"
    email: "lower(value)"
  
  # Stage 2: Create new columns using record references
  - full_name: 'record.first_name + " " + record.last_name'
    email_hash: 'hash(record.email, "md5")'
  
  # Stage 3: Conditional logic
  - category: 'record.amount >= 1000 ? "premium" : "standard"'
```

**Use case**: When you need complex, multi-step transformations with conditional logic, cross-column references, or computed fields.

## Staged Transforms

Staged transforms provide the most powerful and flexible transformation capabilities in Sling. They allow you to:

* **Multi-stage processing**: Apply transformations in sequential stages
* **Cross-column references**: Use `record.<column>` to reference other column values
* **Conditional logic**: Apply `if/then/else` logic based on multiple conditions
* **Create new columns**: Generate computed columns based on existing data
* **Access to 50+ functions**: Use string, numeric, date, and utility functions

### Syntax

Staged transforms use an array of transformation stages. Each stage can modify existing columns or create new ones using expressions parsed by [Goval](https://github.com/maja42/goval), a powerful Go expression evaluator:

```yaml
transforms:
  # Stage 1: Clean text fields
  - text_field: "trim_space(value)"
    email: "lower(value)"
  
  # Stage 2: Create new columns using record references
  - full_name: 'record.first_name + " " + record.last_name'
    email_hash: 'hash(record.email, "md5")'
  
  # Stage 3: Conditional logic
  - category: 'record.amount >= 1000 ? "premium" : "standard"'
```

### Key Features

#### 1. Value Transformations

Use `value` to reference the current column's value:

```yaml
transforms:
  - name: 'upper(value)'          # Convert to uppercase
    age: 'cast(value, "int")'     # Convert to integer

  - "*": "trim_space(value)"      # Then, apply to all columns
```

#### 2. Record References

Use `record.<column>` to reference other columns in the same row:

```yaml
transforms:
  - full_name: 'record.first_name + " " + record.last_name'
    total_amount: 'record.quantity * record.price'
    display_name: 'record.last_name + ", " + record.first_name'
```

{% hint style="info" %}
**Name Casing**: When using `record.<column>`, the key name is always **lowercase**. If your source database column has uppercase letters (e.g., `FirstName` or `LAST_NAME`), use the lowercase version in your expression: `record.firstname` or `record.last_name`.
{% endhint %}

#### 3. Multi-stage Processing

Each stage can build upon the results of previous stages:

```yaml
transforms:
  # Stage 1: Clean the data
  - name: "trim_space(value)"
    email: "lower(value)"
  
  # Stage 2: Use cleaned data from stage 1
  - full_name: 'record.name + " (" + record.email + ")"'
  
  # Stage 3: Use results from previous stages  
  - summary: 'record.full_name + " - processed"'
```

#### 4. Conditional Transformations

Use ternary operators and conditional logic:

```yaml
transforms:
  - status: 'record.amount >= 1000 ? "premium" : (record.amount >= 500 ? "standard" : "basic")'
    discount: 'record.status == "premium" ? record.amount * 0.15 : record.amount * 0.05'
    message: 'record.age >= 65 ? "Senior discount available" : "Standard pricing"'
```

#### 5. Available Functions

Access to 50+ built-in functions including:

* **String functions**: `upper()`, `lower()`, `trim_space()`, `replace()`, `substring()`
* **Numeric functions**: `int_parse()`, `float_parse()`, `greatest()`, `least()`
* **Date functions**: `now()`, `date_parse()`, `date_format()`, `date_add()`
* **Utility functions**: `hash()`, `coalesce()`, `cast()`, `uuid()`
* **Conditional functions**: `if()`, `equals()`, `is_null()`, `is_empty()`

See the complete [Functions Reference](/concepts/functions) for all available functions.

#### 6. Creating New Columns On-The-Fly

One of the most powerful features of staged transforms is the ability to create new columns dynamically during data processing. Unlike traditional transformations that only modify existing columns, staged transforms let you add computed columns, derived fields, and calculated values without modifying your source data or schema.

```yaml
streams:
  my_stream:
    transforms:
      # first stage: replace column (using value)
      - mycol: 'coalesce(value, "N/A")'     # replace column
        email_hashed: 'hash(record.email, "md5")'  # create new column
      
      # second stage: create new column  
      - new_col: 'record.mycol * 100'       # create new column
      
      # final stage: cast all as string and replace accents
      - "*": 'replace_accents(cast(value, "string"))'        
```

**Key Benefits:**

* **No schema changes needed**: Add columns without altering source tables
* **Dynamic calculations**: Create computed fields based on existing data
* **Multi-stage logic**: Build complex columns using results from previous stages
* **Flexible data enrichment**: Add metadata, hashes, flags, or derived metrics

**Common Use Cases:**

* **Data enrichment**: Add calculated fields, hash values, or lookup results
* **Business logic**: Create status flags, categories, or computed metrics
* **Data quality**: Add validation flags, completeness scores, or data lineage
* **Analytics preparation**: Create derived dimensions or calculated measures

### Examples

#### Example 1: Customer Data Processing

```yaml
streams:
  customers:
    transforms:
      # Stage 1: Clean and normalize
      - first_name: "trim_space(value)"
        last_name: "trim_space(value)"
        email: "lower(value)"
      
      # Stage 2: Create computed columns
      - full_name: 'record.first_name + " " + record.last_name'
        email_hash: 'hash(record.email, "md5")'
      
      # Stage 3: Customer categorization
      - customer_type: |
          record.total_orders >= 50 ? "vip" : (
            record.total_orders >= 10 ? "regular" : "new"
          )
```

#### Example 2: E-commerce Order Processing

```yaml
streams:
  orders:
    transforms:
      # Stage 1: Parse and clean numeric values
      - quantity: "int_parse(value)"
        unit_price: "float_parse(value)"
        status: "lower(trim_space(value))"
      
      # Stage 2: Calculate totals
      - subtotal: 'record.quantity * record.unit_price'
        tax_amount: 'record.subtotal * 0.08'
        total_amount: 'record.subtotal + record.tax_amount'
      
      # Stage 3: Status and priority logic
      - priority: |
          record.total_amount >= 1000 ? "high" : (
            record.status == "urgent" ? "high" : "normal"
          )
        shipping_method: |
          record.priority == "high" ? "express" : "standard"
```

#### Example 3: Using CLI

```bash
# Using staged transforms with CLI
sling run \
  --src-conn POSTGRES \
  --src-stream my_schema.raw_data \
  --tgt-conn SNOWFLAKE \
  --tgt-object processed.clean_data \
  --transforms '[
    {"name": "trim_space(value)", "email": "lower(value)"},
    {"full_name": "record.name + \" (\" + record.email + \")\""},
    {"status": "record.active ? \"enabled\" : \"disabled\""}
  ]'
```

## Other Transformation Methods

Beyond the main transform structures, Sling provides additional methods for data transformation:

### Custom SQL Transformations

Use custom `SELECT` queries as the source stream for complex transformations that go beyond what built-in transforms can handle:

```bash
sling run \
  --src-conn STARROCKS \
  --src-stream "SELECT columnB, columnA FROM tbl WHERE columnB > 6000" \
  --tgt-conn MYSQL \
  --tgt-object mysql.tbl \
  --mode full-refresh
```

```yaml
source: STARROCKS
target: MYSQL

streams:
  my_table:
    sql: "SELECT columnB, columnA FROM tbl WHERE columnB > 6000"
    object: mysql.tbl
    mode: full-refresh
```

### JSON Flattening

Automatically flatten nested JSON structures using the `flatten` key in `source.options` (see [configuration docs](https://docs.slingdata.io/sling-cli/run/configuration#source)):

```bash
sling run \
  --src-conn AWS_S3 \
  --src-stream s3://path/to/file.json \
  --tgt-object file://./target/models.csv \
  --src-options '{flatten: true}'
```

```yaml
source: AWS_S3
target: FILE

streams:
  s3://path/to/file.json:
    object: file://./target/models.csv
    source_options:
      flatten: true
```

### JMESPath Transforms

Extract and transform specific data from JSON responses using [JMESPath](https://jmespath.org/) expressions via the `jmespath` key in `source.options`. Here's an example extracting models information from a DBT manifest file:

{% code overflow="wrap" %}

```bash
sling run \
  --src-stream file://./target/manifest.json \
  --tgt-object file://./target/models.csv \
  --src-options '{
      jmespath: "nodes.*.{resource_type: resource_type, database: database, schema: schema, name: name, relation_name: relation_name, original_file_path: original_file_path, materialized: config.materialized }",
      flatten: true
    }'
```

{% endcode %}

{% code overflow="wrap" %}

```yaml
source: LOCAL
target: LOCAL

streams:
  file://./target/manifest.json:
    object: file://./target/models.csv
    source_options:
      jmespath: "nodes.*.{resource_type: resource_type, database: database, schema: schema, name: name, relation_name: relation_name, original_file_path: original_file_path, materialized: config.materialized }"
      flatten: true
```

{% endcode %}

## Schema Evolution

When using Sling to extract/load data in a [`incremental`](https://docs.slingdata.io/sling-cli/run/configuration#incremental-mode-strategies) manner, it will attempt to match whatever columns are present in both the source stream and target table. If an extra column is present in the source stream, it will add it in the target table. If no columns match from source stream at all, it will error. At least the `primary_key` or `update_key` must be present in the target table.

See below for a simple example, mimicking the addition and removal of columns.

```bash
# Initial data

$ echo 'a,b,c
1,2,3
4,5,6' > test1.csv

$ sling run \
  --src-stream file://./test1.csv \
  --tgt-conn postgres \
  --tgt-object public.test1

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559
```

```bash
# test2.csv is missing column b

echo 'a,c
7,8' > test2.csv

$ sling run \
  --src-stream file://./test2.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559
7,,8,1707869689
```

```bash
# test3.csv is missing column b, c and has extra column d

$ echo 'a,d
9,10' > test3.csv

$ sling run \
  --src-stream file://./test3.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at,d
1,2,3,1707869559,
4,5,6,1707869559,
7,,8,1707869689,
9,,,1707870320,10
```

We can see that sling handled the changes properly, in a non-destructive manner. If the source stream were from a database, the same rules would apply, whether a column disappeared or appeared.

## Legacy Transform Syntax (Before v1.4.17)

> **Note**: This is the legacy syntax for transforms used before v1.4.17. While still supported for backwards compatibility, we recommend using the new [Transform Input Structures](#transform-input-structures) for new implementations as they provide much more powerful capabilities and access to functions.

These are legacy single-column transforms that can be applied to clean and modify data:

* `parse_bit`: Parses binary data as bits
* `parse_fix`: Parses FIX (Financial Information eXchange) protocol messages into JSON format
* `parse_uuid`: Parses 16-byte UUID into string format
* `parse_ms_uuid`: Parses 16-byte Microsoft UUID into string format
* `replace_0x00`: Replaces null characters (0x00) with an empty string
* `replace_accents`: Replaces accented characters with their non-accented equivalents
* `replace_non_printable`: Replaces or removes non-printable characters
* `trim_space`: Removes leading and trailing whitespace
* `empty_as_null`: If value is empty, set as null


# Runtime Variables

Learn how to use Runtime & Environment Variables with Sling

## Runtime Variables

A powerful feature that allows dynamic configuration. The used parts will be replaced at runtime with the corresponding values. So you could name your target object `{target_schema}.{stream_schema}_{stream_table}`, and at runtime it will be formatted correctly as depicted below.

* **`source_account`**: the name of the account of the source connection (when source conn is AZURE)
* **`source_bucket`**: the name of the bucket of the source connection (when source conn is GCS or S3)
* **`source_container`**: the name of the container of the source connection (when source conn is AZURE)
* **`source_name`**: the name of the source connection
* **`stream_file_folder`**: the file parent folder name of the stream (when source is a file system)
* **`stream_file_name`**: the file name of the stream (when source is a file system)
* **`stream_file_ext`**: the file extension of the stream (when source is a file system)
* **`stream_file_path`**: the file path of the stream (when source is a file system)
* **`stream_name`**: the name of the stream
* **`stream_schema` / `stream_schema_lower` / `stream_schema_upper`**: the schema name of the source stream (when source is a database)
* **`stream_table` / `stream_table_lower` / `stream_table_upper`**: the table name of the source stream (when source is a database)
* **`stream_full_name`**: the full qualified table name of the source stream (when source is a database)
* **`target_account`**: the name of the account of the target connection (when target is AZURE)
* **`target_bucket`**: the name of the bucket of the target connection (when target is GCS or S3)
* **`target_container`**: the name of the container of the target connection (when target is AZURE)
* **`target_name`**: the name of the target connection
* **`target_schema`**: the default target schema defined in connection (when target is a database)
* **`object_schema`**: the target object table schema (when target is a database)
* **`object_table`**: the target object table name (when target is a database)
* **`object_full_name`**: the target object full qualified table name (when target is a database)
* **`object_name`**: the target object name

#### Timestamp Patterns

* **`run_timestamp`**: The run timestamp of the task (`2006_01_02_150405`)
* **`YYYY`**: The 4 digit year of the run timestamp of the task
* **`YY`**: The 2 digit year of the run timestamp of the task
* **`MMM`**: The abbreviation of the month of the run timestamp of the task
* **`MM`**: The 2 digit month of the run timestamp of the task
* **`DD`**: The 2 digit day of the run timestamp of the task
* **`HH`**: The 2 digit 24-hour of the run timestamp of the task
* **`hh`**: The 2 digit 12-hour of the run timestamp of the task
* **`mm`**: The 2 digit minute of the run timestamp of the task
* **`ss`**: The 2 digit second of the run timestamp of the task

#### Partition Patterns

This only applies when writing parquet files. You must specified the `update_key` along with a `part_` variable in the `object_name`, for example: `object: my/folder/{part_year_month}/{part_day}`.

* **`part_year`**: The 4 digit year partition value of the `update_key`.
* **`part_month`**: The 2 digit month partition value of the `update_key`.
* **`part_year_month`**: Combination of the 4 digit year and the 2 digit month partition values of the `update_key` (e.g. `2024-11` as one value).
* **`part_day`**: The 2 digit day partition value of the `update_key`.
* **`part_week`**: The ISO-8601 2 digit week partition value of the `update_key`.
* **`part_hour`**: The 2 digit hour partition value of the `update_key`.
* **`part_minute`**: The 2 digit minute partition value of the `update_key`.

{% hint style="warning" %}
When using partition patterns, by default, sling will set the `write_partition_columns true` so that duckdb includes the partition columns in the dataset. When setting `write_partition_columns` as `true`, the way DuckDB writes the parquet schema may cause some issues with other tools reading the data at folder level (see [here](https://github.com/slingdata-io/sling-cli/issues/634) for more details). If you'd like to disable this behavior, set environment variable `DUCKDB_WRITE_PARTITION_COLS=false` (applies to version *1.4.20+*).
{% endhint %}

## Environment Variables

Sling also allows you to pass-in environment variables in order to further customize configurations in a scalable manner. We are then able to reuse them in various places in our config files.

### Definition

A convenient way to embed global variables is in the `env.yaml` file. You could also simply define it in the environment, the traditional way.

{% code title="env.yaml" %}

```yaml
connections:
  MYSQL:
    type: mysql
  S3_ZONE_A:
    type: s3

# this sets environment variables in sling process
variables:
  path_prefix: /my/path/prefix
  schema_name: main
  SLING_CLI_TOKEN: xxxxxxxxxxxxxxxx  # picked up machine wide
  SLING_LOG_DIR: ~/.sling/logs
```

{% endcode %}

### Replication

Below we are displaying the full use of Environment Variables as well as [Runtime Vars](#runtime-variables) (such as `stream.schema`, `stream.table`, `YYYY`, `MM` and `DD`).

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MYSQL
target: S3_ZONE_A

defaults:
  # {path_prefix} here is filled in from env var
  object: {path_prefix}/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.parquet
  target_options:
    format: parquet

streams:

  # all tables in schema
  my_schema.*:
    # overwrites default object
    object: {stream_schema}/{stream.table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder
  
  mysql.my_table:
    sql: |
      select * from mysql.my_table
      where date between '{start_date}' and '{end_date}'

env:
  # ${path_prefix} pulls from environment variables in sling process or env
  path_prefix: '${path_prefix}' # From env.yaml (not in Environment)
  start_date: '${START_DATE}'   # From Environment
  end_date: '${END_DATE}'       # From Environment
```

{% endcode %}

## Global Environment Variables

Sling utilizes global environment variables to further configure the load behavior. You can simply define them in your environment, the `env.yaml` file or the `env` section in a task or replication. See [Global Environment Variables](/sling-cli/variables) for more details.


# Tags & Wildcards

## Wildcards

Wildcards are a way to match multiple streams, whether tables or files. They are useful to apply defaults to multiple streams, so you don't have to specify each stream configuration individually.

```yaml
source: my_source_db
target: my_target_db

defaults:
  # will dynamically create the object name based on the schema and table name
  object: my_schema.{stream_schema}_{stream_table}

streams:
  # match all tables in my_schema
  my_schema.*: 

  # match all tables in another_schema that start with a prefix
  another_schema.prefix_*:

  # match all tables in another_schema that end with a suffix
  another_schema.*_suffix:
```

Filtering files in a folder:

```yaml
source: my_source_file
target: my_target_db

defaults:
  # will dynamically create the object name based on the folder and file name
  object: my_schema.{stream_folder}_{stream_file_name}

streams:
  # match all files in folder
  folder/*: 

  # match all files in another_folder that start with a prefix
  another_folder/prefix_*:

  # match all files in another_folder that end with a suffix
  another_folder/*.csv:

  # match all files in another_folder that start with prefix and end with suffix
  another_folder/prefix_*.parquet:
```

This also works for the CLI:

```bash
sling run --src-conn MY_SOURCE_FILE \
  --src-stream another_folder/prefix_*.parquet \
  --tgt-conn MY_TARGET ...
```

## Tags

Tags are a way to categorize your streams. They can be used to filter streams when running a replication, or to create a job in the Sling Platform.

```yaml
source: MY_SOURCE_FILE
target: MY_TARGET

defaults:
  tags: [ finance ]

streams:
  # use default tags
  path/to/file1.csv:

  # override default tags
  path/to/file2.csv:
    tags: [ marketing ]
```

### Create a Platform Job for a specific Tag

Using the Sling Platform, you can create a job for specific tags. Below is an example of creating a job only running the streams with the tag `marketing`.

<div align="center"><img src="/files/1HWv4A0L2QU777WsWNmk" alt="Sling Platform Job Tag" width="500"></div>

### Calling with CLI

```bash
# Run all streams with tag:my_tag
sling run -r my_replication.yaml --streams tag:my_tag

# Run all streams with tag:my_tag or tag:another_tag
sling run -r my_replication.yaml --streams tag:my_tag,tag:another_tag
```


# Merge Strategy

## Overview

When using **incremental** or **backfill** modes with a `primary_key`, Sling needs to merge incoming data with existing records in the target table. The **merge strategy** controls exactly how this merge operation is performed. This is available in *v1.5.7*.

By default, Sling uses the optimal merge strategy for each database. However, you can override this behavior using the `merge_strategy` target option to gain more control over how data is merged.

{% hint style="info" %}
**When Does Merge Strategy Apply?**

Merge strategy only applies when:

* Mode is `incremental` or `backfill`
* A `primary_key` is defined for the stream

For `full-refresh`, `truncate`, or `snapshot` modes, data is inserted directly without merging.
{% endhint %}

## Available Strategies

Sling supports four merge strategies:

| Strategy        | Description                           | Behavior                                                                                                         |
| --------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `update_insert` | Update existing rows, insert new rows | Standard upsert behavior. Matches on primary key, updates existing records, inserts new ones.                    |
| `delete_insert` | Delete matching rows, then insert all | Removes rows with matching primary keys from target, then inserts all source rows. Safe and reliable.            |
| `insert`        | Insert only, skip existing            | Only inserts new rows. Does not update existing records (use for append-only scenarios).                         |
| `update`        | Update only, skip new                 | Only updates existing rows. Does not insert new records (use when target should only contain pre-existing keys). |

### Strategy Comparison

| Strategy        | Updates Existing        | Inserts New | Performance | Use Case                                           |
| --------------- | ----------------------- | ----------- | ----------- | -------------------------------------------------- |
| `update_insert` | Yes                     | Yes         | Best        | Default upsert behavior                            |
| `delete_insert` | Yes (via delete+insert) | Yes         | Good        | When native MERGE unavailable or for simpler logic |
| `insert`        | No                      | Yes         | Fast        | Append-only, idempotent loads                      |
| `update`        | Yes                     | No          | Fast        | Updating existing records only                     |

## Database Support

Not all databases support all merge strategies due to differences in SQL capabilities.

| Database      | Default         | Supported Strategies                       |
| ------------- | --------------- | ------------------------------------------ |
| PostgreSQL    | `update_insert` | **all**                                    |
| Snowflake     | `update_insert` | **all**                                    |
| BigQuery      | `delete_insert` | **all**                                    |
| SQL Server    | `update_insert` | **all**                                    |
| Oracle        | `update_insert` | **all**                                    |
| Databricks    | `update_insert` | **all**                                    |
| SQLite        | `update_insert` | **all**                                    |
| Exasol        | `update_insert` | **all**                                    |
| Cloudflare D1 | `update_insert` | **all**                                    |
| MotherDuck    | `update_insert` | **all**                                    |
| MySQL         | `delete_insert` | `insert`, `delete_insert`                  |
| MariaDB       | `delete_insert` | `insert`, `delete_insert`                  |
| Redshift      | `delete_insert` | `insert`, `delete_insert`                  |
| ClickHouse    | `delete_insert` | `insert`, `delete_insert`                  |
| DuckDB        | `delete_insert` | `insert`, `update`, `delete_insert`        |
| StarRocks     | `delete_insert` | `insert`, `update_insert`, `delete_insert` |

{% hint style="warning" %}
**Why Some Strategies Are Unavailable**

* **MySQL/MariaDB**: No native `MERGE` or `UPDATE...FROM` syntax
* **Redshift**: No native `MERGE` statement
* **ClickHouse**: Columnar architecture doesn't support row-level updates
* **DuckDB**: `update_insert` requires explicit PRIMARY KEY constraint, which Sling doesn't create by default
  {% endhint %}

## Usage

### In Replication YAML

Set the merge strategy in `target_options`:

```yaml
source: postgres
target: snowflake

defaults:
  mode: incremental
  primary_key: [id]
  target_options:
    merge_strategy: delete_insert  # Override the default

streams:
  public.customers:
    object: analytics.customers

  public.orders:
    object: analytics.orders
    target_options:
      merge_strategy: update  # Only update existing orders, don't insert new ones
```

### Per-Stream Override

Override the strategy for specific streams:

```yaml
source: postgres
target: bigquery

defaults:
  mode: incremental
  primary_key: [id]

streams:
  # Uses BigQuery default (delete_insert)
  public.users:
    object: analytics.users

  # Override to insert-only for audit log
  public.audit_log:
    object: analytics.audit_log
    target_options:
      merge_strategy: insert  # Append only, never update

  # Override to update-only for dimension table
  public.product_catalog:
    object: analytics.product_catalog
    target_options:
      merge_strategy: update  # Only update known products
```

### CLI Usage

Use JSON format for `--tgt-options`:

```bash
sling run \
  --src-conn POSTGRES \
  --src-stream "public.customers" \
  --tgt-conn SNOWFLAKE \
  --tgt-object "analytics.customers" \
  --mode incremental \
  --primary-key "id" \
  --tgt-options '{"merge_strategy": "delete_insert"}'
```

## Strategy Details

### `update_insert` (Upsert)

The most common merge pattern. Uses the database's native `MERGE` statement (or equivalent) to:

1. **Match** rows on primary key
2. **Update** matched rows with new values
3. **Insert** unmatched rows

**SQL Pattern** (varies by database):

```sql
MERGE INTO target_table tgt
USING source_table src
ON (src.id = tgt.id)
WHEN MATCHED THEN UPDATE SET col1 = src.col1, col2 = src.col2
WHEN NOT MATCHED THEN INSERT (id, col1, col2) VALUES (src.id, src.col1, src.col2)
```

**Use when:**

* You need standard upsert behavior
* Target database supports MERGE or equivalent
* Performance is important

### `delete_insert`

A two-step approach that's universally supported:

1. **Delete** all rows in target that have matching primary keys in source
2. **Insert** all rows from source

**SQL Pattern**:

```sql
DELETE FROM target_table
WHERE id IN (SELECT id FROM source_table);

INSERT INTO target_table (id, col1, col2)
SELECT id, col1, col2 FROM source_table;
```

**Use when:**

* Database doesn't support native MERGE
* You want simpler, more predictable behavior
* Debugging merge issues
* Working with databases like MySQL, Redshift, or ClickHouse

### `insert`

Only inserts new rows, ignoring any that already exist:

1. **Insert** rows from source that don't exist in target (based on primary key)

**SQL Pattern** (varies by database):

```sql
INSERT INTO target_table (id, col1, col2)
SELECT id, col1, col2 FROM source_table src
WHERE NOT EXISTS (
  SELECT 1 FROM target_table tgt WHERE tgt.id = src.id
)
```

**Use when:**

* Append-only tables (audit logs, event streams)
* Idempotent data loads where updates shouldn't occur
* First-wins conflict resolution

### `update`

Only updates existing rows, never inserts new ones:

1. **Update** rows in target that have matching primary keys in source
2. **Skip** source rows that don't exist in target

**SQL Pattern**:

```sql
UPDATE target_table tgt
SET col1 = src.col1, col2 = src.col2
FROM source_table src
WHERE src.id = tgt.id
```

**Use when:**

* Target table should be the source of truth for which records exist
* Updating dimension tables with enrichment data
* Preventing accidental record creation

## Customizing Merge Templates

For advanced use cases, you can customize the SQL templates used for merge operations. See [Template Overrides](/concepts/replication/templates) for details.

Merge strategy templates are defined in the `core` section:

```yaml
# ~/.sling/templates/postgres.yaml
core:
  merge_update_insert: |
    -- Custom MERGE logic
    WITH updates AS (
      UPDATE {tgt_table} tgt
      SET {set_fields}
      FROM {src_table} src
      WHERE {src_tgt_pk_equal}
      RETURNING tgt.*
    )
    INSERT INTO {tgt_table} ({insert_fields})
    SELECT {src_fields} FROM {src_table} src
    WHERE NOT EXISTS (
      SELECT 1 FROM updates upd WHERE {src_upd_pk_equal}
    )
```

### Available Template Variables

| Variable              | Description                     |
| --------------------- | ------------------------------- |
| `{tgt_table}`         | Target table name               |
| `{src_table}`         | Source (temp) table name        |
| `{src_fields}`        | Source fields with type casting |
| `{insert_fields}`     | Fields for INSERT clause        |
| `{src_insert_fields}` | Source fields for VALUES clause |
| `{set_fields}`        | SET clause for UPDATE           |
| `{src_tgt_pk_equal}`  | Primary key equality condition  |
| `{src_pk_fields}`     | Source primary key fields       |
| `{tgt_pk_fields}`     | Target primary key fields       |

### Disabling a Strategy

To disable a strategy (force users to use a different one), set it to `null`:

```yaml
# ~/.sling/templates/custom_db.yaml
core:
  merge_update_insert: null  # Force use of delete_insert instead
```

{% hint style="info" %}
**Built-in Templates Reference**

View the default merge templates in the [sling-cli repository](https://github.com/slingdata-io/sling-cli/tree/main/core/dbio/templates). Each database file (e.g., `postgres.yaml`, `snowflake.yaml`) shows how merge strategies are implemented.
{% endhint %}

## Best Practices

### 1. Use the Default When Possible

Each database has a default merge strategy optimized for its capabilities. Only override when you have a specific need.

### 2. Consider Performance

* `update_insert`: Usually fastest when database has native MERGE
* `delete_insert`: May be slower for large tables with few updates
* `insert` / `update`: Fastest when you only need one operation

### 3. Handle Edge Cases

| Scenario                     | Recommended Strategy                  |
| ---------------------------- | ------------------------------------- |
| First-time load              | Any (all behave same for empty table) |
| Append-only data             | `insert`                              |
| Full record replacement      | `delete_insert`                       |
| Partial field updates        | `update_insert`                       |
| Cross-database compatibility | `delete_insert`                       |

### 4. Testing

When changing merge strategies:

1. Test with a small dataset first
2. Verify row counts match expectations
3. Check that updates/inserts behave correctly
4. Monitor performance differences

## Troubleshooting

### "merge strategy not supported"

**Error**: `merge strategy 'update_insert' not supported for mysql`

**Cause**: The requested strategy isn't available for the target database.

**Solution**: Use a supported strategy for that database. Check the [Database Support](#database-support) table above.

### Data not updating

**Cause**: Using `insert` strategy when you need `update_insert`.

**Solution**: Change to `update_insert` or `delete_insert`.

### Unexpected row deletions

**Cause**: Using `delete_insert` with partial data loads.

**Solution**: If source only contains some records, consider `update_insert` to avoid deleting non-present records.

### Performance issues

**Cause**: `delete_insert` on large tables with few changes.

**Solution**:

* Use `update_insert` if supported
* Consider partitioning/filtering to reduce scope
* Use `update` if new records shouldn't be inserted


# Templates

## Overview

Sling uses **database templates** to generate SQL statements for operations like creating tables, inserting data, querying metadata, and type mappings. These templates are built-in for each database connector (PostgreSQL, Snowflake, MySQL, SQL Server, etc.).

Starting in *v1.5.3*, you can customize these templates by creating **user override files** in `~/.sling/templates/` directory (or env var `$SLING_HOME_DIR/templates`). This allows you to:

* Customize table creation with database-specific options (e.g., compression, partitioning)
* Modify metadata queries for special schema configurations
* Override type mappings for your data types
* Add database-specific optimizations

## File Location and Naming

User template override files must be placed in:

```
~/.sling/templates/{database_type}.yaml
```

Where `{database_type}` matches your database connection type exactly.

### Examples

* PostgreSQL: `~/.sling/templates/postgres.yaml`
* Snowflake: `~/.sling/templates/snowflake.yaml`
* MySQL: `~/.sling/templates/mysql.yaml`
* SQL Server: `~/.sling/templates/sqlserver.yaml`
* BigQuery: `~/.sling/templates/bigquery.yaml`
* Redshift: `~/.sling/templates/redshift.yaml`

To find your database type, run:

```bash
sling conns list
```

## Template Structure

Each template file contains up to 7 sections. You only need to include the sections you want to override:

### 1. `core` - SQL DDL/DML Templates

Defines SQL statements for basic operations:

```yaml
core:
  drop_table: drop table if exists {table}
  create_table: create table if not exists {table} ({col_types})
  insert: insert into {table} ({fields}) values ({values})
  replace: insert into {table} ({fields}) values ({values}) on conflict do update
  truncate: truncate table {table}
  update_temp: update {table} set {set_fields} where {pk_fields_equal}
```

### 2. `metadata` - Schema Information Queries

Queries to retrieve database metadata:

```yaml
metadata:
  tables: select table_name from information_schema.tables where table_schema = '{schema}'
  columns: select column_name, data_type from information_schema.columns where table_name = '{table}'
  primary_keys: select column_name from information_schema.key_column_usage where table_name = '{table}' and constraint_name like 'PK%'
```

### 3. `analysis` - Analytical Queries

Used for data profiling and validation:

```yaml
analysis:
  count: select count(*) as cnt from {table}
  sample: select {fields} from {table} limit {n}
```

### 4. `function` - Database-Specific Functions

Custom functions available in templates:

```yaml
function:
  cast_to_date: "cast({val} as date)"
  cast_to_string: "cast({val} as varchar)"
```

### 5. `general_type_map` - Generic to Database Type Mapping

Maps generic data types to database-specific types:

```yaml
general_type_map:
  string: varchar(500)
  integer: bigint
  decimal: decimal(38,10)
  boolean: boolean
  datetime: timestamp
```

### 6. `native_type_map` - Database Type to Generic Mapping

Maps database-specific types back to generic types:

```yaml
native_type_map:
  varchar: string
  text: string
  int: integer
  bigint: integer
  numeric: decimal
  timestamp: datetime
```

### 7. `variable` - Database Variables

Database-specific configuration variables:

```yaml
variable:
  quote_char: '"'
  schema_separator: '.'
```

## Examples

### Example 1: Custom PostgreSQL Table Creation with Compression

Add compression and optimization to PostgreSQL table creation:

**File: `~/.sling/templates/postgres.yaml`**

```yaml
core:
  create_table: create table if not exists {table} ({col_types}) with (compression = lz4)
```

This overrides only the `create_table` template. All other PostgreSQL templates remain unchanged.

### Example 2: Custom Snowflake Type Mappings

Override type mappings for Snowflake to use specific precision:

**File: `~/.sling/templates/snowflake.yaml`**

```yaml
general_type_map:
  decimal: number(38,10)
  integer: number(38,0)
  string: varchar(4000)
```

### Example 3: Modified Metadata Query for Custom Schemas

Customize the columns query for a specific schema configuration:

**File: `~/.sling/templates/snowflake.yaml`**

```yaml
metadata:
  columns: |
    select
      column_name,
      data_type,
      ordinal_position
    from information_schema.columns
    where table_catalog = '{database}'
      and table_schema = '{schema}'
      and table_name = '{table}'
    order by ordinal_position
```

### Example 4: MySQL with Specific Collation

**File: `~/.sling/templates/mysql.yaml`**

```yaml
core:
  create_table: create table if not exists {table} ({col_types}) collate utf8mb4_unicode_ci
```

## How It Works

Templates are loaded in a layered hierarchy, with each layer overwriting nested keys from the previous:

1. **Base Template**: Sling first loads [`base.yaml`](https://github.com/slingdata-io/sling-cli/blob/main/core/dbio/templates/base.yaml), which contains common SQL templates shared across all databases
2. **Database-Specific Template**: The database-specific template (e.g., `postgres.yaml`, `snowflake.yaml`) is loaded next, overwriting any nested keys from the base template with database-specific implementations
3. **User Override**: If a user template exists at `~/.sling/templates/{database_type}.yaml`, it is merged last, overwriting any nested keys from the previous layers
4. **Partial Overrides**: Only the keys you define override the built-in templates - all other values remain unchanged
5. **Caching**: The merged template is cached for performance
6. **Debug Logging**: With `--debug` flag, you can see when overrides are loaded

{% hint style="info" %}
**Built-in Templates Reference**

View the default built-in templates at <https://github.com/slingdata-io/sling-cli/tree/main/core/dbio/templates>. The [`base.yaml`](https://github.com/slingdata-io/sling-cli/blob/main/core/dbio/templates/base.yaml) file contains the foundational templates, while database-specific files (e.g., `postgres.yaml`, `snowflake.yaml`, `mysql.yaml`) show how each database overrides the base. These serve as examples when creating your own custom overrides.
{% endhint %}


# API Specs

Sling API specs are YAML files that define how to interact with REST APIs. They provide a structured way to specify authentication methods, define endpoints, configure request parameters, handle pagination, process responses, manage state for incremental loads, and more.

{% hint style="success" %}
**CLI Pro Required**: APIs require a [CLI Pro token](/sling-cli/cli-pro) or [Platform Plan](https://github.com/slingdata-io/sling-docs/blob/master/concepts/sling-platform/platform.md).
{% endhint %}

## Quick Start to build a Spec

Here's a complete, working example of a Sling API spec that fetches user data from a REST API:

```yaml
name: "Example API"
description: "Simple API to get user data"

defaults:
  state:
    base_url: https://api.example.com/v1

  request:
    headers:
      Accept: "application/json"
      Authorization: "Bearer {secrets.api_token}"  # read from env.yaml secrets section

endpoints:
  users:
    description: "Retrieve list of users"
    docs: https://docs.example.com/users # docs url for endpoint
    disabled: false                      # Set to true to temporarily disable
    
    # State variables control request parameters and track values between runs
    state:
      page: 1          # Start at page 1
      limit: 100       # Fetch 100 records per request
    
    request:
      # Will resolve to https://api.example.com/v1/users
      url: '{state.base_url}/users'
      parameters:
        page: '{state.page}'
        limit: '{state.limit}'
    
    # Control how to fetch next pages
    pagination:
      next_state:
        page: '{state.page + 1}'  # Increment page number for next request
      stop_condition: "length(response.records) < state.limit"  # Stop when page isn't full
    
    # Define how to extract and process response data
    response:
      records:
        jmespath: "data.users[]"  # Extract array of users from response
        primary_key: ["id"]       # Use 'id' field to deduplicate records
```

To run this spec with Sling:

1. Save your spec somewhere (local disk, S3, SFTP, http). You can access your spec by the [location string](/sling-cli/environment#location-string) convention.
2. Create a connection in your [env.yaml](/sling-cli/environment#sling-env-file-envyaml) file, like this:

```yaml
connections:
  my_api:
    type: api
    spec: file:///path/to/my_api.spec.yaml  # or Github repo file, HTTP URL
    secrets:
      api_key: xxxxxxxxxxxxxxxxxx
  
  my_postgres:
    url: postgres://....
```

3. Create a replication

```yaml
source: my_api
target: my_postgres

streams:
   # '*' will read from all endpoints (endpoint name = stream name)
   # or you can simply input the specific endpoint names
  '*':
    object: my_schema.{stream_name}
    mode: full-refresh
```

4. Run Replication.

```bash
sling run -r replication.api_postgres.yaml
```

That's it!

{% hint style="info" %}
**Build Specs with AI:** You can use AI to automatically research APIs and build specs for you. See [Using AI to build API specs](/sling-cli/ai#building-a-custom-api-connector) for a guided workflow.
{% endhint %}

## Official Specs

We are actively building the number of "official" Sling API Specs offered, such as:

* [Airtable](/connections/api-connections/airtable)
* [HubSpot](/connections/api-connections/hubspot)
* [Stripe](/connections/api-connections/stripe)
* [Github](/connections/api-connections/github)

Go [here](/connections/api-connections) to see the full list.

Official specs don't require a full URL or Path. They are maintained by the Sling team and can be fetched by specifying the corresponding ID, such as `stripe`, `salesforce` or `github`.

```yaml
connections:
  stripe_api:
    type: api
    spec: stripe  # Use official stripe spec
    secrets:
      api_key: sk_live_xxxxxx
```

{% hint style="info" %}
If you'd like us to build a new spec, please submit a request by filling out this [Google Form](https://docs.google.com/forms/d/e/1FAIpQLScIgkpTC6C-nWaW6atc5eLFl3uUNiIakw37WL69HuPUks08aQ/viewform?usp=dialog), or by submitting a new [Github issue](https://github.com/slingdata-io/sling-cli/issues) (choosing the API Spec option).
{% endhint %}

### Forking Official Specs

When you run a connection test or use an official spec, Sling automatically downloads and caches the spec file locally at:

```
$SLING_HOME_DIR/api/specs/{spec_id}.yaml
```

For example, if you're using the `stripe` spec, after running `sling conns test my_stripe_conn`, you'll find the spec at `~/.sling/api/specs/stripe.yaml`.

To fork and customize an official spec:

1. Run a connection test to download the spec:

   ```bash
   sling conns test my_stripe_conn
   ```
2. Copy the spec from the cache folder:

   ```bash
   cp ~/.sling/api/specs/stripe.yaml ./my_custom_stripe.spec.yaml
   ```
3. Modify the spec as needed for your use case
4. Update your connection to use your custom spec:

   ```yaml
   connections:
     my_stripe:
       type: api
       spec: file://./my_custom_stripe.spec.yaml
       secrets:
         api_key: sk_live_xxxxxx
   ```

## API Spec Documentation

This section covers the details of building Sling API specifications:

* [**Structure**](/concepts/api-specs/structure)**:** Understand the fundamental YAML structure of API specs, including endpoints, state management, sync variables, stream overrides, and lifecycle sequences (setup/teardown).
* [**Authentication**](/concepts/api-specs/authentication)**:** Configure authentication methods including Bearer tokens, Basic Auth, OAuth2, AWS Signature V4, and custom sequence-based authentication workflows.
* [**Requests & Iteration**](/concepts/api-specs/request)**:** Define HTTP requests with URLs, methods, headers, parameters, and payloads. Configure iteration to loop requests over data sets or queues, and use setup/teardown sequences for multi-step workflows.
* [**Response Processing**](/concepts/api-specs/response)**:** Process API responses in multiple formats (JSON, CSV, XML, JSON Lines), extract records with JMESPath, configure deduplication strategies, and access response state for pagination and rules.
* [**Queues**](/concepts/api-specs/queues)**:** Pass data between endpoints using queues for multi-step extraction workflows, such as collecting IDs from one endpoint to use in detail requests from another endpoint.
* [**Dynamic Endpoints**](/concepts/api-specs/dynamic-endpoints)**:** Programmatically generate multiple endpoint configurations based on runtime data, perfect for APIs where available resources aren't known until queried.
* [**Advanced Features**](/concepts/api-specs/advanced)**:** Master pagination strategies (cursor, offset, page-based), response processors for data transformation, sync state for incremental loads, and rules for error handling with intelligent retry and backoff strategies.
* [**Expression Functions**](/concepts/functions)**:** Leverage built-in functions within expressions (`{...}`) for data manipulation, date operations, type casting, string operations, and control flow throughout your API spec configuration.
* [**Testing & Debugging**](/concepts/api-specs/testing-debug)**:** Test your API specs using the `sling conns test` command with `--debug` and `--trace` flags to inspect request/response details, troubleshoot issues, and optimize your configuration.
* [**Troubleshooting**](/concepts/api-specs/troubleshooting)**:** Common error messages, debugging techniques, and solutions for authentication, pagination, and JMESPath issues.

## API Workflow Overview

{% @mermaid/diagram content="graph TD
A\[Define API Spec YAML] --> B\[Configure Authentication]
B --> C\[Define Endpoints]
C --> D\[Configure Requests]
D --> E\[Setup Response Processing]
E --> F\[Handle Pagination]
F -->|Optional| G\[Setup Queues]
G -->|Optional| H\[Configure Additional Endpoints]
E -->|Optional| I\[Define Incremental Sync]

```
style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style D fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style E fill:#26c6da,stroke:#ffffff,stroke-width:2px,color:#ffffff
style F fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff
style H fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

For practical examples, see the [API Examples](https://github.com/slingdata-io/sling-docs/blob/master/examples/api/README.md) section.

## Common Use Cases

| Use Case               | Key Features                                                    |
| ---------------------- | --------------------------------------------------------------- |
| Simple data extraction | Basic endpoint definition, JMESPath extraction                  |
| Paginated data         | Pagination configuration with `next_state` and `stop_condition` |
| Incremental updates    | `sync` state variables, timestamp filtering                     |
| Dependent requests     | Queues, iteration over IDs                                      |
| Authentication         | Bearer tokens, Basic auth, OAuth2                               |
| Rate limiting          | Response rules, backoff strategies                              |

> 💡 **Tip:** Start with a minimal working spec and gradually add more advanced features as needed.


# Structure

This document covers the fundamental structure of a Sling API specification file.

## Root Level

At the root level, we have the following keys:

```yaml
# 'name' and 'endpoints' (or 'dynamic_endpoints') are required
name: <API display name>
description: <API description>

defaults: <endpoint configuration map>

authentication: <authentication configuration map>

endpoints:
  <endpoint name>: <endpoint configuration map>
```

## Endpoint Level

The `<endpoint name>` identifies the API endpoint to interact with. This can be any descriptive name for the endpoint.

The `<endpoint configuration map>` is a map object which accepts the following keys:

```yaml
name: <endpoint name>
description: <endpoint description>
docs: <documentation URL>
disabled: true | false
queue_only: true | false  # endpoint only populates queues; produces no record stream

state: {<map of state variables>}
sync: [<array of state variable names to persist>]

request: <request configuration map>
pagination: <pagination configuration map>
response: <response configuration map>

iterate: <iteration configuration map>
setup: [<array of setup calls>]
teardown: [<array of teardown calls>]

depends_on: [<array of upstream endpoint names>]
overrides: <stream processor configuration overrides>
```

## Request Configuration

The `<request configuration map>` accepts the keys below:

```yaml
url: <endpoint URL>
method: GET | POST | PUT | PATCH | DELETE | HEAD | OPTIONS | TRACE | CONNECT
timeout: <timeout in seconds>
headers: {<map of header name to value>}
parameters: {<map of parameter name to value>}
payload: <request body data>
rate: <maximum requests per second>
concurrency: <maximum concurrent requests>
```

## Pagination Configuration

The `<pagination configuration map>` accepts the keys below:

```yaml
next_state: {<map of state variables to update for next page>}
stop_condition: <expression to determine when to stop paginating>
```

## Response Configuration

The `<response configuration map>` accepts the keys below:

```yaml
format: json | csv | xml
records: <records extraction configuration map>
processors: [<array of processor configurations>]
rules: [<array of response rule configurations>]
```

## Records Configuration

The `<records extraction configuration map>` accepts the keys below:

```yaml
jmespath: <JMESPath expression to extract records>
jq: <jq expression to extract records>
primary_key: [<array of column names for primary key>]
update_key: <column name for incremental updates>
limit: <maximum number of records to process>
duplicate_tolerance: <bloom filter settings: "capacity,error_rate">
```

> ⚠️ **Note:** `jmespath` and `jq` are mutually exclusive — use one or the other, not both. See [Functions](/concepts/functions) for syntax differences between the two.

> 💡 **Primary Key Priority:** When using API specs in replications, the primary key defined in the replication stream configuration takes priority over the primary key defined in the API spec. If no primary key is specified in the stream, the primary key from the spec will be used.

## Processor Configuration

Each processor in the processors array accepts:

```yaml
aggregation: none | maximum | minimum | collect | first | last
expression: <transformation expression>
output: <output destination (record field, state variable, queue, environment variable, or store)>
# Examples:
# - record.field_name (add/update field in record)
# - record (replace entire record)
# - state.variable_name (store in state, requires aggregation)
# - queue.queue_name (send to queue)
# - env.VAR_NAME (set environment variable, requires aggregation)
# - context.store.key_name (store in replication store, requires aggregation)
```

## Response Rules

Each rule in the rules array accepts:

```yaml
action: retry | continue | stop | fail
condition: <boolean expression>
max_attempts: <maximum retry attempts>
backoff: none | constant | linear | exponential | jitter
backoff_base: <base duration in seconds for backoff>
message: <custom message for rule execution>
```

## Authentication Configuration

The `<authentication configuration map>` accepts the keys below:

```yaml
type: none | static | basic | oauth2 | aws-sigv4 | hmac | sequence
expires: <re-authentication interval in seconds>

# Static header authentication
headers: {<map of header name to value>}

# Basic authentication
username: <username>
password: <password>

# OAuth2 authentication
flow: client_credentials | authorization_code | device_code
authentication_url: <OAuth token URL>
authorization_url: <OAuth authorization URL>
device_auth_url: <OAuth device auth URL>
client_id: <OAuth client ID>
client_secret: <OAuth client secret>
scopes: [<array of OAuth scopes>]
redirect_uri: <OAuth redirect URI>

# AWS Signature V4 authentication
aws_service: <AWS service name>
aws_access_key_id: <AWS access key>
aws_secret_access_key: <AWS secret key>
aws_session_token: <AWS session token>
aws_region: <AWS region>
aws_profile: <AWS profile>

# HMAC authentication
algorithm: sha256 | sha512
secret: <HMAC secret key>
signing_string: <template for string to sign>
request_headers: {<map of header name to value template>}
nonce_length: <random nonce length in bytes>

# Sequence authentication (custom calls)
sequence: [<array of authentication calls>]
```

## Iteration Configuration

The `<iteration configuration map>` accepts the keys below:

```yaml
over: <expression that evaluates to an array or queue>
into: <state variable name to store current iteration value>
if: <condition expression to evaluate before iteration>
concurrency: <maximum parallel iterations>
```

## Endpoint Dependencies

The `depends_on` field explicitly declares that an endpoint depends on other endpoints completing first. This is useful for controlling execution order.

```yaml
endpoints:
  # First endpoint: Collects customer IDs
  customers:
    request:
      url: "{state.base_url}/customers"
    response:
      processors:
        - expression: "record.id"
          output: "queue.customer_ids"

  # Second endpoint: Depends on customers endpoint
  customer_orders:
    depends_on: ["customers"]  # Wait for customers to complete first
    iterate:
      over: "queue.customer_ids"
      into: "state.customer_id"
    request:
      url: "{state.base_url}/customers/{state.customer_id}/orders"
```

> 📝 **Note:** When using queues with `iterate.over`, Sling automatically infers dependencies. The `depends_on` field is optional but can make dependencies explicit.

## Stream Overrides

The `overrides` field allows you to configure how the endpoint's data is processed when writing to a destination. This is used during replication to control stream-specific behavior.

### Basic Overrides

Control the replication mode for specific endpoints:

```yaml
endpoints:
  # Full refresh for dimension tables
  customers:
    request:
      url: "{state.base_url}/customers"
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]

    overrides:
      mode: full-refresh  # Always replace all data

  # Incremental for fact tables
  transactions:
    request:
      url: "{state.base_url}/transactions"
      parameters:
        updated_since: "{state.last_sync_timestamp}"
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
        update_key: "updated_at"

    overrides:
      mode: incremental  # Only new/updated records. User would have to manually drop/truncate the table.
```

Available modes:

* `full-refresh`: Replace all data (truncate and load)
* `incremental`: Append new records only
* `snapshot`: Create versioned snapshots
* `backfill`: Historical data loading

### Hooks Override

Add post-processing hooks for specific endpoints. This is powerful for merge operations, data cleanup, or custom transformations:

```yaml
endpoints:
  customer_balance_transaction:
    request:
      url: "{state.base_url}/customers/{state.customer_id}/balance_transactions"

    iterate:
      over: "queue.customer_ids"
      into: "state.customer_id"

    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]

    overrides:
      mode: full-refresh
      hooks:
        post:
          # Check that parent customer data exists
          - type: check
            check: '!is_null(runs["customer"]) && run.total_rows > 0'
            failure_message: no customer records to merge with
            on_failure: break

          # Merge balance transactions into customer table
          - type: query
            id: customer-update-merge
            connection: '{target.name}'
            operation: merge
            on_failure: abort
            params:
              strategy: update
              source_table: '{run.object.full_name}'
              target_table: '{runs["customer"].object.full_name}'
              primary_key: [id]

          # Clean up temporary staging table
          - type: query
            connection: '{target.name}'
            operation: drop_table
            params:
              table: '{run.object.full_name}'
```

**Hook Types Available:**

* `check`: Validate conditions before proceeding
* `query`: Execute SQL operations (merge, drop, etc.)
* `log`: Log messages for debugging
* `http`: Call external APIs
* `command`: Run shell commands

See [Hooks documentation](/concepts/hooks) for complete details.

> 💡 **Tip:** Overrides are most useful when extracting large datasets that need special handling during the write phase, or when implementing complex merge/upsert logic.

## State vs. Sync

Understanding the difference between `state` and `sync`:

### State Variables

The `state` field defines variables available during endpoint execution. State is:

* **Temporary**: Exists only during current run
* **Per-endpoint**: Each endpoint has its own state
* **Per-iteration**: Each iteration (if using `iterate`) gets its own state copy

```yaml
endpoints:
  daily_data:
    state:
      start_date: "{date_format(date_add(now(), -1, 'day'), '%Y-%m-%d')}"
      end_date: "{date_format(now(), '%Y-%m-%d')}"
      page_size: 100

    request:
      url: "{state.base_url}/data"
      parameters:
        from: "{state.start_date}"
        to: "{state.end_date}"
        limit: "{state.page_size}"
```

### Sync Variables

The `sync` field lists which state variables should **persist between runs**. This enables incremental data loading:

```yaml
endpoints:
  incremental_data:
    state:
      # Initialize from previous run, or default to 7 days ago
      last_sync_timestamp: >
        {
          coalesce(
            sync.last_sync_timestamp,
            date_format(date_add(now(), -7, 'day'), '%Y-%m-%dT%H:%M:%SZ')
          )
        }

    # Persist this variable for next run
    sync: [last_sync_timestamp]

    request:
      url: "{state.base_url}/data"
      parameters:
        updated_since: "{state.last_sync_timestamp}"

    response:
      processors:
        # Track the maximum timestamp seen
        - expression: "record.updated_at"
          output: "state.last_sync_timestamp"
          aggregation: maximum
```

**Key Differences:**

| Feature         | State                       | Sync                                            |
| --------------- | --------------------------- | ----------------------------------------------- |
| **Scope**       | Current run only            | Persisted between runs                          |
| **Purpose**     | Runtime variables           | Incremental tracking                            |
| **Declaration** | `state: {key: value}`       | `sync: [key]`                                   |
| **Access**      | `state.key`                 | `sync.key` (on load) → `state.key` (during run) |
| **Use Case**    | Configuration, calculations | Timestamps, cursors, offsets                    |

## Context Variables

Context variables are **read-only runtime values** passed from the replication configuration to the API spec. They enable endpoints to support both backfill and incremental modes with a single configuration.

**Available Context Variables:**

| Variable              | Type    | Description                                            | Set From                                                 |
| --------------------- | ------- | ------------------------------------------------------ | -------------------------------------------------------- |
| `context.mode`        | string  | Replication mode                                       | Replication config `mode` field                          |
| `context.store`       | map     | [Store](/concepts/hooks/store) values from replication | Replication `store` variable                             |
| `context.limit`       | integer | Maximum records to fetch                               | Replication config `source_options.limit`                |
| `context.range_start` | string  | Backfill range start                                   | Replication config `source_options.range` (first value)  |
| `context.range_end`   | string  | Backfill range end                                     | Replication config `source_options.range` (second value) |

**Context vs. State vs. Sync:**

| Feature        | Context            | State       | Sync              |
| -------------- | ------------------ | ----------- | ----------------- |
| **Source**     | Replication config | API spec    | Persisted storage |
| **Scope**      | Current run        | Current run | Between runs      |
| **Modifiable** | No (read-only)     | Yes         | Yes (via state)   |

**Common Pattern: Backfill with Incremental Fallback**

This pattern supports backfill (with range), incremental (with sync state), and first run (with default):

```yaml
endpoints:
  daily_events:
    sync: [last_date]  # Persist for incremental runs

    iterate:
      # Priority: context.range_start → sync.last_date → default
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events/daily/{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]
      processors:
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"
```

**Replication Configs:**

```yaml
# Backfill mode: Process specific date range
source_options:
  range: '2024-01-01,2024-01-31'  # Sets context.range_start and context.range_end

# Incremental mode: Use sync state (no range specified)
# Falls back to sync.last_date from previous run

# Testing mode: Limit records
source_options:
  limit: 100  # Sets context.limit
```

**Other Common Uses:**

```yaml
# Mode-specific behavior
state:
  batch_size: '{if(context.mode == "backfill", 1000, 100)}'

# Limit for testing/development
response:
  records:
    limit: '{coalesce(context.limit, null)}'

# Numeric ID ranges
iterate:
  over: >
    range(
      coalesce(context.range_start, sync.last_id, "1"),
      coalesce(context.range_end, "999999"),
      "1000"
    )
```

> 💡 **Best Practice:** Always use `coalesce()` with context variables to provide fallback values for when they're not set.

## Using Inputs

Inputs are custom configuration values passed from the connection definition to the API spec. Unlike secrets (which are for credentials), inputs are for non-sensitive options like field mappings, account IDs, or feature flags. Inputs are accessed via `{inputs.var_name}`, similar to `secrets` and `env`.

**Defining inputs in env.yaml:**

```yaml
# ~/.sling/env.yaml
connections:
  AIRTABLE:
    type: api
    spec: airtable
    secrets:
      api_key: "patXXXXXXXXXXXXXX"
    inputs:
      last_modified_field_map:
        'My Base Name':
          'My Table Name': 'Updated At'
        'Another Base':
          'Customers': 'Last Modified'
```

**Accessing inputs in your API spec:**

```yaml
# In your API spec
state:
  modified_field: >
    {
      jmespath(
        coalesce(inputs.last_modified_field_map, object()),
        "\"" + state.base_name + "\".\"" + state.table_name + "\""
      )
    }
```

**When to use inputs vs. secrets:**

| Use Case                     | Use `secrets` | Use `inputs` |
| ---------------------------- | ------------- | ------------ |
| API keys, tokens, passwords  | ✅             |              |
| Client IDs/secrets           | ✅             |              |
| Account IDs (non-sensitive)  |               | ✅            |
| Field name mappings          |               | ✅            |
| Feature flags                |               | ✅            |
| Custom configuration options |               | ✅            |

> 📝 **Note:** Inputs are defined by the API spec author. Check the specific API connector documentation to see what inputs are available.

## Queues

Queues allow you to pass data from one endpoint to another in a multi-step workflow. They are auto-detected from the endpoint definitions below — no top-level declaration needed.

```yaml
endpoints:
  list_orders:
    response:
      processors:
        - expression: "record.id"
          output: "queue.order_ids"

  get_order_details:
    iterate:
      over: "queue.order_ids"
      into: "state.current_order_id"
    request:
      url: "{state.base_url}/orders/{state.current_order_id}"
```

For detailed information on queues, see [Queues](/concepts/api-specs/queues).

## Sequence of Calls

A sequence is an ordered array of API calls that can be executed in workflows, authentication processes, and lifecycle hooks. Sequences are perfect for multi-step operations like async job workflows, custom authentication flows, or complex setup/teardown processes.

For detailed information on sequences, see [Sequences: Setup and Teardown](/concepts/api-specs/request#sequences-setup-and-teardown).

```yaml
if: <condition expression to evaluate before executing the call>
request: <request configuration map>
pagination: <pagination configuration map>
response: <response configuration map>
```

## Component Relationships

The following diagram shows how the major components relate to each other:

```mermaid
graph TD
    A[API Spec] --> B[Authentication]
    A --> C[Defaults]
    A --> D[Endpoints]
    A --> E[Queues]
    A --> F[Dynamic Endpoints]

    C --> C1[Default State]
    C --> C2[Default Request]
    C --> C3[Default Pagination]
    C --> C4[Default Response]

    D --> D1[Endpoint 1]
    D --> D2[Endpoint 2]
    D --> D3[Endpoint 3]

    D1 --> F1[State]
    D1 --> F2[Request]
    D1 --> F3[Pagination]
    D1 --> F4[Response]
    D1 --> F5[Iterate]
    D1 --> F6[Setup/Teardown]

    E -.-> F5
    F4 -.-> E

    classDef main fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
    classDef section fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
    classDef endpoint fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff

    class A main
    class B,C,D,E,F section
    class D1,D2,D3 endpoint
```

## Basic Example

Here's a minimal example showing the essential components:

```yaml
name: "GitHub API"
description: "API for accessing GitHub repositories and issues"

defaults:
  state:
    base_url: "https://api.github.com"
  request:
    headers:
      Accept: "application/vnd.github.v3+json"

endpoints:
  repos:
    description: "List repositories for a user"
    request:
      url: "{state.base_url}/users/{env.GITHUB_USERNAME}/repos"
    response:
      records:
        jmespath: "[*]"
```

## API Specification

Here we have the definitions for the accepted keys.

<table data-full-width="false"><thead><tr><th width="328">API Config Key</th><th>Description</th></tr></thead><tbody><tr><td><code>name</code></td><td>The display name of the API specification.</td></tr><tr><td><code>description</code></td><td>Brief description of what the API does.</td></tr><tr><td><code>queues</code></td><td>Array of queue names for passing data between endpoints.</td></tr><tr><td><code>defaults</code></td><td>Default endpoint configuration applied to all endpoints.</td></tr><tr><td><code>authentication</code></td><td>Authentication configuration for the API. See <a href="/pages/IzibikbkPh7c7IDdTHpW">Authentication</a> for details.</td></tr><tr><td><code>endpoints.&#x3C;key></code></td><td>Named endpoints that define API interactions.</td></tr><tr><td><code>dynamic_endpoints</code></td><td>Array of endpoint configurations for dynamic endpoint generation. See <a href="/pages/WOVuebqa6XPY5MFaMxfr">Dynamic Endpoints</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.name</code></p><p>or <code>defaults.name</code></p></td><td>The endpoint name (defaults to the key).</td></tr><tr><td><p><code>endpoints.&#x3C;key>.description</code></p><p>or <code>defaults.description</code></p></td><td>Description of what the endpoint does.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.docs</code></p><p>or <code>defaults.docs</code></p></td><td>URL to endpoint documentation.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.disabled</code></p><p>or <code>defaults.disabled</code></p></td><td>Whether the endpoint is disabled (default: false).</td></tr><tr><td><p><code>endpoints.&#x3C;key>.state</code></p><p>or <code>defaults.state</code></p></td><td>Map of state variables available to the endpoint. See State vs. Sync section above.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.sync</code></p><p>or <code>defaults.sync</code></p></td><td>Array of state variable names to persist between runs. See State vs. Sync section above.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.request</code></p><p>or <code>defaults.request</code></p></td><td>HTTP request configuration. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/requests.md">Requests</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.pagination</code></p><p>or <code>defaults.pagination</code></p></td><td>Pagination configuration. See <a href="/pages/bn3pd1smR9NevOI6eAAA#pagination">Pagination</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.response</code></p><p>or <code>defaults.response</code></p></td><td>Response processing configuration. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/response-processing.md">Response Processing</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.iterate</code></p><p>or <code>defaults.iterate</code></p></td><td>Iteration configuration for looping over data. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/requests.md#iteration-looping-requests">Iteration</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.setup</code></p><p>or <code>defaults.setup</code></p></td><td>Array of calls to execute before the main request. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/requests.md#sequences-setup-and-teardown">Sequences</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.teardown</code></p><p>or <code>defaults.teardown</code></p></td><td>Array of calls to execute after the main request. See <a href="https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/requests.md#sequences-setup-and-teardown">Sequences</a> for details.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.depends_on</code></p><p>or <code>defaults.depends_on</code></p></td><td>Array of endpoint names this endpoint depends on. See Endpoint Dependencies section above.</td></tr><tr><td><p><code>endpoints.&#x3C;key>.overrides</code></p><p>or <code>defaults.overrides</code></p></td><td>Stream processing overrides for destination writing. See Stream Overrides section above.</td></tr></tbody></table>

> 💡 **Tip:** Start with the basic example and gradually add complexity as needed. Use the defaults section to avoid repetition across endpoints.


# Authentication

This document covers the authentication methods for Sling API specifications. For the basic structure, see [Structure](/concepts/api-specs/structure).

Sling supports several authentication methods, configured under the `authentication` key.

> ⚠️ **Important:** It's best practice to use environment variables or secrets for sensitive values instead of hardcoding them.

## Managing Secrets and Environment Variables

Sling provides flexible ways to manage sensitive authentication data through environment variables and the `env.yaml` file. Here's how to configure them:

### Using the env.yaml File

The primary method for managing secrets is through the [`env.yaml`](/sling-cli/environment) file located at `~/.sling/env.yaml`. When defining API connections, you can specify secrets that will be available to your API specs.

```yaml
# ~/.sling/env.yaml
connections:
  my_api:
    type: api
    spec: file:///path/to/my_api.spec.yaml
    secrets:
      api_key: "your-secret-api-key"
      client_id: "your-oauth-client-id"
      client_secret: "your-oauth-client-secret"
      username: "your-username"
      password: "your-password"
      
  github_api:
    type: api
    spec: github
    secrets:
      token: "ghp_xxxxxxxxxxxxxxxxxxxx"
      
  stripe_api:
    type: api
    spec: stripe
    secrets:
      api_key: "sk_test_xxxxxxxxxxxxxxxxxxxx"
```

You'll be able to access the secrets values via the `{secrets.var_name}` expression.

```yaml
# In your API spec
defaults:
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"
```

### Using Environment Variables

You can also provide secrets via environment variables. In your API spec, use the `{env.VARIABLE_NAME}` syntax:

```bash
# Set environment variables
export API_USERNAME="myuser"
export API_PASSWORD="mypassword"
```

```yaml
# In your API spec
authentication:
  type: "basic"
  username: "{env.API_USERNAME}"
  password: "{env.API_PASSWORD}"
```

## Authentication Methods

Now that you understand how to manage secrets, here are the different authentication methods you can configure:

### No Authentication

```yaml
# Simply omit the authentication block for APIs that don't require authentication
```

### Static Header Authentication

For APIs that require headers for authentication (e.g., API keys, bearer tokens):

```yaml
authentication:
  type: "static"
  headers:
    Authorization: "Bearer {secrets.api_token}"
```

This is ideal for:

* API keys passed in headers (e.g., `X-API-Key: your-key`)
* Bearer tokens (e.g., `Authorization: Bearer your-token`)
* Custom authentication headers
* Multiple authentication headers

**Examples:**

```yaml
# API Key in custom header
authentication:
  type: "static"
  headers:
    X-API-Key: "{secrets.api_key}"

# Bearer token
# we can omit `type: static` and only specify `headers`
authentication:
  headers:
    Authorization: "Bearer {secrets.access_token}"

# Multiple headers
authentication:
  headers:
    Authorization: "Bearer {secrets.access_token}"
    X-API-Key: "{secrets.api_key}"
    X-Tenant-ID: "{secrets.tenant_id}"
```

**Shorthand Syntax:**

When only `headers` are provided without any other authentication configuration, the type defaults to `static`:

```yaml
# This is equivalent to type: "static"
authentication:
  headers:
    Authorization: "Bearer {secrets.api_token}"
```

> 📝 **Note:** For dynamic token retrieval (e.g., login flow), use the `sequence` authentication type instead.

### Basic Auth

```yaml
authentication:
  type: "basic"
  username: "{secrets.username}"
  password: "{secrets.password}"
```

### OAuth2 Authentication

For OAuth2 flows, use the `oauth2` type. Sling supports three OAuth2 flows:

| Flow                 | Use Case                             | Interactive             |
| -------------------- | ------------------------------------ | ----------------------- |
| `client_credentials` | Server-to-server, machine-to-machine | No                      |
| `authorization_code` | User-facing apps, browser-based auth | Yes                     |
| `device_code`        | CLI tools, headless environments     | Yes (on another device) |

**Full Property Reference:**

```yaml
authentication:
  type: "oauth2"
  flow: "client_credentials"  # client_credentials|authorization_code|device_code
  client_id: "{secrets.client_id}"
  client_secret: "{secrets.client_secret}"
  authentication_url: "https://api.example.com/oauth/token"      # Token endpoint (required)
  authorization_url: "https://api.example.com/oauth/authorize"   # Auth endpoint (for authorization_code)
  device_auth_url: "https://api.example.com/oauth/device/code"   # Device auth endpoint (for device_code)
  redirect_uri: "http://localhost:8080/callback"                 # Redirect URI (for authorization_code)
  scopes:
    - "read:data"
    - "write:data"
```

> **Token Persistence:** Sling automatically stores OAuth tokens in `~/.sling/api/tokens/{connection_name}.json` and refreshes them when they expire. You don't need to manage token refresh manually.

> **PKCE Support:** For public clients (when `client_secret` is empty), Sling automatically enables PKCE (Proof Key for Code Exchange) for added security.

**Client Credentials Flow (Server-to-Server):**

The most common flow for automated data pipelines. No user interaction required.

```yaml
authentication:
  type: "oauth2"
  flow: "client_credentials"
  client_id: "{secrets.client_id}"
  client_secret: "{secrets.client_secret}"
  authentication_url: "https://api.example.com/oauth/token"
  scopes:
    - "read:data"
```

**Authorization Code Flow (Browser-Based):**

For user-interactive authentication. Sling opens a browser for authorization and handles the callback automatically.

```yaml
authentication:
  type: "oauth2"
  flow: "authorization_code"
  client_id: "{secrets.client_id}"
  client_secret: "{secrets.client_secret}"
  authentication_url: "https://api.example.com/oauth/token"
  authorization_url: "https://api.example.com/oauth/authorize"
  scopes:
    - "read:data"
```

**Device Code Flow (Headless/CLI):**

For environments without a browser. Sling displays a URL and code for the user to enter on another device.

```yaml
authentication:
  type: "oauth2"
  flow: "device_code"
  client_id: "{secrets.client_id}"
  authentication_url: "https://api.example.com/oauth/token"
  device_auth_url: "https://api.example.com/oauth/device/code"
  scopes:
    - "read:data"
```

### AWS Signature V4 Authentication

For AWS services that require Signature V4 authentication (e.g., S3, AppSync, API Gateway):

```yaml
authentication:
  type: "aws-sigv4"
  aws_service: "execute-api"  # The AWS service name (e.g., s3, execute-api, appsync)
  aws_region: "{env.AWS_REGION}"
  aws_access_key_id: "{secrets.aws_access_key_id}"
  aws_secret_access_key: "{secrets.aws_secret_access_key}"
  aws_session_token: "{secrets.aws_session_token}"  # Optional, for temporary credentials
  aws_profile: "{env.AWS_PROFILE}"  # Optional, use AWS profile instead of explicit keys
```

> 📝 **Note:** For AWS authentication, you can use either explicit credentials (`aws_access_key_id`, `aws_secret_access_key`) or AWS profiles (`aws_profile`). The AWS SDK credential chain is also supported.

### HMAC Authentication

For APIs that require HMAC (Hash-based Message Authentication Code) request signing, such as cryptocurrency exchanges (Kraken, Binance) or custom enterprise APIs:

```yaml
authentication:
  type: "hmac"
  algorithm: "sha256"  # sha256 or sha512
  secret: "{secrets.api_secret}"
  secret_encoding: "hex"  # Optional: "hex", "base64", or "raw" (default)
  signing_string: "{http_method}{http_path}{unix_time}{http_body_sha256}"
  request_headers:
    X-Signature: "{signature}"
    X-Timestamp: "{unix_time}"
    X-API-Key: "{secrets.api_key}"
  nonce_length: 16  # Optional: generates random nonce (in bytes)
```

**How HMAC Works:**

1. A signing string is constructed from request components (method, path, timestamp, body hash, etc.)
2. The string is signed using HMAC-SHA256 or HMAC-SHA512 with your secret key
3. The signature and related headers are automatically added to each request

**Secret Encoding:**

Some APIs provide secrets in encoded formats (hex or base64) that must be decoded to raw bytes before signing. Use `secret_encoding` (*v1.5.6+*) to specify how to decode the secret:

* `"raw"` or `""` (default) - Use secret as-is (plain string)
* `"hex"` - Decode secret from hexadecimal string to bytes
* `"base64"` - Decode secret from base64 string to bytes

**Available Variables for Signing:**

* `http_method` - HTTP method (GET, POST, etc.)
* `http_path` - Request path with query parameters
* `http_query` - Canonical query string (sorted alphabetically by key)
* `http_headers` - Canonical headers (lowercase, sorted, newline-separated)
* `http_body_raw` - Raw request body as string
* `http_body_md5` - MD5 hash of request body (hex-encoded)
* `http_body_sha1` - SHA1 hash of request body (hex-encoded)
* `http_body_sha256` - SHA256 hash of request body (hex-encoded)
* `http_body_sha512` - SHA512 hash of request body (hex-encoded)
* `unix_time` - Unix timestamp in seconds
* `unix_time_ms` - Unix timestamp in milliseconds
* `date_iso` - ISO 8601 formatted date (e.g., "2023-10-31T15:30:32Z")
* `date_rfc1123` - RFC 1123 formatted date (e.g., "Tue, 31 Oct 2023 15:30:32 GMT")
* `nonce` - Random hex string (if `nonce_length` is set)
* `signature` - Computed HMAC signature (hex-encoded, only available in `request_headers`)

**Example: Kraken-style Authentication**

```yaml
authentication:
  type: "hmac"
  algorithm: "sha256"
  secret: "{secrets.api_secret}"
  signing_string: "{http_method}{http_path}{nonce}{http_body_sha256}"
  request_headers:
    API-Key: "{secrets.api_key}"
    API-Sign: "{signature}"
    API-Nonce: "{nonce}"
  nonce_length: 16
```

**Example: Hex-encoded Secret**

Some APIs provide secrets as hex strings that must be decoded before signing:

```yaml
authentication:
  type: "hmac"
  algorithm: "sha256"
  secret: "{secrets.api_secret}"
  secret_encoding: "hex"
  signing_string: "{http_body_raw}"
  request_headers:
    Content-Type: "application/json"
    X-API-Key: "{secrets.api_key}"
    X-API-Signature: "{signature}"
```

> 📝 **Note:** The `signing_string` and `request_headers` templates support all Sling template variables including `{secrets.*}`, `{env.*}`, and `{state.*}`.

### Sequence Authentication (Custom Workflows)

For APIs requiring a custom authentication sequence (e.g., multi-step login), use the `sequence` type. This allows defining a series of API calls to obtain authentication tokens or session data.

```yaml
authentication:
  type: "sequence"
  expires: 3600   # re-auth every 1 hr
  sequence:
    - request:
        url: "/login"
        method: POST
        payload:
          site_id: "{secrets.site_id}"
          user_id: "{secrets.user_id}"
          password: "{secrets.password}"
      response:
        processors:
          - expression: "response.json.token"
            output: "state.token"   # you can use this in you main request block
            aggregation: last
```

This performs a login request and extracts the token into `state.token` for use in subsequent requests. See [Requests & Responses](https://github.com/slingdata-io/sling-docs/blob/master/concepts/api/requests.md) for more on sequences.

## Endpoint-Level Authentication

By default, all endpoints use the authentication configured at the spec level. However, individual endpoints can override or disable authentication.

### Overriding Authentication

An endpoint can specify its own authentication that overrides the spec-level configuration:

```yaml
authentication:
  type: "basic"
  username: "{secrets.username}"
  password: "{secrets.password}"

endpoints:
  # Uses spec-level basic auth
  users:
    request:
      url: "{state.base_url}/users"

  # Uses its own OAuth2 auth instead
  admin_data:
    authentication:
      type: "oauth2"
      flow: "client_credentials"
      client_id: "{secrets.admin_client_id}"
      client_secret: "{secrets.admin_client_secret}"
      authentication_url: "https://api.example.com/oauth/token"
    request:
      url: "{state.base_url}/admin/data"
```

### Disabling Authentication

Some endpoints (like health checks or public data) may not require authentication. Set `authentication: null` to disable it:

```yaml
authentication:
  type: "oauth2"
  # ... OAuth config for most endpoints

endpoints:
  # No authentication needed for health check
  health:
    authentication: null
    request:
      url: "{state.base_url}/health"

  # Uses spec-level OAuth2
  protected_data:
    request:
      url: "{state.base_url}/data"
```

## Authentication Expiry

The `expires` property can be used with any authentication type to force re-authentication after a specified number of seconds. This is useful when tokens or sessions have a fixed lifetime.

```yaml
authentication:
  type: "basic"
  username: "{secrets.username}"
  password: "{secrets.password}"
  expires: 3600  # Re-authenticate every hour
```

When authentication expires, Sling automatically re-authenticates before the next request. This happens transparently without interrupting data extraction.

## Authentication Method Comparison

| Method      | Best For                          | Auto-Refresh       | Interactive     |
| ----------- | --------------------------------- | ------------------ | --------------- |
| `static`    | API keys, bearer tokens           | N/A                | No              |
| `basic`     | Username/password APIs            | N/A                | No              |
| `oauth2`    | Modern APIs, delegated auth       | Yes                | Depends on flow |
| `aws-sigv4` | AWS services                      | Yes                | No              |
| `hmac`      | Crypto exchanges, signed requests | N/A                | No              |
| `sequence`  | Custom auth workflows             | No (use `expires`) | No              |


# Requests & Iteration

This page details how to configure HTTP requests and use the iteration feature for looping within Sling API specifications.

## Request Configuration

Each endpoint defines its HTTP request details under the `request` key. These settings merge with and override the `defaults.request` configuration.

### Request Properties

| Property      | Required | Description                                           | Example                                        |
| ------------- | -------- | ----------------------------------------------------- | ---------------------------------------------- |
| `url`         | Yes      | Path (relative to `defaults.request.url`) or full URL | `"users"` or `"https://api.example.com/users"` |
| `method`      | No       | HTTP method (default: `"GET"`)                        | `"POST"`, `"PUT"`, `"PATCH"`, `"DELETE"`       |
| `headers`     | No       | HTTP headers to send                                  | `{"Content-Type": "application/json"}`         |
| `parameters`  | No       | Query parameters (or form fields for POST)            | `{"page": 1, "limit": 100}`                    |
| `payload`     | No       | Request body for POST/PUT/PATCH                       | `{"name": "New User"}`                         |
| `timeout`     | No       | Request timeout in seconds (default: 30)              | `60`                                           |
| `rate`        | No       | Max requests per second (default: 2)                  | `5`                                            |
| `concurrency` | No       | Max concurrent requests (default: 5)                  | `10`                                           |

### Example Request Configuration

```yaml
request:
  state:
    base_url: https://api.example.com/v1
    user_id: 123
  
  url: '{state.base_url}/users/{state.user_id}'
  
  # HTTP Method
  method: "POST"
  
  # Headers (merged with defaults.request.headers)
  headers:
    Content-Type: "application/json"
    Authorization: "Bearer {auth.token}"
    X-Request-ID: "{uuid()}"
    
  # Query Parameters (or form parameters for POST with application/x-www-form-urlencoded)
  parameters:
    active: true
    department: "{state.department}"
    
  # Request Body (for POST/PUT/PATCH methods)
  payload:
    user:
      name: "New User"
      email: "{state.user_email}"
      
  # Request timeout (in seconds)
  timeout: 60
  
  # Rate limiting (max requests per second)
  rate: 5
  
  # Concurrency (max in-flight requests)
  concurrency: 3
```

> 📝 **Note:** When `method` is `GET` or `DELETE`, the `parameters` are added as URL query parameters. When `method` is `POST`, `PUT`, or `PATCH` and `Content-Type` is `application/x-www-form-urlencoded`, the `parameters` are sent as form fields.

> 💡 **Tip:** Use `{...}` expressions to make request values dynamic based on state variables, authentication details, or environment variables.

## Iteration (Looping Requests)

The `iterate` section allows an endpoint to make multiple requests based on a list of items (e.g., IDs from a queue, date ranges).

### How Iteration Works

{% @mermaid/diagram content="graph TD
A\[Start Endpoint] --> B{Has iterate section?}
B -->|Yes| C\[Evaluate 'over' expression]
B -->|No| D\[Run single request with pagination]

```
C --> E[Create item list]
E --> F[For each item in parallel]
F --> G[Create iteration with own state]
G --> H[Set 'into' variable to current item]
H --> I[Run requests with pagination]

D --> J[Process response]
I --> J

style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style F fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style J fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Iteration Properties

| Property      | Required | Description                                     | Example                             |
| ------------- | -------- | ----------------------------------------------- | ----------------------------------- |
| `over`        | Yes      | Expression to get items to iterate over         | `"queue.user_ids"` or `"[1, 2, 3]"` |
| `into`        | Yes      | State variable to store current item            | `"state.current_id"`                |
| `concurrency` | No       | Max iterations to run in parallel (default: 10) | `5`                                 |
| `if`          | No       | Condition to evaluate before starting iteration | `"state.should_process == true"`    |

### Example: Basic Iteration

```yaml
iterate:
  # Get items from a queue filled by another endpoint
  over: "queue.product_ids"
  
  # Store current item in state variable
  into: "state.current_product_id"
  
  # Run up to 5 iterations concurrently
  concurrency: 5
```

> ⚠️ **Important:** Each iteration gets its own state. Changes to state variables in one iteration don't affect other iterations.

### Example: Date Range Iteration

This pattern is great for splitting large data extractions into daily chunks:

```yaml
endpoints:
  daily_reports:
    state:
      # Configure date range (last 7 days by default)
      start_date: '{coalesce(env.START_DATE, date_format(date_add(now(), -7, "day"), "%Y-%m-%d"))}'
      end_date: '{coalesce(env.END_DATE, date_format(now(), "%Y-%m-%d"))}'
    
    iterate:
      # Generate an array of dates between start_date and end_date, one day at a time
      over: >
        range(
          date_parse(state.start_date, "%Y-%m-%d"), 
          date_parse(state.end_date, "%Y-%m-%d"),
          "1d"
        )
      # Store the current date in state.current_day
      into: state.current_day
      concurrency: 10
    
    request:
      url: '{state.base_url}/reports'
      parameters:
        # Format the date for the API
        date: '{date_format(state.current_day, "%Y-%m-%d")}'
```

### Example: Batch Processing with `chunk()`

When an API accepts multiple IDs in a single request, use `chunk()` to process them in batches:

```yaml
endpoints:
  lookup_variants:
    iterate:
      # Split queue into batches of 50 IDs each
      over: "chunk(queue.variant_ids, 50)"
      # state.variant_id_batch will be an array of up to 50 IDs
      into: "state.variant_id_batch"
      concurrency: 5

    request:
      url: '{state.base_url}/variants/lookup'
      parameters:
        # Join IDs into comma-separated string
        ids: '{join(state.variant_id_batch, ",")}'
```

### Using Context Variables for Backfill Ranges

**Context variables** are runtime values passed from the replication configuration to the API spec. They're particularly useful for supporting both backfill and incremental modes with the same endpoint.

**Key context variables:**

* `context.range_start` - Start of backfill range (from `source_options.range`)
* `context.range_end` - End of backfill range (from `source_options.range`)

**Example: Date Range with Context Support**

```yaml
endpoints:
  events:
    # Persist last processed date for incremental runs
    sync: [last_date]

    iterate:
      # Backfill mode: Use context.range_start/range_end from config
      # Incremental mode: Use sync.last_date
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events"
      parameters:
        date: "{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]

      processors:
        # Track last processed date
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"
```

**Replication Config for Backfill:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    source_options:
      # Backfill specific date range
      range: '2024-01-01,2024-01-31'
```

**Replication Config for Incremental:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    # No range - uses sync.last_date for incremental
```

This pattern allows a single endpoint to handle both historical backfills and ongoing incremental updates seamlessly. See [Context Variables](/concepts/api-specs/structure#context-variables) for complete details.

## Request Flow Control

You can control the request flow using these features:

| Feature       | Location  | Purpose                   | Example                |
| ------------- | --------- | ------------------------- | ---------------------- |
| `rate`        | `request` | Limit requests per second | `rate: 5`              |
| `concurrency` | `request` | Limit parallel requests   | `concurrency: 3`       |
| `concurrency` | `iterate` | Limit parallel iterations | `concurrency: 10`      |
| `if`          | `iterate` | Conditional iteration     | `if: "state.has_data"` |
| `timeout`     | `request` | Set request timeout       | `timeout: 30`          |

> 💡 **Tip:** Balance performance and API limits:
>
> * Too low `concurrency`: slower extraction but gentler on the API
> * Too high `concurrency`: faster extraction but may trigger rate limits
> * `iterate.concurrency` controls parallelism across different IDs/items
> * `request.concurrency` controls parallelism across paginated requests

## Sequences: Setup and Teardown

Endpoints can define optional `setup` and `teardown` sequences that run before and after the main requests. These are multi-step workflows perfect for initialization, cleanup, or complex processes.

### What are Sequences?

A sequence is an ordered array of API calls. Each call in a sequence can:

* Make HTTP requests
* Process responses
* Update state variables
* Use pagination
* Execute conditionally

Sequences are used in three places:

1. **Authentication** - Custom authentication workflows
2. **Setup** - Pre-execution initialization
3. **Teardown** - Post-execution cleanup

### Sequence Call Structure

Each call in a sequence has the same structure:

```yaml
- if: <conditional expression>      # Optional: Execute only if true
  request: <request configuration>   # Same as endpoint request
  pagination: <pagination config>    # Optional: Paginate this call
  response: <response configuration> # Process response
```

### Setup Sequences

The `setup` sequence runs **once before** the endpoint's main requests/iterations begin. Perfect for:

* Fetching configuration data
* Initializing session data
* Validating prerequisites
* Loading dynamic parameters

#### Basic Setup Example

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
      method: GET
    response:
      processors:
        - expression: 'jmespath(response.json, "config.api_version")'
          output: "state.api_version"
          aggregation: last
        - expression: 'jmespath(response.json, "config.base_url")'
          output: "state.base_url"
          aggregation: last
        - expression: 'jmespath(response.json, "config.rate_limit")'
          output: "state.rate_limit"
          aggregation: last
```

#### Multi-Step Setup Example

```yaml
setup:
  # Step 1: Get session token
  - request:
      url: "{state.base_url}/session"
      method: POST
      payload:
        app_id: "{secrets.app_id}"
    response:
      processors:
        - expression: "response.json.session_token"
          output: "state.session_token"
          aggregation: last

  # Step 2: Use session token to get user info
  - request:
      url: "{state.base_url}/user/me"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: 'jmespath(response.json, "user.id")'
          output: "state.user_id"
          aggregation: last
        - expression: 'jmespath(response.json, "user.permissions")'
          output: "state.user_permissions"
          aggregation: last

  # Step 3: Conditionally load admin data
  - if: contains(state.user_permissions, "admin")
    request:
      url: "{state.base_url}/admin/settings"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.admin_config"
          output: "state.admin_config"
          aggregation: last
```

### Teardown Sequences

The `teardown` sequence runs **once after** all main requests complete. Perfect for:

* Closing sessions
* Cleaning up temporary resources
* Logging completion status
* Archiving processed data

#### Basic Teardown Example

```yaml
teardown:
  - request:
      url: "{state.base_url}/session/close"
      method: POST
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.status"
          output: "state.session_close_status"
          aggregation: last
```

#### Conditional Teardown Example

```yaml
teardown:
  # Only cleanup if we created temporary data
  - if: state.created_temp_data == true
    request:
      url: "{state.base_url}/cleanup"
      method: POST
      payload:
        session_id: "{state.session_id}"
        temp_ids: "{state.temp_resource_ids}"
    response:
      processors:
        - expression: 'jmespath(response.json, "cleanup.status")'
          output: "state.cleanup_status"
          aggregation: last

  # Always log final statistics
  - request:
      url: "{state.base_url}/analytics/log"
      method: POST
      payload:
        endpoint_name: "{env.ENDPOINT_NAME}"
        records_processed: "{state.total_records}"
        duration_seconds: "{state.duration}"
```

### Conditional Execution with `if`

Each call in a sequence can execute conditionally using the `if` field:

```yaml
setup:
  # Always runs
  - request:
      url: "{state.base_url}/status"
    response:
      processors:
        - expression: "response.json.api_status"
          output: "state.api_status"
          aggregation: last

  # Only runs if API is in maintenance mode
  - if: state.api_status == "maintenance"
    request:
      url: "{state.base_url}/maintenance/info"
    response:
      processors:
        - expression: "response.json.maintenance_until"
          output: "state.maintenance_until"
          aggregation: last
        - expression: log("API in maintenance until: " + state.maintenance_until)
          output: ""
```

**Common `if` Patterns:**

```yaml
# Check state variable existence
if: '!is_null(state.session_token)'

# Check for specific value
if: state.environment == "production"

# Check array membership
if: contains(state.enabled_features, "advanced_mode")

# Multiple conditions
if: state.user_type == "admin" && !is_null(state.admin_token)

# Check environment variable
if: env.ENABLE_FEATURE == "true"
```

### Pagination in Sequences

Sequence calls support pagination, useful for multi-page setup data:

```yaml
setup:
  # Load all available categories (paginated)
  - request:
      url: "{state.base_url}/categories"
      parameters:
        limit: 100

    pagination:
      next_state:
        offset: "{state.offset + 100}"
      stop_condition: "length(response.records) < 100"

    response:
      records:
        jmespath: "categories[]"
      processors:
        # Collect all category IDs
        - expression: "record.id"
          output: "queue.category_ids"
```

### State Management in Sequences

**Important behaviors:**

1. **State Isolation**: Setup/teardown sequences have their own state copy
2. **State Merging**: Changes to state persist back to the main endpoint
3. **Headers Inherited**: Request headers from the main endpoint are copied
4. **No Iteration State**: Sequences don't have iteration-specific state

```yaml
endpoints:
  my_endpoint:
    state:
      base_url: "https://api.example.com"
      initial_value: 100

    setup:
      - request:
          url: "{state.base_url}/init"
        response:
          processors:
            # This updates the endpoint's state
            - expression: "response.json.config_value"
              output: "state.config_value"
              aggregation: last

    request:
      # Can now use state.config_value from setup
      url: "{state.base_url}/data"
      parameters:
        config: "{state.config_value}"
```

### Complete Sequence Example

Here's a real-world example showing setup, main execution, and teardown:

```yaml
endpoints:
  export_data:
    state:
      base_url: "https://api.example.com/v2"

    setup:
      # Step 1: Request export job creation
      - request:
          url: "{state.base_url}/exports"
          method: POST
          payload:
            format: "csv"
            filters:
              date_from: "{state.start_date}"
              date_to: "{state.end_date}"
        response:
          processors:
            - expression: "response.json.export_id"
              output: "state.export_id"
              aggregation: last
            - expression: log("Created export job: " + response.json.export_id)
              output: ""

      # Step 2: Poll until export is ready
      - request:
          url: "{state.base_url}/exports/{state.export_id}/status"

        pagination:
          next_state:
            poll_count: "{coalesce(state.poll_count, 0) + 1}"
          stop_condition: >
            jmespath(response.json, "status") == "completed" ||
            jmespath(response.json, "status") == "failed" ||
            state.poll_count >= 60

        response:
          processors:
            - expression: 'jmespath(response.json, "status")'
              output: "state.export_status"
              aggregation: last
            - if: 'jmespath(response.json, "status") == "processing"'
              expression: log("Export still processing, poll " + string(state.poll_count))
              output: ""

      # Step 3: Get download URL (only if completed)
      - if: state.export_status == "completed"
        request:
          url: "{state.base_url}/exports/{state.export_id}"
        response:
          processors:
            - expression: 'jmespath(response.json, "download_url")'
              output: "state.download_url"
              aggregation: last

    # Main request downloads the export file
    request:
      url: "{state.download_url}"

    response:
      format: csv
      records:
        jmespath: "[*]"

    teardown:
      # Clean up the export job
      - request:
          url: "{state.base_url}/exports/{state.export_id}"
          method: DELETE
        response:
          processors:
            - expression: log("Deleted export job: " + state.export_id)
              output: ""
```

### Sequence vs. Main Request

| Feature                   | Sequences (Setup/Teardown) | Main Request                 |
| ------------------------- | -------------------------- | ---------------------------- |
| **When Executes**         | Before/after main requests | During endpoint execution    |
| **Supports Iteration**    | No                         | Yes (via `iterate`)          |
| **Supports Pagination**   | Yes (per call)             | Yes                          |
| **State Scope**           | Shared with endpoint       | Per-iteration (if iterating) |
| **Output Records**        | No (state only)            | Yes (to destination)         |
| **Conditional Execution** | Yes (via `if` per call)    | Yes (via `iterate.if`)       |

### Best Practices for Sequences

#### 1. Use Logging for Visibility

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: "response.json.config"
          output: "state.config"
          aggregation: last
        # Log what we loaded
        - expression: log("Loaded config: " + string(state.config))
          output: ""
```

#### 2. Handle Errors Gracefully

```yaml
setup:
  - request:
      url: "{state.base_url}/optional-config"
    response:
      rules:
        # Don't fail if optional config is missing
        - action: continue
          condition: "response.status == 404"
          message: "Optional config not found, using defaults"
```

#### 3. Keep Sequences Focused

```yaml
# Good: Each step has a clear purpose
setup:
  - request: # Get auth token
  - request: # Get user info
  - request: # Get feature flags

# Avoid: Mixing unrelated concerns
setup:
  - request: # Get auth token
  - request: # Process data (should be main request)
  - request: # Generate report (should be teardown)
```

#### 4. Use Conditional Steps Wisely

```yaml
setup:
  # Always get config
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: 'jmespath(response.json, "features")'
          output: "state.features"
          aggregation: last

  # Only load beta features if enabled
  - if: contains(state.features, "beta")
    request:
      url: "{state.base_url}/beta/features"
```

> 📝 **Note:** Setup/Teardown sequences use the same structure as authentication sequences. State changes persist to the main endpoint, making them perfect for initialization and cleanup tasks.

> 💡 **Tip:** Use `log()` function liberally in sequences during development to understand the execution flow and debug issues.


# Response Processing

This document explains how Sling processes API responses, including format handling, record extraction, and data transformations.

## Response Flow Overview

```mermaid
graph TD
    A[HTTP Response] --> B{Detect Format}
    B -->|JSON| C[Parse JSON]
    B -->|CSV| D[Parse CSV to Records]
    B -->|XML| E[Parse XML to JSON]
    B -->|Auto-detect| F[Check Content-Type Header]

    C --> G[Apply JMESPath / jq]
    D --> G
    E --> G

    G --> H[Extract Records Array]
    H --> I[Deduplicate Records]
    I --> J[Apply Processors]
    J --> K[Output to Destination]

    style A fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style B fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
    style G fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style J fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style K fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff
```

## Response Formats

Sling can automatically handle multiple response formats based on the API's `Content-Type` header or explicit configuration.

### Automatic Format Detection

By default, Sling detects the format from the `Content-Type` response header:

| Content-Type Header             | Format Detected | Processing                  |
| ------------------------------- | --------------- | --------------------------- |
| `application/json`              | JSON            | Direct JSON parsing         |
| `application/xml` or `text/xml` | XML             | Converted to JSON structure |
| `text/csv`                      | CSV             | Converted to JSON records   |
| Others                          | JSON (default)  | Attempts JSON parsing       |

### Explicit Format Configuration

You can override automatic detection by specifying the format explicitly:

```yaml
response:
  format: json  # Force format interpretation
  records:
    jmespath: "data[]"
```

Supported format values:

* `json` - Standard JSON response
* `csv` - Comma-separated values
* `xml` - XML response
* `jsonlines` - JSON Lines (one JSON object per line)

## Format-Specific Processing

### JSON Responses

The most common API response format. Sling parses JSON and extracts records using JMESPath or jq:

```yaml
response:
  format: json  # Optional, auto-detected
  records:
    # Extract the array of user objects
    jmespath: "data.users[]"
    # or use jq syntax:
    # jq: ".data.users[]"
```

Example JSON response:

```json
{
  "data": {
    "users": [
      {"id": 1, "name": "Alice"},
      {"id": 2, "name": "Bob"}
    ]
  },
  "meta": {
    "total": 2
  }
}
```

### CSV Responses

CSV responses are automatically converted to JSON records:

```yaml
response:
  format: csv
  records:
    # For CSV, jmespath typically extracts all records
    jmespath: "[*]"
    primary_key: ["id"]
```

**CSV Processing Rules:**

1. First row is treated as the header row (column names)
2. Subsequent rows become records
3. Minimum 2 rows required (header + at least one data row)
4. Each row is converted to a JSON object with header names as keys

Example CSV response:

```csv
id,name,email
1,Alice,alice@example.com
2,Bob,bob@example.com
```

Becomes:

```json
[
  {"id": "1", "name": "Alice", "email": "alice@example.com"},
  {"id": "2", "name": "Bob", "email": "bob@example.com"}
]
```

> 📝 **Note:** CSV values are always strings. Use [processors](/concepts/api-specs/advanced#data-processors) to convert them to other types if needed.

### XML Responses

XML responses are automatically converted to JSON before record extraction:

```yaml
response:
  format: xml
  records:
    jmespath: "root.users.user[]"
```

Example XML response:

```xml
<root>
  <users>
    <user>
      <id>1</id>
      <name>Alice</name>
    </user>
    <user>
      <id>2</id>
      <name>Bob</name>
    </user>
  </users>
</root>
```

Becomes JSON:

```json
{
  "root": {
    "users": {
      "user": [
        {"id": "1", "name": "Alice"},
        {"id": "2", "name": "Bob"}
      ]
    }
  }
}
```

> ⚠️ **Warning:** XML to JSON conversion follows standard rules: attributes become fields with `@` prefix, text content becomes `#text` field.

### JSON Lines (JSONL)

For streaming JSON responses where each line is a complete JSON object:

```yaml
response:
  format: jsonlines
  records:
    # Each line is already a record
    jmespath: "[*]"
```

Example JSONL response:

```jsonl
{"id": 1, "name": "Alice", "email": "alice@example.com"}
{"id": 2, "name": "Bob", "email": "bob@example.com"}
{"id": 3, "name": "Charlie", "email": "charlie@example.com"}
```

## Record Extraction

After format conversion, records are extracted using JMESPath or jq expressions.

### Basic Extraction

```yaml
response:
  records:
    # Extract top-level array
    jmespath: "[*]"
```

### Nested Extraction

```yaml
response:
  records:
    # Extract nested array
    jmespath: "response.data.items[]"
```

### Conditional Extraction

```yaml
response:
  records:
    # Extract only active users (JMESPath)
    jmespath: "users[?status=='active']"
    # or with jq:
    # jq: '[.users[] | select(.status == "active")]'
```

### Projection and Transformation

```yaml
response:
  records:
    # Extract and reshape data
    jmespath: "data[].{user_id: id, full_name: name, contact: email}"
```

## Deduplication

When `primary_key` is defined, Sling automatically deduplicates records:

```yaml
response:
  records:
    jmespath: "data[]"
    primary_key: ["id"]  # Single field
    # OR
    # primary_key: ["id", "location_id"]  # Composite key
```

### Deduplication Strategies

#### 1. In-Memory Deduplication (Default)

For datasets with reasonable record counts:

```yaml
response:
  records:
    primary_key: ["id"]
    # Uses hash map in memory
```

**Characteristics:**

* Fast and accurate
* Memory usage grows with unique record count
* Suitable for datasets up to \~1 million records

#### 2. Bloom Filter Deduplication

For very large datasets where memory is constrained:

```yaml
response:
  records:
    primary_key: ["id"]
    duplicate_tolerance: "10000000,0.001"  # capacity,error_rate
```

**Characteristics:**

* Probabilistic deduplication (small false positive rate)
* Fixed memory footprint
* Suitable for datasets with millions of records

**Format:** `"capacity,error_rate"`

* `capacity`: Expected number of unique records
* `error_rate`: Acceptable false positive rate (e.g., 0.001 = 0.1%)

> 💡 **Tip:** Use Bloom filter for datasets over 1 million records or when memory is limited. The error rate determines memory usage - lower rates use more memory.

## Response State

All response data is accessible in the `response` state variable for use in expressions:

| Response Property  | Description             | Example Usage               |
| ------------------ | ----------------------- | --------------------------- |
| `response.status`  | HTTP status code        | `response.status == 200`    |
| `response.headers` | Response headers        | `response.headers.link`     |
| `response.text`    | Raw response body       | `length(response.text) > 0` |
| `response.json`    | Parsed JSON response    | `response.json.has_more`    |
| `response.records` | Extracted records array | `length(response.records)`  |

### Using Response State in Pagination

```yaml
pagination:
  next_state:
    cursor: '{jmespath(response.json, "pagination.next_cursor")}'
  stop_condition: 'jmespath(response.json, "has_more") == false'
```

### Using Response State in Rules

```yaml
rules:
  - action: retry
    condition: "response.status == 429"
    max_attempts: 5

  - action: stop
    condition: "length(response.records) == 0"
    message: "No more records available"
```

### Using Response State in [Processors](/concepts/api-specs/advanced#data-processors)

```yaml
processors:
  # Add metadata from response to each record
  - expression: "response.json.request_id"
    output: "record.api_request_id"

  # Conditional processing based on response
  - if: "response.status == 206"  # Partial content
    expression: "record.id"
    output: "queue.incomplete_records"
```

## Conditional Processing with IF Conditions

Processors support an optional `if` field to conditionally execute based on runtime conditions.

### Basic Syntax

```yaml
processors:
  # Only process non-null values
  - expression: "lower(record.email)"
    if: "!is_null(record.email) && record.email != ''"
    output: "record.email_normalized"

  # Only queue US customers
  - expression: "record.id"
    if: "record.country == 'US'"
    output: "queue.us_customer_ids"

  # Track max timestamp only for completed records
  - expression: "record.updated_at"
    if: "record.status == 'completed'"
    output: "state.last_completed_timestamp"
    aggregation: "maximum"
```

### How It Works

* **Evaluation**: The `if` condition is evaluated **before** the expression
* **Skip on False**: If false, the entire processor is skipped for that record
* **Access**: Has access to `record`, `state`, `response`, `env`, `secrets`

### Common Patterns

```yaml
processors:
  # Null/empty checks
  - expression: 'cast(record.age, "int")'
    if: "!is_null(record.age)"
    output: "record.age_int"

  # Type validation with try_cast
  - expression: 'cast(record.value, "int")'
    if: "is_null(try_cast(record.value, 'int')) == false"
    output: "record.value_int"

  # Date filtering
  - expression: "record.id"
    if: "date_parse(record.created_at, 'auto') > date_add(now(), -7, 'day')"
    output: "queue.recent_ids"

  # Response-based conditions
  - expression: "response.json.request_id"
    if: "response.status == 200"
    output: "record.api_request_id"
```

> 💡 **Tip:** Always check for null before accessing field properties to avoid errors.

> ⚠️ **Warning:** IF conditions are evaluated for every record. Avoid expensive operations.

## Overwriting Records with `output: "record"`

Setting `output: "record"` completely replaces the entire record with the result of the expression. All existing fields are discarded unless explicitly included.

### Common Use Cases

**1. Select Specific Fields**

Keep only essential fields from large API responses:

```yaml
processors:
  - expression: >
      object(
        "user_id", record.id,
        "username", record.username,
        "email", record.email
      )
    output: "record"
```

**2. Rename Fields**

Transform field names to match your schema:

```yaml
processors:
  - expression: >
      object(
        "customer_id", record.id,
        "full_name", record.name,
        "contact_email", record.email
      )
    output: "record"
```

**3. Flatten Nested Data**

Convert nested structures into flat records using JMESPath or jq:

```yaml
processors:
  # Using JMESPath:
  - expression: >
      jmespath(record, "{
        id: id,
        name: user.profile.name,
        email: user.contact.email,
        country: user.address.country,
        plan_type: subscription.plan.type
      }")
    output: "record"

  # Or using jq:
  - expression: >
      jq(record, "{id, name: .user.profile.name, email: .user.contact.email, country: .user.address.country, plan_type: .subscription.plan.type}")
    output: "record"
```

**4. Add Computed Fields**

Create records with derived values:

```yaml
processors:
  - expression: >
      object(
        "order_id", record.id,
        "subtotal", record.subtotal,
        "tax", record.subtotal * 0.08,
        "total", record.subtotal * 1.08
      )
    output: "record"
```

### Important Warnings

⚠️ **All previous fields are discarded** - Must explicitly include every field you want to keep

⚠️ **Order matters** - If you overwrite the record, then add fields afterward:

```yaml
processors:
  # First: Overwrite to simplify
  - expression: 'object("id", record.id, "name", record.name)'
    output: "record"

  # Then: Add new fields to simplified record
  - expression: "upper(record.name)"
    output: "record.name_upper"
```

⚠️ **Include primary keys** - For deduplication to work, primary key fields must be in the new record

> 💡 **Tip:** Use JMESPath projection syntax for cleaner nested data transformations.

## Error Handling

### Invalid Response Format

When Sling cannot parse the response in the expected format:

```yaml
rules:
  - action: fail
    condition: "response.status >= 400"
    message: "API returned error: {response.status}"
```

### Empty or Missing Records

Handle cases where no records are found:

```yaml
pagination:
  # Stop if no records returned
  stop_condition: "length(response.records) == 0"
```

### Partial Responses

Some APIs return partial data on errors:

```yaml
rules:
  # Continue processing partial results
  - action: continue
    condition: "response.status == 206"
    message: "Partial content received, processing available data"
```

## Complete Example

Here's a comprehensive example showing all response processing features:

```yaml
endpoints:
  user_activity:
    request:
      url: "{state.base_url}/users/activity"
      parameters:
        limit: 100

    response:
      # Explicitly set format (usually auto-detected)
      format: json

      records:
        # Extract nested records
        jmespath: "data.activities[]"

        # Deduplicate by composite key
        primary_key: ["user_id", "activity_id"]

        # Limit total records for testing
        limit: 5000

        # Use Bloom filter for large datasets
        duplicate_tolerance: "1000000,0.001"

      processors:
        # Transform timestamp field
        - expression: 'date_parse(record.timestamp, "auto")'
          output: "record.activity_date"

        # Add response metadata
        - expression: "response.json.request_id"
          output: "record.api_request_id"

        # Track max timestamp for incremental sync
        - expression: "record.timestamp"
          output: "state.last_activity_timestamp"
          aggregation: maximum

        # Send user IDs to queue for detail lookup
        - expression: "record.user_id"
          output: "queue.user_ids"

      rules:
        # Retry on rate limit
        - action: retry
          condition: "response.status == 429"
          max_attempts: 5
          backoff: exponential

        # Continue on not found (user may have been deleted)
        - action: continue
          condition: "response.status == 404"
          message: "Resource not found, continuing"

        # Fail on auth errors
        - action: fail
          condition: "response.status == 401 || response.status == 403"
          message: "Authentication failed"

    pagination:
      next_state:
        cursor: '{jmespath(response.json, "pagination.next_cursor")}'
      stop_condition: 'is_null(jmespath(response.json, "pagination.next_cursor")) || length(response.records) == 0'
```

## Best Practices

### 1. Always Define Primary Keys

Even if the API doesn't explicitly require deduplication, defining primary keys helps ensure data quality:

```yaml
response:
  records:
    primary_key: ["id"]  # Prevents accidental duplicates
```

### 2. Use Appropriate Deduplication

Choose the right strategy based on your dataset size:

```yaml
# For < 1M records (default)
primary_key: ["id"]

# For > 1M records
primary_key: ["id"]
duplicate_tolerance: "10000000,0.001"
```

### 3. Handle Multiple Content Types

If your API might return different formats:

```yaml
rules:
  # Handle JSON errors
  - action: fail
    condition: 'response.status >= 400 && response.headers["content-type"] == "application/json"'
    message: "API error: {response.json.error}"

  # Handle HTML errors (often 500 errors)
  - action: fail
    condition: 'response.status >= 400 && jmespath(response.headers, "\"content-type\"") == "text/html"'
    message: "Server error (HTML response)"
```

### 4. Validate Records Structure

Use processors to validate critical fields:

```yaml
processors:
  # Ensure required field exists
  - expression: 'require(record.id, "Record missing required id field")'
    output: "record.id_validated"
```

### 5. Log Response Details for Debugging

During development, use processors to log response information:

```yaml
processors:
  # Log response summary
  - expression: >
      log("Response status: " + string(response.status) +
          ", Records: " + string(length(response.records)))
    output: ""  # Empty output means don't store anywhere
```

## Troubleshooting

### No Records Extracted

If you're not getting any records:

1. Check your JMESPath or jq expression:

```bash
sling conns test API_NAME --endpoints ENDPOINT_NAME --trace
```

2. Look at the raw response in trace output
3. Verify the path to your records array
4. Test expressions using online tools: [jmespath.org](https://jmespath.org/) for JMESPath, [jqplay.org](https://jqplay.org/) for jq

### CSV Parsing Errors

Common CSV issues:

```yaml
# Error: "need at least 2 lines to build records from csv"
# Solution: Ensure API returns header + at least one data row
```

### Deduplication Not Working

Verify your primary key fields exist:

```yaml
processors:
  # Log primary key values
  - if: "!is_null(record.id)"
    expression: 'log("Found ID: " + string(record.id))'
    output: ""
```

> 💡 **Tip:** Use `--trace` flag to see detailed response processing including format detection, record extraction, and deduplication results.


# Advanced Features

This document covers advanced capabilities within Sling API specifications: pagination strategies, expression functions, incremental sync, and rules for error handling with retry logic.

For response processing basics (format handling, record extraction, deduplication, processors), see [Response Processing](/concepts/api-specs/response).

## Content Overview

* [Pagination](#pagination)
* [Functions](#functions)
* [Sync State for Incremental Loads](#sync-state-for-incremental-loads)
* [Rules & Retries](#rules-and-retries)

## Pagination

Pagination controls how Sling navigates through multiple pages of results for *each iteration* (if `iterate` is used) or for the single endpoint execution (if `iterate` is not used).

### Pagination Flow

{% @mermaid/diagram content="graph TD
A\[Make Request] --> B\[Process Response]
B --> C{Check stop\_condition}
C -->|True| D\[Stop Pagination]
C -->|False| E\[Evaluate next\_state]
E --> F\[Update state variables]
F --> A

```
style A fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style D fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff
style F fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Common Pagination Patterns

#### 1. Cursor-based Pagination

Uses the ID of the last record to fetch the next page.

```yaml
pagination:
  next_state:
    # Use ID of last record for next request
    starting_after: "{response.records[-1].id}"
  stop_condition: 'jmespath(response.json, "has_more") == false || length(response.records) == 0'
```

#### 2. Page Number Pagination

Increments a page number for each request.

```yaml
pagination:
  next_state:
    # Increment page number
    page: "{state.page + 1}"
  stop_condition: 'state.page >= jmespath(response.json, "total_pages") || length(response.records) == 0'
```

#### 3. Offset Pagination

Increments an offset value based on records received.

```yaml
pagination:
  next_state:
    # Increase offset by limit
    offset: "{state.offset + state.limit}"
  stop_condition: "length(response.records) < state.limit"
```

#### 4. Link Header Pagination

Extracts the next page URL from the response headers.

```yaml
pagination:
  next_state:
    # Extract URL from Link header
    url: >
      {
        if(
          contains(response.headers.link, "rel=\"next\""),
          trim(split_part(split(response.headers.link, ",")[0], ";", 0), "<>"),
          null
        )
      }
  stop_condition: '!contains(response.headers.link, "rel=\"next\"")'
```

> 💡 **Tip:** For better performance, avoid using `response` variables in `next_state` expressions when possible. This allows Sling to prepare the next request before the current one finishes, increasing parallelism.

## Functions

Functions are the building blocks of dynamic expressions in Sling API specifications. They enable sophisticated data transformations, validations, and manipulations within your API configurations.

### Using Functions

Functions can be used throughout your API specification wherever expressions are supported, including:

* **Request Configuration**: Dynamic URLs, headers, parameters, and payloads
* **Response Processing**: Data transformation and extraction
* **Pagination Logic**: Computing next page parameters
* **Conditional Logic**: Rules, iteration conditions, and stop conditions
* **State Management**: Transforming and aggregating state variables

### Common Function Patterns

#### Dynamic Request Construction

```yaml
request:
  url: '{state.base_url}/users/{state.user_id}'
  headers:
    Authorization: "Bearer {auth.token}"
    X-Request-ID: "{uuid()}"
  parameters:
    updated_since: '{date_format(date_add(now(), -1, "day"), "%Y-%m-%dT%H:%M:%SZ")}'
    limit: '{coalesce(state.page_size, 100)}'
```

#### Data Transformation in Processors

```yaml
processors:
  # Parse and format timestamps
  - expression: 'date_parse(record.created_at, "auto")'
    output: "record.created_timestamp"
  
  # Clean and validate data
  - expression: "trim(upper(record.status))"
    output: "record.status_clean"
  
  # Extract nested values
  - expression: 'get_path(record, "user.profile.email")'
    output: "record.user_email"
  
  # Conditional processing
  - expression: 'if(record.active, "ACTIVE", "INACTIVE")'
    output: "record.status_label"
```

#### Pagination with Functions

```yaml
pagination:
  next_state:
    # Extract cursor from response
    cursor: "get_path(response.json, 'pagination.next_cursor')"
    
    # Increment page number
    page: "{coalesce(state.page, 0) + 1}"
    
    # Dynamic limit based on response size
    limit: "{if(length(response.records) < 100, 50, state.limit)}"

  # jmespath: is_null(jmespath(response.json, "pagination.next_cursor"))
  # jq:      is_null(jq(response.json, ".pagination.next_cursor")[0])
  stop_condition: 'is_null(jmespath(response.json, "pagination.next_cursor")) || length(response.records) == 0'
```

#### Advanced Data Processing

```yaml
processors:
  # Filter and transform arrays
  - expression: 'filter(record.tags, "length(value) > 0")'
    output: "record.valid_tags"
  
  # Extract specific fields using jq
  - expression: 'jq(record, "[.items[] | select(.price > 100) | {name, price}]")[0]'
    output: "record.expensive_items"
  
  # Hash sensitive data
  - expression: hash(record.email, "sha256")
    output: "record.email_hash"
  
  # Generate derived fields
  - expression: join([record.first_name, record.last_name], " ")
    output: "record.full_name"
```

### Function Error Handling

Functions can help with graceful error handling and fallback values:

```yaml
processors:
  # Safe parsing with fallback
  - expression: coalesce(try_cast(record.age, "int"), 0)
    output: "record.age_int"
  
  # Required field validation
  - expression: require(record.user_id, "User ID is required")
    output: "record.validated_user_id"
  
  # Conditional field access
  - expression: if(is_null(record.metadata), "", get_path(record.metadata, "source"))
    output: "record.source"
```

### Best Practices

1. **Use `coalesce()` for Defaults**: Always provide fallback values for optional fields
2. **Validate Required Fields**: Use `require()` to ensure critical data is present
3. **Handle Date Formats**: Use `date_parse()` with "auto" format when possible
4. **Escape Special Characters**: Use encoding functions for URLs and other special contexts
5. **Test Complex Expressions**: Break down complex expressions into smaller, testable parts

For a complete reference of all available functions, see the [Functions documentation](/concepts/functions).

> 💡 **Tip**: Use the `log()` function during development to debug complex expressions: `log("Processing record: " + record.id)`

> ⚠️ **Warning**: Functions are evaluated for each record or iteration. Avoid expensive operations in frequently-called expressions.

## Sync State for Incremental Loads

The `sync` key allows persisting state variables between runs, enabling incremental data loading (fetching only new or updated data).

### Incremental Sync Workflow

{% @mermaid/diagram content="sequenceDiagram
participant Previous as Previous Run
participant Current as Current Run
participant API as API
participant Next as Next Run

```
Note over Previous: Stores state.last_sync_ts
Previous->>Current: sync.last_sync_ts

Note over Current: Init state variables
Current->>Current: state.start_timestamp = sync.last_sync_ts or default

Current->>API: Request with updated_since=state.start_timestamp
API->>Current: Response with records

Note over Current: Process & track max timestamp
Current->>Current: Find max of record.updated_at → state.last_sync_ts

Current->>Next: Persist state.last_sync_ts as sync.last_sync_ts" %}
```

### Example: Timestamp-Based Incremental Sync

```yaml
endpoints:
  incremental_data:
    state:
      # Get previous timestamp or default to 7 days ago
      start_timestamp: >
        {
          coalesce(
            sync.last_sync_ts,
            date_format(date_add(now(), -7, 'day'), '%Y-%m-%dT%H:%M:%SZ')
          )
        }
      
      # Initialize tracking variable with start timestamp
      last_sync_ts: '{state.start_timestamp}'

    # List of state variables to persist for next run
    sync: [last_sync_ts]

    request:
      parameters:
        # Filter by timestamp from last run
        updated_since: '{state.start_timestamp}'

    response:
      processors:
        # Track maximum timestamp seen
        - expression: "record.updated_at"
          output: "state.last_sync_ts"
          aggregation: "maximum"
```

> 💡 **Tip:** Always use `coalesce()` with sync variables to handle the first run when no previous state exists.

### Combining Incremental Sync with Context Variables

For advanced scenarios, you can combine sync state with **context variables** to support both incremental loading and backfilling. Context variables are runtime values passed from the replication configuration.

**Key context variables for incremental sync:**

* `context.range_start` - Start of backfill range (from `source_options.range`)
* `context.range_end` - End of backfill range (from `source_options.range`)
* `context.mode` - Replication mode (`incremental`, `full-refresh`, `backfill`)
* `context.limit` - Maximum records to fetch (from `source_options.limit`)

**Example: Incremental with Backfill Support**

```yaml
endpoints:
  events:
    sync: [last_date]

    iterate:
      # Backfill mode: Use context.range_start/range_end
      # Incremental mode: Use sync.last_date
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events"
      parameters:
        date: "{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]

      processors:
        # Track last processed date for next incremental run
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"
```

**Backfill Usage:**

```yaml
# replication.yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    source_options:
      # Backfill January 2024
      range: '2024-01-01,2024-01-31'
```

**Incremental Usage:**

```yaml
# replication.yaml (without range)
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    # No range - uses sync.last_date
```

This pattern allows the same endpoint to handle both historical backfills and ongoing incremental updates. See [Context Variables](/concepts/api-specs/structure#context-variables) for full details.

## Rules & Retries

Rules define actions based on response conditions (status codes, headers, body content), providing fine-grained control over error handling and retries.

### Rules Evaluation Flow

{% @mermaid/diagram content="graph TD
A\[Receive Response] --> B\[Evaluate Rules in Order]
B --> C{Rule Condition Match?}
C -->|No| D\[Try Next Rule]
D --> C
C -->|Yes| E{What Action?}
E -->|retry| F\[Wait Based on Backoff]
F --> G\[Retry Request]
E -->|continue| H\[Process Response]
E -->|skip| I\[Skip This Request]
E -->|break| J\[Stop Iteration Gracefully]
E -->|stop| K\[Stop This Endpoint]
E -->|fail| L\[Fail with Error]

```
style A fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style E fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style F fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style H fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style J fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff
style K fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff
style L fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Rule Properties

| Property       | Required       | Description                                | Example                                                          |
| -------------- | -------------- | ------------------------------------------ | ---------------------------------------------------------------- |
| `action`       | Yes            | Action to take when condition is true      | `"retry"`, `"continue"`, `"stop"`, `"break"`, `"skip"`, `"fail"` |
| `condition`    | Yes            | Expression that triggers the action        | `"response.status == 429"`                                       |
| `max_attempts` | No (for retry) | Max number of retry attempts               | `5` (default: 3)                                                 |
| `backoff`      | No (for retry) | Strategy for delay between retries         | `"exponential"`, `"linear"`, `"constant"`, `"jitter"`, `"none"`  |
| `backoff_base` | No (for retry) | Initial delay in seconds                   | `2` (default: 1)                                                 |
| `message`      | No             | Message for logging (supports expressions) | `"Rate limit hit, retrying..."`                                  |

### Rule Actions

| Action     | Description                                                 | Use Case                                               |
| ---------- | ----------------------------------------------------------- | ------------------------------------------------------ |
| `retry`    | Retry the request after delay                               | Rate limits (429), server errors (>=500)               |
| `continue` | Process response, ignore error                              | Non-critical errors (e.g., 404 for optional resources) |
| `skip`     | Break out of the rule evaluation loop and skip this request | When a request should not be processed                 |
| `break`    | Stop the current iteration gracefully without error         | Stop iteration within loops when processing complete   |
| `stop`     | Stop current endpoint/iteration                             | When further requests would be useless                 |
| `fail`     | Stop Sling run with error                                   | Critical errors (auth failure, invalid parameters)     |

### Example Rules

```yaml
rules:
  # Rule 1: Retry rate limits and server errors
  - action: "retry"
    condition: "response.status == 429 || response.status >= 500"
    max_attempts: 5
    backoff: "exponential"
    backoff_base: 2
    message: "Server error or rate limit hit, retrying..."

  # Rule 2: Fail on authentication errors
  - action: "fail"
    condition: "response.status == 401 || response.status == 403"
    message: "Authentication failed"

  # Rule 3: Ignore 404 errors
  - action: "continue"
    condition: "response.status == 404"
    message: "Resource not found, continuing"

  # Rule 4: Skip invalid records in iteration
  - action: "skip"
    condition: "is_null(record.id)"
    message: "Skipping record without ID"

  # Rule 5: Stop iteration when reaching limit
  - action: "break"
    condition: "state.records_processed >= state.limit"
    message: "Processed limit reached, breaking iteration"
```

> 📝 **Note:** Rules are evaluated in order. The first matching rule's action is executed.

## Backoff Strategies

When a rule uses the `retry` action, the backoff strategy determines how long to wait between retry attempts.

### Backoff Types

| Type          | Calculation                 | Use Case                           | Example Delays (backoff\_base=1) |
| ------------- | --------------------------- | ---------------------------------- | -------------------------------- |
| `none`        | No delay                    | Immediate retries (use cautiously) | 0s, 0s, 0s, ...                  |
| `constant`    | Fixed delay                 | Predictable retry timing           | 1s, 1s, 1s, ...                  |
| `linear`      | base × attempt              | Gradual backoff                    | 1s, 2s, 3s, 4s, 5s, ...          |
| `exponential` | base × 2^(attempt-1)        | Aggressive backoff (recommended)   | 1s, 2s, 4s, 8s, 16s, ...         |
| `jitter`      | exponential + random(0-50%) | Avoid thundering herd              | 1s, 3s, 5s, 10s, 20s, ...        |

### Backoff Examples with Timing

#### None (No Backoff)

```yaml
rules:
  - action: retry
    condition: "response.status >= 500"
    max_attempts: 3
    backoff: none
```

**Retry Timeline:**

* Request 1 (fails) → 0s wait
* Request 2 (fails) → 0s wait
* Request 3 (fails) → Give up

> ⚠️ **Warning:** No backoff can overwhelm failing services. Use only when retries must be immediate.

#### Constant Backoff

```yaml
rules:
  - action: retry
    condition: "response.status >= 500"
    max_attempts: 5
    backoff: constant
    backoff_base: 2  # 2 seconds between each retry
```

**Retry Timeline:**

* Request 1 (fails) → Wait 2s
* Request 2 (fails) → Wait 2s
* Request 3 (fails) → Wait 2s
* Request 4 (fails) → Wait 2s
* Request 5 (fails) → Give up

**Total time:** \~8 seconds

#### Linear Backoff

```yaml
rules:
  - action: retry
    condition: "response.status == 429"
    max_attempts: 5
    backoff: linear
    backoff_base: 3  # Base delay of 3 seconds
```

**Retry Timeline:**

* Request 1 (fails) → Wait 3s (3 × 1)
* Request 2 (fails) → Wait 6s (3 × 2)
* Request 3 (fails) → Wait 9s (3 × 3)
* Request 4 (fails) → Wait 12s (3 × 4)
* Request 5 (fails) → Give up

**Total time:** \~30 seconds

#### Exponential Backoff (Recommended)

```yaml
rules:
  - action: retry
    condition: "response.status == 429 || response.status >= 500"
    max_attempts: 5
    backoff: exponential
    backoff_base: 2  # Base delay of 2 seconds
```

**Retry Timeline:**

* Request 1 (fails) → Wait 2s (2 × 2⁰ = 2)
* Request 2 (fails) → Wait 4s (2 × 2¹ = 4)
* Request 3 (fails) → Wait 8s (2 × 2² = 8)
* Request 4 (fails) → Wait 16s (2 × 2³ = 16)
* Request 5 (fails) → Give up

**Total time:** \~30 seconds

> 💡 **Tip:** Exponential backoff is the industry standard for API retries. It quickly backs off from transient failures while giving services time to recover.

#### Jitter Backoff (Best for High Concurrency)

```yaml
rules:
  - action: retry
    condition: "response.status == 429"
    max_attempts: 5
    backoff: jitter
    backoff_base: 2
```

**Retry Timeline (example with random jitter):**

* Request 1 (fails) → Wait 2.3s (2s + 15% jitter)
* Request 2 (fails) → Wait 5.1s (4s + 28% jitter)
* Request 3 (fails) → Wait 10.4s (8s + 30% jitter)
* Request 4 (fails) → Wait 20.8s (16s + 30% jitter)
* Request 5 (fails) → Give up

**Total time:** \~38 seconds (varies due to randomness)

> 📝 **Note:** Jitter adds 0-50% random delay to exponential backoff. This prevents multiple clients from retrying simultaneously (thundering herd problem).

### Choosing the Right Backoff Strategy

| Scenario                   | Recommended Strategy      | Reasoning                       |
| -------------------------- | ------------------------- | ------------------------------- |
| Rate limits (429)          | `exponential` or `jitter` | Gives API time to recover quota |
| Server errors (5xx)        | `exponential`             | Allows server recovery time     |
| Temporary network issues   | `linear`                  | Moderate, predictable backoff   |
| Must retry immediately     | `constant` with low base  | Fast retries, simple timing     |
| High-concurrency scenarios | `jitter`                  | Prevents retry storms           |

## Rate Limit Handling

Sling automatically detects and respects rate limit headers from API responses. This works in conjunction with backoff strategies to optimize retry timing.

### Automatic Rate Limit Detection

When a `retry` rule triggers on a 429 status, Sling automatically checks for rate limit headers:

```yaml
rules:
  - action: retry
    condition: "response.status == 429"
    max_attempts: 5
    backoff: exponential  # Fallback if no rate limit headers
    backoff_base: 2
```

### Supported Rate Limit Headers

Sling checks for these headers in order of priority:

#### 1. IETF Standard Headers (Preferred)

| Header                | Description                  | Example                |
| --------------------- | ---------------------------- | ---------------------- |
| `RateLimit-Reset`     | Seconds until quota resets   | `60` (wait 60 seconds) |
| `RateLimit-Remaining` | Requests remaining in window | `0`                    |
| `RateLimit-Policy`    | Rate limit window and quota  | `"60;q=100;w=60"`      |

```http
HTTP/1.1 429 Too Many Requests
RateLimit-Reset: 30
RateLimit-Remaining: 0
RateLimit-Policy: "minute";q=60;w=60
```

**Behavior:** Sling will wait 30 seconds before retrying.

#### 2. Legacy/Alternative Headers

| Header              | Description                         | Example                                  |
| ------------------- | ----------------------------------- | ---------------------------------------- |
| `Retry-After`       | Seconds or HTTP date to retry after | `120` or `Wed, 21 Oct 2025 07:28:00 GMT` |
| `X-RateLimit-Reset` | Unix timestamp when quota resets    | `1743158739`                             |

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 60
```

**Behavior:** Sling will wait 60 seconds before retrying.

### Rate Limit Header Processing

When rate limit headers are detected, they **override** the backoff calculation:

{% @mermaid/diagram content="graph TD
A\[Retry Triggered] --> B{Status 429?}
B -->|No| C\[Use Backoff Strategy]
B -->|Yes| D{RateLimit Headers?}
D -->|Yes| E\[Extract Reset Time]
D -->|No| F{Retry-After Header?}
F -->|Yes| G\[Parse Retry-After]
F -->|No| C
E --> H\[Wait for Reset Time]
G --> H
C --> I\[Wait for Backoff Duration]
H --> J\[Retry Request]
I --> J

```
style A fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style D fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style H fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Rate Limit Policy Parsing

For APIs using the IETF `RateLimit-Policy` header:

```http
RateLimit-Policy: "hour";q=1000;w=3600, "day";q=5000;w=86400
RateLimit-Remaining: 0
```

Format: `"name";q=quota;w=window`

* `q`: Quota (number of requests)
* `w`: Window duration (seconds)

**Sling's behavior:**

* If `RateLimit-Remaining` is 0, waits for the full window
* Otherwise, calculates proportional wait: `window × (1 - remaining/quota)`

### Complete Rate Limit Example

```yaml
endpoints:
  api_data:
    request:
      url: "{state.base_url}/data"
      rate: 10  # Max 10 requests per second normally

    response:
      rules:
        # Rule 1: Handle rate limits with header-aware retry
        - action: retry
          condition: "response.status == 429"
          max_attempts: 5
          backoff: exponential  # Fallback strategy
          backoff_base: 2
          message: "Rate limited - waiting {response.headers['ratelimit-reset']}s"

        # Rule 2: Fail on repeated rate limits
        - action: fail
          condition: "response.status == 429 && request.attempts >= 5"
          message: "Rate limit exceeded after 5 retries"

        # Rule 3: Handle server errors differently
        - action: retry
          condition: "response.status >= 500"
          max_attempts: 3
          backoff: jitter
          backoff_base: 5
```

**What happens:**

1. On first 429, checks for `RateLimit-Reset` header
2. If found, waits that duration (ignoring backoff calculation)
3. If not found, uses exponential backoff (2s, 4s, 8s, ...)
4. Retries up to 5 times
5. Fails if still getting 429 after all retries

### Testing Rate Limits

Use the trace flag to see rate limit handling in action:

```bash
sling conns test MY_API --endpoints data_endpoint --trace
```

Look for output like:

```
DBG r.0001.abc   response code=429 duration=234ms
DBG r.0001.abc   using rate limit headers for backoff: 30s
DBG r.0001.abc   rule met to retry (attempt=2) with backoff=30s: response.status == 429
```

> 💡 **Tip:** Most well-designed APIs include rate limit headers. Always use `exponential` or `jitter` backoff as a fallback for APIs that don't.


# Queues

Queues enable sophisticated multi-step data extraction workflows where one endpoint collects data (such as records or identifiers )that is used by subsequent endpoints. This is essential for APIs that require separate calls to fetch related data.

## Queue Architecture

```mermaid
graph TD
    A[Endpoint 1: List Items] --> B[Extract IDs]
    B --> C[Queue: item_ids]
    C --> D[Endpoint 2: Item Details]
    D --> E[Iterate over Queue]
    E --> F[Make Detail Requests]

    style A fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style B fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style C fill:#5c6bc0,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style D fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style E fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
    style F fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff
```

## Queue Usage

Queues are **auto-detected** from your endpoint definitions — Sling discovers them by scanning processor `output:` and `iterate.over:` expressions. You do not need to declare them.

> ⚠️ **Deprecated:** The top-level `queues:` list is no longer required and is ignored at runtime. Old specs that still include it continue to load, but Sling emits a one-time deprecation warning. Remove the field when you next touch the spec.

At load time, Sling enforces that every consumed queue has at least one producer, so typos surface immediately as a `queue(s) with no producer` error rather than at runtime.

### 1. Sending Data to Queues

Use processors to send data from one endpoint to a queue:

```yaml
endpoints:
  list_customers:
    description: "Get list of customers"
    
    request:
      url: "customers"
    
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
      
      processors:
        # Send customer IDs to queue for detailed processing
        - expression: "record.id"
          output: "queue.customer_ids"
```

### 2. Consuming Data from Queues

Use the `iterate` section to process each item from a queue:

```yaml
endpoints:
  customer_details:
    description: "Get detailed customer information"
    
    # Process each customer ID from the queue
    iterate:
      over: "queue.customer_ids"
      into: "state.current_customer_id"
    
    request:
      url: "customers/{state.current_customer_id}"
    
    response:
      records:
        jmespath: "[*]"  # Single customer object
        primary_key: ["id"]
      
      processors:
        # Add the customer ID to each record for reference
        - expression: "state.current_customer_id"
          output: "record.customer_id"
```

## Queue Consumption: `deferred` vs `immediate`

The `iterate.consume` option controls **when** a consumer reads its queue:

| Value                | Behavior                                                                                                                                      |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `deferred` (default) | The consumer waits for the producer to fully finish, then reads the queue from the start.                                                     |
| `immediate`          | The consumer tails the queue **live** — it starts processing as soon as the producer appends records, running concurrently with the producer. |

```yaml
endpoints:
  search:
    queue_only: true
    request:
      url: "{state.base_url}/search"
    response:
      processors:
        - expression: "record.id"
          output: "queue.item_ids"

  details:
    iterate:
      over: "queue.item_ids"
      into: "state.item_id"
      consume: immediate   # tail the queue live instead of waiting
    request:
      url: "{state.base_url}/items/{state.item_id}"
```

### When to use each

* **`deferred` (default)** — Use when the consumer needs the *complete* set of queue items before it can do its job, such as deduplication or aggregation across all records. Also the safest choice for ordinary fan-out where you don't need pipelining.
* **`immediate`** — Use when you want **pipelined throughput** (the consumer works while the producer is still fetching) and **fail-fast** behavior.

### Fail-fast with `immediate`

When a consumer uses `consume: immediate`, Sling runs the producer and its live consumers concurrently (it forces threaded execution, equivalent to setting `SLING_THREADS`). The producer plus all the consumers that tail its queue form a **fail-fast group**:

* If a **consumer** fails, its **producer** (and the other consumers in the same group) are terminated — Sling stops pulling data that nothing will use.
* If a **producer** fails, the **consumers** tailing its queue are terminated — they would otherwise block forever waiting for a queue that will never complete.
* **Unrelated streams and other queue groups are not affected** — they continue and can finish successfully.

This avoids wasted work: with the default `deferred` consumption, a producer always runs to completion before the consumer even starts, so a later consumer failure means the entire producer fetch was wasted. `immediate` surfaces failures early while the producer is still running.

> 💡 **Tip:** `immediate` only changes behavior for queue iteration. It has no effect on non-queue `iterate.over` expressions (arrays, `range()`, etc.).

## Queue-Only Endpoints

Some producer endpoints exist only to fan IDs into a queue for a downstream consumer — they don't produce records anyone wants to read. Mark them with `queue_only: true` and Sling will:

* run the endpoint and drain its records into the queue(s) it populates via processors,
* skip emitting a record stream (no target write, no row count),
* hide the endpoint from `*` wildcard discovery and from `sling conns discover`,
* still schedule the endpoint to run before any consumer that iterates over its queue (dependency order is auto-detected).

The endpoint is still selectable by explicit name when you need to debug it.

### Example: Search → Detail

```yaml
endpoints:
  # Producer: queue_only — no record output, just populates queue.imdb_ids
  search:
    queue_only: true
    request:
      url: "{state.base_url}/"
      method: GET
      parameters:
        apikey: "{env.OMDB_API_KEY}"
        s: "batman"
        type: "movie"
    response:
      records:
        jmespath: "Search"
      processors:
        - expression: "record.imdbID"
          output: "queue.imdb_ids"

  # Consumer: iterates the queue populated by `search`
  details:
    iterate:
      over: "queue.imdb_ids"
      into: "state.imdb_id"
      concurrency: 2
    request:
      url: "{state.base_url}/"
      method: GET
      parameters:
        apikey: "{env.OMDB_API_KEY}"
        i: "{state.imdb_id}"
    response:
      records:
        jmespath: "@"
        primary_key: ["imdbID"]
```

In a replication that targets `*`, Sling will run `search` first (filling `queue.imdb_ids`) and then `details` (writing one row per movie). `search` does not appear in `conns discover` output or in the wildcard endpoint list — list it explicitly by name to inspect it.

> 💡 **Tip:** `queue_only` replaces the older pattern of writing throwaway records from a producer endpoint just to satisfy the "every endpoint must produce a stream" assumption. Combined with auto-detected queues, your producer/consumer wiring stays declared in exactly one place: the endpoints themselves.

## Queue Functions

Queues can be used with built-in functions for advanced processing:

### Chunking Queue Data

Process queue items in batches for more efficient API calls:

```yaml
endpoints:
  batch_customer_details:
    description: "Process customers in batches"
    
    iterate:
      # Process 50 customer IDs at a time
      over: "chunk(queue.customer_ids, 50)"
      into: "state.customer_batch"          # as an array
      concurrency: 3  # Process 3 batches concurrently
    
    request:
      url: "customers/batch"
      method: "POST"
      payload:
        ids: '{join(state.customer_batch, ","}'
    
    response:
      records:
        jmespath: "customers[]"
        primary_key: ["id"]
```

## Real-World Example: Stripe API

This example from the Stripe API demonstrates a complete queue-based workflow:

```yaml
endpoints:
  # Step 1: Collect customer IDs
  customer:
    description: "Retrieve list of customers"
    
    request:
      url: "customers"
    
    pagination:
      next_state:
        starting_after: '{response.records[-1].id}'
      stop_condition: "response.json.has_more == false"
    
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
      
      processors:
        # Send each customer ID to the queue
        - expression: "record.id"
          output: "queue.customer_ids"

  # Step 2: Get customer balance transactions
  customer_balance_transaction:
    description: "Retrieve customer balance transactions"
    
    iterate:
      over: "queue.customer_ids"
      into: "state.customer_id"
    
    request:
      url: "customers/{state.customer_id}/balance_transactions"
    
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
      
      processors:
        # Add customer_id to each transaction record
        - expression: "state.customer_id"
          output: "record.customer_id"

  # Step 3: Collect invoice IDs while getting invoice data
  invoice:
    description: "Retrieve invoices and queue IDs for line items"
    
    request:
      url: "invoices"
    
    pagination:
      next_state:
        starting_after: '{response.records[-1].id}'
      stop_condition: 'jmespath(response.json, "has_more") == false'
    
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
      
      processors:
        # Queue invoice IDs for line item extraction
        - expression: "record.id"
          output: "queue.invoice_ids"

  # Step 4: Get line items for each invoice
  invoice_line_item:
    description: "Retrieve invoice line items"
    
    iterate:
      over: "queue.invoice_ids"
      into: "state.invoice_id"
    
    request:
      url: "invoices/{state.invoice_id}/lines"
    
    response:
      records:
        jmespath: "data[]"
        primary_key: ["id"]
      
      processors:
        # Link line items back to their invoice
        - expression: "state.invoice_id"
          output: "record.invoice_id"
```

## Queue Properties and Behavior

### Queue Characteristics

| Property              | Description                                       | Example                                    |
| --------------------- | ------------------------------------------------- | ------------------------------------------ |
| **Temporary Storage** | Queues are backed by temporary files              | Automatically cleaned up after run         |
| **FIFO Order**        | Items are processed in first-in, first-out order  | IDs processed in the order they were added |
| **JSON Encoding**     | All data is JSON-encoded for safe storage         | Handles strings, numbers, objects, arrays  |
| **Single Run Scope**  | Queues exist only within a single Sling execution | Cannot persist between separate runs       |

### Queue Lifecycle

```mermaid
sequenceDiagram
    participant E1 as Producer Endpoint
    participant Q as Queue (temp file)
    participant E2 as Consumer Endpoint

    Note over E1: Start processing records
    E1->>Q: Append ID 1
    E1->>Q: Append ID 2
    E1->>Q: Append ID 3
    Note over E1: Finish writing

    Note over Q: Switch to read mode
    E2->>Q: Read ID 1
    E2->>Q: Read ID 2
    E2->>Q: Read ID 3
    Note over E2: Process each ID

    Note over Q: Auto cleanup on completion
```

## Direct Queue-to-Records Pattern

You can pipe queue data **directly to records** without making HTTP requests using the special syntax `iterate.into: "response.records"`.

### Basic Syntax

```yaml
endpoints:
  deduplicate_customers:
    description: "Remove duplicate customer IDs"

    iterate:
      over: "queue.raw_customer_ids"
      into: "response.records"  # Special: Direct to records, no HTTP call

    # No request block needed!

    response:
      records:
        primary_key: ["customer_id"]  # Deduplicate

      processors:
        - expression: "record"
          output: "queue.clean_customer_ids"
```

### When to Use

* **Deduplicate** queue items before further processing
* **Enrich** queue data with state variables
* **Transform** queue structure
* **Export** queue contents as a separate dataset

### Example: Deduplication Workflow

```yaml
endpoints:
  # Step 1: Collect IDs (may have duplicates)
  list_orders:
    request:
      url: "{state.base_url}/orders"
    response:
      processors:
        - expression: "record.customer_id"
          output: "queue.raw_customer_ids"

  # Step 2: Deduplicate using direct queue-to-records
  deduplicate_customers:
    iterate:
      over: "queue.raw_customer_ids"
      into: "response.records"
    response:
      records:
        primary_key: ["customer_id"]
      processors:
        - expression: "record"
          output: "queue.clean_customer_ids"

  # Step 3: Use clean IDs
  customer_details:
    iterate:
      over: "queue.clean_customer_ids"
      into: "state.customer_id"
    request:
      url: "{state.base_url}/customers/{state.customer_id}"
```

### Data Type Handling

**Scalar values** (strings/numbers) are wrapped: `"user123"` → `{"value": "user123"}`

```yaml
processors:
  - expression: "record.value"  # Access via .value
    output: "record.user_id"
```

**Object values** are used as-is: `{"id": 1, "name": "Alice"}` → same structure

```yaml
processors:
  - expression: "record.id"  # Access fields directly
    output: "record.user_id"
```

### Limitations

* No HTTP response data (`response.status`, `response.headers` unavailable)
* Cannot use pagination or response rules
* Don't define a `request` block (it will be ignored)

> 💡 **Tip:** This pattern is much faster than unnecessary HTTP requests for queue transformation steps.

## Advanced Queue Patterns

### Pattern 1: Multi-Level Hierarchies

Process nested data structures with multiple queue levels:

```yaml
endpoints:
  accounts:
    response:
      # Level 1: Get accounts
      processors:
        - expression: "record.id"
          output: "queue.account_ids"

  customers:
    # Level 2: Get customers for each account
    iterate:
      over: "queue.account_ids"
      into: "state.account_id"
    response:
      processors:
        - expression: "record.id"
          output: "queue.customer_ids"

  subscriptions:
    # Level 3: Get subscriptions for each customer
    iterate:
      over: "queue.customer_ids"
      into: "state.customer_id"
    response:
      processors:
        - expression: "record.id"
          output: "queue.subscription_ids"
```

### Pattern 2: Conditional Queue Population

Only queue certain items based on conditions:

```yaml
processors:
  # Only queue active customers
  - expression: >
      if(record.status == "active", record.id, null)
    output: "queue.active_customer_ids"
  
  # Queue high-value customers for special processing
  - expression: >
      if(record.total_spent > 10000, record.id, null)
    output: "queue.vip_customer_ids"
```

### Pattern 3: Queue Transformation

Transform data before queuing:

```yaml
processors:
  # Create composite keys for the queue
  - expression: >
      {
        "customer_id": record.id,
        "type": record.customer_type,
        "priority": if(record.is_vip, "high", "normal")
      }
    output: "queue.enriched_customers"
```

## Queue Best Practices

### Performance Optimization

```yaml
iterate:
  over: "chunk(queue.large_dataset, 100)"  # Process in batches
  into: "state.batch"
  concurrency: 5  # Parallel processing
```

### Error Handling with Queues

```yaml
rules:
  # Continue processing other queue items even if one fails
  - action: "continue"
    condition: "response.status == 404"
    message: "Item not found, skipping"
  
  # Retry transient errors
  - action: "retry"
    condition: "response.status >= 500"
    max_attempts: 3
```

> 💡 **Tip:** Use descriptive queue names that clearly indicate their purpose (e.g., `customer_ids`, `pending_order_ids`, `failed_payment_ids`).

> ⚠️ **Warning:** Queues consume disk space proportional to the number of items. For very large datasets (millions of items), monitor available disk space.

> 📝 **Note:** Queue items are automatically JSON-encoded, so complex objects, arrays, and special characters are handled safely.


# Dynamic Endpoints

Dynamic endpoints allow you to programmatically generate multiple endpoint configurations based on runtime data. This is powerful for APIs where the list of available endpoints or resources isn't known until you query the API itself.

## When to Use Dynamic Endpoints

Use dynamic endpoints when:

* The list of available resources/endpoints is determined at runtime
* You need to generate similar endpoints for multiple entities (e.g., one endpoint per table, per user, per organization)
* Endpoint configuration depends on data fetched from the API
* You want to avoid manually listing hundreds of similar endpoints

## Static vs. Dynamic Endpoints

### Static Endpoints (Standard)

```yaml
endpoints:
  users:
    request:
      url: "{state.base_url}/users"
    response:
      records:
        jmespath: "data[]"

  orders:
    request:
      url: "{state.base_url}/orders"
    response:
      records:
        jmespath: "data[]"

  products:
    request:
      url: "{state.base_url}/products"
    response:
      records:
        jmespath: "data[]"
```

**Limitations:**

* Must know all endpoints in advance
* Repetitive configuration for similar endpoints
* Manual updates needed when new resources are added

### Dynamic Endpoints (Advanced)

```yaml
dynamic_endpoints:
  - setup:
      # Fetch list of available tables
      - request:
          url: "{state.base_url}/tables"
        response:
          processors:
            - expression: 'jmespath(response.json, "tables")'
              output: "state.table_list"
              aggregation: last

    iterate: "state.table_list"
    into: "state.table_name"

    endpoint:
      name: "{state.table_name}"
      request:
        url: "{state.base_url}/tables/{state.table_name}/data"
      response:
        records:
          jmespath: "data[]"
```

**Benefits:**

* Automatically discovers available endpoints
* Single configuration for many similar endpoints
* Adapts automatically to API changes

## Dynamic Endpoint Structure

A dynamic endpoint definition has these parts:

```yaml
dynamic_endpoints:
  - setup: [<array of sequence calls>]    # Optional: Get data for iteration
    iterate: <expression or state variable> # What to iterate over
    into: <variable name>                  # Variable to store current item
    endpoint: <endpoint configuration>     # Template for generated endpoints
```

### Properties

| Property   | Required | Description                                                    | Example                                                            |
| ---------- | -------- | -------------------------------------------------------------- | ------------------------------------------------------------------ |
| `setup`    | No       | Sequence of calls to prepare data                              | Fetch list of resources                                            |
| `iterate`  | Yes      | Expression, JSON literal, or inline list / object to loop over | `state.table_list`, `'["a","b"]'`, or a raw YAML array (see below) |
| `into`     | Yes      | Variable name for current item                                 | `state.table_name`                                                 |
| `endpoint` | Yes      | Endpoint configuration template                                | Standard endpoint config                                           |

### Forms of `iterate`

`iterate` accepts four shapes; pick whichever is clearest for your use case:

1. **JMESPath expression string** — evaluated against state. Use when the list is built dynamically during `setup` (or comes from `defaults.state`).

   ```yaml
   iterate: 'state.config.resources[].name'
   ```
2. **JSON literal string** — a one-line inline array. Useful for small static lists.

   ```yaml
   iterate: '["users", "orders", "products"]'
   ```
3. **Inline YAML array (recommended for nested objects)** — write the list directly as YAML, with full support for nested fields. Each element is bound to the `into` variable as-is, so the template can reference fields with dot notation (e.g. `{state.report.name}`).

   ```yaml
   dynamic_endpoints:
     - iterate:
         - name: 'daily_signups'
           dimensions: [date, country, plan]
         - name: 'weekly_revenue'
           dimensions: [week, product_id, currency]
       into: 'state.report'
       endpoint:
         name: '{state.report.name}'
         request:
           url: '{state.base_url}/reports/{state.report.name}'
           parameters:
             dimensions: '{join(state.report.dimensions, ",")}'
         response:
           records:
             jmespath: 'rows[]'
   ```
4. **Inline YAML single object** — treated as a one-element list. Useful when you only have one item but want the same templating contract.

   ```yaml
   iterate:
     name: 'daily_signups'
     dimensions: [date, country, plan]
   into: 'state.report'
   ```

## Examples

### Example 1: Database Tables

Dynamically create endpoints for each table in a database API:

```yaml
name: "Database API"
description: "API for accessing database tables"

defaults:
  state:
    base_url: "https://api.example.com/v1"

dynamic_endpoints:
  - setup:
      # Get list of all tables
      - request:
          url: "{state.base_url}/metadata/tables"
        response:
          processors:
            - expression: 'jmespath(response.json, "tables[].name")'
              output: "state.available_tables"
              aggregation: collect

    # Create one endpoint per table
    iterate: "state.available_tables"
    into: "state.table_name"

    endpoint:
      name: "table_{state.table_name}"
      description: "Data from {state.table_name} table"

      request:
        url: "{state.base_url}/tables/{state.table_name}"
        parameters:
          limit: 1000

      pagination:
        next_state:
          offset: "{state.offset + 1000}"
        stop_condition: "length(response.records) < 1000"

      response:
        records:
          jmespath: "rows[]"
          primary_key: ["id"]
```

**Result:** If the API returns tables `["users", "orders", "products"]`, Sling automatically creates three endpoints:

* `table_users`
* `table_orders`
* `table_products`

### Example 2: Multi-Organization Data

Create endpoints for each organization the user has access to:

```yaml
name: "Multi-Org API"
description: "API with per-organization endpoints"

defaults:
  state:
    base_url: "https://api.example.com/v2"
    # Default date range for event queries
    start_date: '{date_format(date_add(now(), -30, "day"), "%Y-%m-%d")}'
    end_date: '{date_format(now(), "%Y-%m-%d")}'

authentication:
  type: oauth2
  flow: client_credentials
  client_id: "{secrets.client_id}"
  client_secret: "{secrets.client_secret}"
  authentication_url: "{state.base_url}/oauth/token"

dynamic_endpoints:
  - setup:
      # Get list of organizations user can access
      - request:
          url: "{state.base_url}/user/organizations"
        response:
          processors:
            - expression: 'jmespath(response.json, "organizations")'
              output: "state.org_list"
              aggregation: last

    # Create one endpoint per organization
    iterate: "state.org_list"
    into: "state.current_org"

    endpoint:
      name: "org_{state.current_org.id}_events"
      description: "Events for organization: {state.current_org.name}"

      request:
        url: "{state.base_url}/organizations/{state.current_org.id}/events"
        parameters:
          from_date: "{state.start_date}"
          to_date: "{state.end_date}"

      response:
        records:
          jmespath: "events[]"
          primary_key: ["event_id"]

        processors:
          # Add organization context to each record
          - expression: "state.current_org.id"
            output: "record.organization_id"
          - expression: "state.current_org.name"
            output: "record.organization_name"
```

### Example 3: Geographic Regions

Create endpoints for each geographic region:

```yaml
name: "Regional Sales API"
description: "Sales data by region"

defaults:
  state:
    base_url: "https://api.example.com"
    regions:
      - code: "us-east"
        name: "US East Coast"
      - code: "us-west"
        name: "US West Coast"
      - code: "eu-west"
        name: "Europe West"
      - code: "apac"
        name: "Asia Pacific"

dynamic_endpoints:
  # No setup needed - iterate over predefined regions
  - iterate: "state.regions"
    into: "state.region"

    endpoint:
      name: "sales_{state.region.code}"
      description: "Sales data for {state.region.name}"

      request:
        url: "{state.base_url}/regions/{state.region.code}/sales"
        parameters:
          date_from: "{state.start_date}"
          date_to: "{state.end_date}"

      pagination:
        next_state:
          page: "{state.page + 1}"
        stop_condition: 'jmespath(response.json, "has_more") == false'

      response:
        records:
          jmespath: "sales[]"
          primary_key: ["sale_id"]

        processors:
          # Tag each record with region info
          - expression: "state.region.code"
            output: "record.region_code"
          - expression: "state.region.name"
            output: "record.region_name"
```

### Example 4: Inline YAML Array with Nested Objects

When the list of resources is known up-front and each entry needs more than just a name (e.g. a report name plus the dimensions to pull), use a raw YAML array directly in `iterate`. No `setup` call, no JSON-string escaping — the YAML is the data.

```yaml
name: "Analytics Reports API"

defaults:
  state:
    base_url: "https://api.example.com/v2"

dynamic_endpoints:
  - iterate:
      - name: 'daily_signups'
        dimensions: [date, country, plan]
        granularity: day
      - name: 'weekly_revenue'
        dimensions: [week, product_id, currency]
        granularity: week
      - name: 'monthly_churn'
        dimensions: [month, segment, reason]
        granularity: month
    into: 'state.report'

    endpoint:
      name: '{state.report.name}'
      description: 'Analytics report: {state.report.name}'

      request:
        url: '{state.base_url}/reports/{state.report.name}'
        parameters:
          dimensions: '{join(state.report.dimensions, ",")}'
          granularity: '{state.report.granularity}'

      response:
        records:
          jmespath: 'rows[]'
          primary_key: ['id']
```

**Result:** three endpoints — `daily_signups`, `weekly_revenue`, `monthly_churn` — each fetching only the dimensions declared inline. To add another report, append one more YAML entry; no code or setup step changes.

## How Dynamic Endpoints Work

```mermaid
graph TD
    A[Start API Spec Loading] --> B{Has dynamic_endpoints?}
    B -->|No| C[Load Static Endpoints]
    B -->|Yes| D[Execute Setup Sequence]

    D --> E[Evaluate iterate Expression]
    E --> F[Get List of Items]

    F --> G{For Each Item}
    G --> H[Set into Variable]
    H --> I[Render Endpoint Template]
    I --> J[Add to Endpoint List]

    J --> G
    G -->|Done| K[Combine with Static Endpoints]
    C --> K

    K --> L[Final Endpoint List]

    style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style D fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style E fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
    style I fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
    style L fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff
```

## Execution Flow

1. **Authentication**: API authenticates once
2. **Setup Phase** (if defined):
   * Executes setup sequence calls
   * Fetches data needed for iteration
   * Stores results in state
3. **Iteration Phase**:
   * Evaluates the `iterate` expression
   * For each item, sets the `into` variable
   * Renders the endpoint template with current values
   * Adds the generated endpoint to the list
4. **Endpoint Registration**:
   * Combines dynamic and static endpoints
   * Applies defaults
   * Validates all endpoints
5. **Execution**: Endpoints run normally (can be selected via patterns)

## Variable Scoping

When rendering dynamic endpoints, variables are available in this order of precedence:

1. **Current iteration value** (`into` variable) - Highest priority
2. **State variables** from setup
3. **Environment variables**
4. **Secrets**
5. **Defaults**

```yaml
dynamic_endpoints:
  - setup:
      - request:
          url: "{state.base_url}/config"
        response:
          processors:
            - expression: 'jmespath(response.json, "api_version")'
              output: "state.api_version"  # Available to endpoint template
              aggregation: last

    iterate: '["users", "orders", "products"]'
    into: "state.resource_type"  # Available as {state.resource_type} in template

    endpoint:
      name: "{state.resource_type}_v{state.api_version}"  # Uses both
      request:
        url: "{env.BASE_URL}/{state.resource_type}"  # Environment variable
        headers:
          Authorization: "Bearer {secrets.api_token}"  # Secret
```

## Combining Static and Dynamic Endpoints

You can mix static and dynamic endpoints in the same spec:

```yaml
name: "Hybrid API"

# Static endpoints
endpoints:
  health_check:
    request:
      url: "{state.base_url}/health"

  metadata:
    request:
      url: "{state.base_url}/metadata"

# Dynamic endpoints
dynamic_endpoints:
  - iterate: '["users", "orders", "products"]'
    into: "state.entity"
    endpoint:
      name: "{state.entity}"
      request:
        url: "{state.base_url}/{state.entity}"
```

**Result:** 5 total endpoints:

* `health_check` (static)
* `metadata` (static)
* `users` (dynamic)
* `posts` (dynamic)
* `comments` (dynamic)

## Selecting Dynamic Endpoints

When running, you can select dynamic endpoints just like static ones:

```bash
# Run all endpoints (static + dynamic)
sling conns test MY_API

# Run specific dynamic endpoint(s)
sling conns test MY_API --endpoints table_users

# Run pattern matching
sling conns test MY_API --endpoints "table_*"

# Run multiple
sling conns test MY_API --endpoints "org_123_*,org_456_*"
```

## Limitations and Considerations

### Performance Considerations

1. **Setup Overhead**: Setup sequence runs once before endpoint generation
2. **Large Lists**: Many dynamic endpoints increase memory usage
3. **Discovery Time**: Fetching endpoint list adds latency

### Best Practices

#### 1. Filter in Setup

Don't generate endpoints you won't use:

```yaml
setup:
  - request:
      url: "{state.base_url}/tables"
    response:
      processors:
        # Only include tables matching pattern
        - expression: >
            filter(jmespath(response.json, "tables"), "starts_with(name, 'prod_')")
          output: "state.filtered_tables"
          aggregation: last
```

#### 2. Use Meaningful Names

Make generated endpoint names descriptive:

```yaml
# Good: Clear what this endpoint does
name: "region_{region.code}_sales_daily"

# Avoid: Unclear names
name: "ep_{index}"
```

#### 3. Add Metadata

Include helpful information in generated endpoints:

```yaml
endpoint:
  name: "{state.table_name}"
  description: "Data from {state.table_name} table (schema: {table_schema})"
  docs: "https://docs.example.com/tables/{state.table_name}"
```

#### 4. Validate Iteration Data

Ensure the iteration data is valid:

```yaml
setup:
  - request:
      url: "{state.base_url}/tables"
    response:
      processors:
        - expression: 'jmespath(response.json, "tables")'
          output: "state.tables"
          aggregation: last
```

## Troubleshooting

### No Endpoints Generated

Check that:

1. Setup sequence succeeds
2. Iterate expression returns non-empty array
3. Into variable is properly referenced in endpoint template

Use debug logging:

```yaml
setup:
  - request:
      url: "{state.base_url}/resources"
    response:
      processors:
        - expression: 'jmespath(response.json, "resources")'
          output: "state.resource_list"
          aggregation: last
        # Debug: Log what we got
        - expression: log("Found " + string(length(state.resource_list)) + " resources")
          output: ""
```

### Templates Not Rendering

Verify variable names match:

```yaml
# ❌ Wrong: Mismatched variable names
iterate: "state.items"
into: "state.item"
endpoint:
  name: "{state.resource}"  # Should be {item}

# ✓ Correct: Matching variable names
iterate: "state.items"
into: "state.item"
endpoint:
  name: "{state.item}"
```

### Duplicate Endpoint Names

Ensure each generated endpoint has a unique name:

```yaml
# ❌ Wrong: Same name for all
endpoint:
  name: "data_endpoint"  # All will have same name!

# ✓ Correct: Unique names
endpoint:
  name: "data_{state.resource_id}"  # Each gets unique name
```

> 💡 **Tip:** While dynamic endpoints is powerful, start with static endpoints for simpler APIs. Use dynamic endpoints when you have many similar endpoints or when endpoint discovery is necessary.


# Testing & Debugging

Testing your Sling API specifications is a crucial step before deploying them in production. This document covers tools and techniques to verify your specs work correctly, debug issues, and optimize performance.

## Content Overview

* [Testing Workflow](#testing-workflow)
* [Creating a Connection](#creating-a-connection-with-an-api-spec)
* [Using the Test Command](#using-the-test-command)
* [Debugging Tools](#debugging-tools)
* [Common Issues](#common-issues)
* [Testing Best Practices](#testing-best-practices)

## Testing Workflow

The general workflow for developing and testing API specs is:

{% @mermaid/diagram content="graph TD
A\[Create Initial API Spec] --> B\[Create & Test Connection]
B -->|Success| C\[Discover Available Endpoints]
B -->|Fails| D\[Fix Authentication]
D --> B
C --> E\[Test Individual Endpoints]
E -->|Works| F\[Add More Endpoints]
E -->|Issues| G\[Debug with --trace]
G --> H\[Fix Issues]
H --> E
F --> I\[Create Replication YAML]
I --> J\[Run Full Replication Test]

```
style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style J fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#26c6da,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

## Creating a Connection with an API Spec

The first step is create and save your Spec YAML file somewhere accessible. This can be your local drive, or any other be any [storage connection](/connections/file-connections) you have setup, such a S3/GCP bucket, or FTP/SFTP. Furthermore, sling supports reading API Specs from a HTTP URL (such as Github URLs).

Once you have an API Spec file to use, you can then create a connection in your [env.yaml](/sling-cli/environment#sling-env-file-envyaml) file, like this:

```yaml
connections:
  stripe:
    type: api
    # fetch from github repo
    spec: https://github.com/my-org/my-repo/blob/main/api/specs/stripe.yaml
    secrets:
      api_key: xxxxxxxxxxxxxxx

  my_api:
    type: api
    spec: file:///path/to/my_api.yaml  # read from local file
    secrets:
      account_id: xxxxxxxxxxx
      token: xxxxxxxxxxxxxxx

  my_other_api:
    type: api
    spec: aws_s3/path/to/my_other_api.yaml  # fetches from your s3 connection
```

You can also use an environment variable (YAML or JSON format):

```shell
# Windows Powershell
$env:DBT_CLOUD_API='{ type: api, spec: https://github.com/slingdata-io/sling-cli/blob/main/api/specs/dbt_cloud.yaml, secrets: { account_id: xxxxx, api_token: xxxxxxx } }'

# Linux or Mac
export DBT_CLOUD_API='{ type: api, spec: https://github.com/slingdata-io/sling-cli/blob/main/api/specs/dbt_cloud.yaml, secrets: { account_id: xxxxx, api_token: xxxxxxx } }'
```

The connection should show up like the others:

```shell
# List all your connections
$ sling conns list
+----------------+-----------------+----------------+
| CONN NAME      | CONN TYPE       | SOURCE         |
+----------------+-----------------+----------------+
| POSTGRES       | DB - PostgreSQL | sling env yaml |
| SNOWFLAKE      | DB - Snowflake  | sling env yaml |
| STRIPE         | API - Spec      | sling env yaml |
| MY_API         | API - Spec      | sling env yaml |
| MY_OTHER_API   | API - Spec      | sling env yaml |
| DBT_CLOUD_API  | API - Spec      | env variable   |
+----------------+-----------------+----------------+
```

## Using the Test Command

Sling provides the `conns test` command to verify your API connection and test individual endpoints.

### Testing the Connection

```bash
# Test that your connection is properly configured
sling conns test API_CONNECTION_NAME

# Example testing a Stripe connection
sling conns test STRIPE
```

### Testing Specific Endpoints

```bash
# Test specific endpoints
sling conns test API_CONNECTION_NAME --endpoints endpoint1,endpoint2

# Example testing specific Stripe endpoints
sling conns test STRIPE --endpoints customer,charge
```

### Discovering Available Endpoints

To see the available endpoints in your API spec:

```bash
# List all endpoints in your API spec
sling conns discover API_CONNECTION_NAME

# Example for Shopify
sling conns discover STRIPE
```

This command is particularly useful for verifying that all your endpoints are correctly defined and visible to Sling.

## Debugging Tools

Sling offers two levels of debug output to help diagnose issues with your API specs.

### Debug Flag

The `--debug` flag provides basic information about request flows, pagination, and data processing:

```bash
# Basic debugging information
sling conns test STRIPE --endpoints customer --debug
```

Debug output includes:

* API requests being made
* Response status codes
* Record counts
* Pagination details
* State variable changes

### Trace Flag

For more detailed debugging, use the `--trace` flag:

```bash
# Detailed trace information
sling conns test STRIPE --endpoints customer --trace
```

The trace output includes everything from debug plus:

* Full request headers and parameters
* Response headers
* JSON response bodies (truncated for large responses)
* Detailed expression evaluation
* Auth token refresh events
* Queue operations

> ⚠️ **Warning:** The `--trace` flag may expose sensitive information in logs, such as authorization tokens. Use carefully and don't share unredacted logs.

### Examining Request and Response Flow

With trace enabled, you can see the complete flow of HTTP requests and responses. This is invaluable for debugging pagination, authentication, or data extraction issues:

```
2025-04-27 07:45:39 DBG iteration 1 initial state: {"base_url":"https://api.stripe.com/v1","created_gte":"1743158739","limit":100,"starting_after":null}
2025-04-27 07:45:39 DBG r.0001.jlw GET @ https://api.stripe.com/v1/customers?limit=100
2025-04-27 07:45:39 TRC r.0001.jlw request: {"method":"GET","url":"https://api.stripe.com/v1/customers?limit=100","headers":{"accept":"application/json","authorization":"Bearer xxxxxxxx","content-type":"application/json","stripe-version":"2023-10-16"},"payload":null,"attempts":1}
2025-04-27 07:45:40 DBG r.0001.jlw   response code=200 content-type=application/json duration=839ms size=36kB
2025-04-27 07:45:40 TRC r.0001.jlw response: {"headers":{"access-control-allow-credentials":"true","access-control-allow-methods":"GET, HEAD, PUT, PATCH, POST, DELETE","access-control-allow-origin":"*","access-control-expose-headers":"Request-Id, Stripe-Manage-Version, Stripe-Should-Retry, X-Stripe-External-Auth-Required, X-Stripe-Privileged-Session-Required","access-control-max-age":"300","cache-control":"no-cache, no-store","content-length":"35925","content-security-policy":"base-uri 'none'; default-src 'none'; form-action 'none'; frame-ancestors 'none'; img-src 'self'; script-src 'self' 'report-sample'; style-src 'self'; worker-src 'none'; upgrade-insecure-requests},"records":39,"size":35925,"status":200}
2025-04-27 07:45:40 DBG iteration 1 ending state: {"base_url":"https://api.stripe.com/v1","created_gte":"1743158739","last_id":"cus_xxxxxxxxxx","limit":100,"starting_after":null}
```

## Common Issues

### Authentication Problems

If you're getting `401 Unauthorized` or `403 Forbidden` responses:

```bash
# Check your authentication configuration
sling conns test API_NAME --debug
```

Common fixes:

* Verify the secrets such as API keys or tokens in your environment file
* Check for correct authentication type (bearer, basic, oauth2)
* Ensure required scopes are included for OAuth2

### Pagination Issues

If your endpoint doesn't retrieve all expected data:

```bash
# Trace pagination behavior
sling conns test API_NAME --endpoints ENDPOINT_NAME --trace
```

Look for:

* `stop_condition` evaluation results
* `next_state` changes between requests
* Response headers for Link-based pagination
* `has_more` flags in response bodies

### JMESPath Extraction Problems

If your records aren't being properly extracted:

```bash
# Trace with attention to record extraction
sling conns test API_NAME --endpoints ENDPOINT_NAME --trace
```

Look for:

* Complete response JSON to verify the correct path
* JMESPath extraction results
* Record counts in your output

## Testing Best Practices

### 1. Test Incrementally

When building complex API specs:

1. Start by testing basic authentication
2. Test a simple endpoint without pagination
3. Add and test pagination
4. Test one endpoint that uses iteration
5. Test queue-based workflows with multiple endpoints
6. Finally, test complex transformations and processors

### 2. Limit Data During Testing

Use these techniques to limit data volume during testing:

Set the environment variable `SLING_TEST_ENDPOINT_LIMIT`:

```bash
# Windows Powershell
$env:SLING_TEST_ENDPOINT_LIMIT='30'

# Linux or Mac
export SLING_TEST_ENDPOINT_LIMIT=30
```

```yaml
# In your endpoint definition
response:
  records:
    # Limit total records processed during testing
    limit: 10
```

Or use date filtering if the API supports it:

```yaml
# Example from Shopify endpoint
request:
  parameters:
    # Limit to recent data only
    updated_at_min: '{date_format(date_add(now(), -2, "day"), "%Y-%m-%dT%H:%M:%S%z")}'
```

### 3. Use Replication Files for Full Testing

Create a replication YAML file to test the flow from API to your database:

```yaml
# Example stripe_test.yaml
source: stripe_sling
target: postgres

defaults:
  mode: full-refresh  # Use for testing instead of incremental
  object: apis.stripe_test_{stream_name}  # Use test schema

streams:
  # Test specific streams
  customer:
  charge:

env:
  SLING_LOADED_AT_COLUMN: timestamp
```

Run with:

```bash
sling run -r stripe_test.yaml --debug
```

### 4. Create Environment Variables for Testing

For testing different scenarios:

```bash
# Test with different time ranges
CREATED_GTE=1659312000 sling conns test STRIPE --endpoints charge --debug

# Test with specific user
GITHUB_USERNAME=test-user sling conns test GITHUB --endpoints repos --debug
```

## Real-World Examples

### Testing Stripe Endpoints

This example tests the Stripe customer and balance transaction endpoints:

```bash
# Test specific Stripe endpoints
sling conns test STRIPE --endpoints customer,customer_balance_transaction --debug
```

The corresponding replication file would look like:

```yaml
# r.61.stripe.yaml
source: stripe_sling
target: postgres

defaults:
  mode: incremental
  object: apis.{source_name}_{stream_name}

  source_options:
    flatten: 1 # flatten records 1 level only

streams:
  '*':

env:
  SLING_STATE: postgres/sling_state.stripe # one state table per replication
  SLING_LOADED_AT_COLUMN: timestamp
```

> 💡 **Tip:** When developing a complex API spec, maintain a test script with your commonly used test commands for quick iteration.


# Troubleshooting

Common errors and debugging tips for Sling API specifications

This guide covers common errors you may encounter when working with Sling API specifications and how to resolve them.

## Common Error Messages

### Endpoint Errors

| Error Message                          | Cause                                         | Solution                                                                        |
| -------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------- |
| `endpoint not found: X`                | The endpoint name doesn't exist in the spec   | Check spelling, run `sling conns discover CONN_NAME` to see available endpoints |
| `endpoint is disabled in spec`         | Endpoint has `disabled: true`                 | Remove the `disabled` field or set it to `false`                                |
| `duplicate endpoint name generated: X` | Dynamic endpoints created the same name twice | Ensure your dynamic endpoint name template produces unique names                |

### Queue Errors

| Error Message                            | Cause                                         | Solution                                                        |
| ---------------------------------------- | --------------------------------------------- | --------------------------------------------------------------- |
| `did not declare queue X in queues list` | Queue is used in a processor but not declared | Add the queue name to the top-level `queues:` list in your spec |

### Authentication Errors

| Error Message                                                              | Cause                                   | Solution                                                              |
| -------------------------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------- |
| `could not authenticate`                                                   | Authentication configuration is invalid | Verify secrets, tokens, and credentials in your env.yaml              |
| `unsupported OAuth2 flow: X`                                               | Invalid OAuth2 flow type specified      | Use one of: `client_credentials`, `authorization_code`, `device_code` |
| `client_secret is required for client_credentials flow`                    | Missing OAuth2 client secret            | Add `client_secret` to your connection secrets                        |
| `authorization_url or derived URL is required for authorization_code flow` | Missing authorization URL               | Add `authorization_url` to your authentication config                 |
| `device_auth_url or derived URL is required for device_code flow`          | Missing device auth URL                 | Add `device_auth_url` to your authentication config                   |
| `failed to authenticate`                                                   | Authentication request failed           | Check API credentials, verify auth URLs, enable `--trace` for details |

### Response Processing Errors

| Error Message                                     | Cause                                      | Solution                                                                     |
| ------------------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------- |
| `need at least 2 lines to build records from csv` | CSV response is empty or only has a header | Ensure the API returns a header row plus at least one data row               |
| `could not evaluate jmespath`                     | Invalid JMESPath expression                | Test your expression with an online JMESPath tester; check for syntax errors |
| `error converting record to map`                  | Response data structure is unexpected      | Verify the JMESPath returns an array of objects, not primitive values        |

### Request & Iteration Errors

| Error Message                                   | Cause                                       | Solution                                                           |
| ----------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------ |
| `empty request url`                             | URL is not set or evaluated to empty        | Check `request.url` and ensure state variables are defined         |
| `loop expression is not an array or a queue`    | `iterate.over` doesn't evaluate to an array | Ensure the expression returns an array or references a valid queue |
| `invalid 'into' variable`                       | Wrong format for iteration variable         | Use format `state.variable_name` (e.g., `state.current_id`)        |
| `request loop into value must be state.<field>` | Iteration variable not in state namespace   | Prefix with `state.` (e.g., `state.item` not just `item`)          |

### Processor Errors

| Error Message                                              | Cause                                     | Solution                                                                      |
| ---------------------------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------- |
| `empty response processor expression`                      | Processor has no `expression` field       | Add an `expression` to the processor                                          |
| `empty response processor output`                          | Processor has no `output` field           | Add an `output` destination (e.g., `record.field`, `state.var`)               |
| `invalid aggregation type`                                 | Unknown aggregation specified             | Use one of: `maximum`, `minimum`, `first`, `last`, `collect`                  |
| `cannot write aggregated value to queue`                   | Trying to aggregate into a queue          | Remove `aggregation` when writing to queues; queues receive individual values |
| `env. and context.store. outputs require aggregation type` | Missing aggregation for env/store outputs | Add an `aggregation` field (e.g., `aggregation: last`)                        |

### Rule & Retry Errors

| Error Message                       | Cause                     | Solution                                                                    |
| ----------------------------------- | ------------------------- | --------------------------------------------------------------------------- |
| `number of attempts exhausted: X/Y` | All retry attempts failed | Check API availability, increase `max_attempts`, or review backoff strategy |
| `rule met to fail: <condition>`     | A fail rule was triggered | Check response status code; verify authentication and request parameters    |

### License Errors

| Error Message                                                | Cause                            | Solution                                                      |
| ------------------------------------------------------------ | -------------------------------- | ------------------------------------------------------------- |
| `please use the official sling-cli release for reading APIs` | CLI Pro token missing or invalid | Get a CLI Pro token from [slingdata.io](https://slingdata.io) |

## Debugging Techniques

### Using Debug and Trace Flags

```bash
# Basic debugging - shows request flow and status codes
sling conns test MY_API --endpoints endpoint_name --debug

# Detailed tracing - shows full request/response details
sling conns test MY_API --endpoints endpoint_name --trace
```

**Debug output includes:**

* API requests being made
* Response status codes
* Record counts
* Pagination details
* State variable changes

**Trace output adds:**

* Full request headers and parameters
* Response headers
* JSON response bodies (truncated for large responses)
* Detailed expression evaluation
* Auth token refresh events

> **Warning:** The `--trace` flag may expose sensitive information like authorization tokens. Don't share unredacted logs.

### Testing Individual Endpoints

```bash
# List all available endpoints
sling conns discover MY_API

# Test a specific endpoint
sling conns test MY_API --endpoints users

# Test multiple endpoints
sling conns test MY_API --endpoints users,orders,products

# Test with pattern matching
sling conns test MY_API --endpoints "user_*"
```

### Limiting Data During Testing

Set an environment variable to limit records:

```bash
export SLING_TEST_ENDPOINT_LIMIT=10
sling conns test MY_API --endpoints large_endpoint
```

Or use the `limit` property in your endpoint:

```yaml
response:
  records:
    jmespath: "data[]"
    limit: 10  # Only process 10 records during testing
```

## Authentication Troubleshooting

### OAuth2 Issues

**Token not refreshing:**

* Verify `authentication_url` is correct
* Check that scopes are sufficient
* Tokens are stored in `~/.sling/api/tokens/{connection_name}.json` - delete to force re-auth

**Authorization code flow not working:**

* Ensure `redirect_uri` matches your OAuth app configuration
* Check `authorization_url` is correct
* Verify scopes include required permissions

**Device code flow timing out:**

* The user must complete authorization in the browser within the time limit
* Verify `device_auth_url` is correct

### Static/Bearer Token Issues

* Verify the token hasn't expired
* Check token format (some APIs require `Bearer` prefix, others don't)
* Ensure secrets are correctly referenced: `{secrets.api_key}`

## Pagination Troubleshooting

### Not fetching all pages

Check your `stop_condition`:

```yaml
pagination:
  stop_condition: "length(response.records) == 0"  # Stops when empty
  # OR
  stop_condition: 'jmespath(response.json, "has_more") == false'  # API-specific
```

Use trace to see pagination flow:

```bash
sling conns test MY_API --endpoints paginated_endpoint --trace
```

Look for:

* `stop_condition` evaluation results
* `next_state` changes between requests

### Cursor/offset not updating

Ensure `next_state` expressions reference the correct response fields:

```yaml
pagination:
  next_state:
    cursor: '{jmespath(response.json, "pagination.next_cursor")}'
    # OR
    offset: '{state.offset + state.limit}'
```

## JMESPath Troubleshooting

### Common JMESPath Issues

**Empty records array:**

* Verify the path to your data in the response JSON
* Test with `[*]` to get all top-level elements
* Use `--trace` to see the actual response structure

**Syntax errors:**

* Use an [online JMESPath tester](https://jmespath.org/) to validate expressions
* Remember: property names with special characters need quotes: `"content-type"`
* Arrays use `[]` suffix: `data.items[]`

**Examples:**

```yaml
# Top-level array
jmespath: "[*]"

# Nested array
jmespath: "data.results[]"

# Filter records
jmespath: "users[?status=='active']"

# Project specific fields
jmespath: "data[].{id: id, name: full_name}"
```

## Getting Help

If you're still having issues:

1. **Check the logs** with `--trace` for detailed error information
2. **Verify your spec** syntax by running `sling conns test`
3. **Test the API directly** with curl or Postman to rule out API issues
4. **Search existing issues** on [GitHub](https://github.com/slingdata-io/sling-cli/issues)
5. **Ask the community** on [Discord](https://discord.gg/q5xtaSNDvp)
6. **Contact support** at <support@slingdata.io>


# Change Capture (CDC)

Continuously replicate row-level changes using Change Data Capture (CDC)

Change Data Capture (CDC) continuously replicates row-level changes (inserts, updates, deletes) from a source database to a target database by reading the database's transaction log. Unlike `incremental` mode which polls for new or updated rows, CDC captures every change as it happens, including deletes.

{% hint style="success" %}
**Advanced Plan Required**: CDC requires a [CLI Pro Max](/sling-cli/cli-pro) token or an [Advanced Platform Plan](/sling-platform/platform).
{% endhint %}

## Supported Sources

| Source                                            | Transaction Log         | Status    |
| ------------------------------------------------- | ----------------------- | --------- |
| [MySQL](/concepts/change-capture/mysql)           | Binary log (binlog)     | Available |
| [MariaDB](/concepts/change-capture/mysql)         | Binary log (binlog)     | Available |
| [PostgreSQL](/concepts/change-capture/postgres)   | Write-Ahead Log (WAL)   | Available |
| [SQL Server](/concepts/change-capture/sql-server) | CDC change tables       | Available |
| [Oracle](/concepts/change-capture/oracle)         | GoldenGate Data Streams | Available |
| [MongoDB](/concepts/change-capture/mongodb)       | Change Streams (oplog)  | Available |

## How It Works

CDC operates in two phases: an **initial load** that copies existing data, followed by **incremental change capture** that streams ongoing changes.

### Phase 1: Initial Load

On the first run for a given stream, Sling performs a full table copy from source to target:

1. **Position capture** — Sling records the current transaction log position (e.g., binlog file + offset) *before* the snapshot begins. This ensures no changes are lost between the snapshot and the first incremental run.
2. **Chunked reading** — Large tables are automatically split into primary-key-range chunks (configurable via `snapshot_chunk_size`). Each chunk is read, written, and checkpointed independently.
3. **Resumability** — If the process is interrupted (crash, timeout, kill), the next run detects the in-progress snapshot and resumes from the last completed chunk. No data is re-read.
4. **Completion** — Once all chunks are written, Sling marks the initial load as complete in the state store.

{% hint style="info" %}
Chunked mode requires an integer-like primary key for range splitting. If the table has no primary key or the PK is non-numeric, Sling falls back to a single-shot full table read automatically.
{% endhint %}

### Phase 2: Incremental Changes

On subsequent runs, Sling reads the source database's transaction log from the last saved position:

1. **Read changes** — Reads inserts, updates, and deletes from the transaction log starting at the saved position, up to `run_max_events` or `run_max_duration`.
2. **Merge to target** — Applies changes to the target table using a merge strategy that handles inserts, updates, and deletes.
3. **Save position** — Persists the new log position in the state store so the next run picks up where this one left off.

Each run is bounded and exits after processing its batch. This makes CDC safe to schedule on a recurring interval (e.g., every 30 seconds or every 5 minutes) via cron or the Sling Platform.

### What Is a CDC Event?

A CDC event corresponds to a single statement in the transaction log, not a single row. A bulk insert like `INSERT INTO t VALUES (...), (...), (...)` produces **one event** containing multiple rows. Similarly, an `UPDATE ... WHERE status = 'old'` that modifies 1,000 rows is a single event with 1,000 row changes.

This means `run_max_events: 10000` does not necessarily equal 10,000 rows — it could represent significantly more rows depending on how the source application writes data. Keep this in mind when tuning `run_max_events` for high-throughput workloads.

### Lifecycle Diagram

```
First Run                          Subsequent Runs
─────────                          ────────────────
┌─────────────────────┐            ┌──────────────────────┐
│ Record log position │            │ Read from saved      │
│ (before snapshot)   │            │ log position         │
└─────────┬───────────┘            └──────────┬───────────┘
          │                                   │
┌─────────▼───────────┐            ┌──────────▼───────────┐
│ Read chunk 1        │            │ Capture changes      │
│ Write to target     │            │ (inserts, updates,   │
│ Checkpoint          │            │  deletes)            │
└─────────┬───────────┘            └──────────┬───────────┘
          │                                   │
┌─────────▼───────────┐            ┌──────────▼───────────┐
│ Read chunk 2...N    │            │ Merge into target    │
│ Write + checkpoint  │            │ table                │
└─────────┬───────────┘            └──────────┬───────────┘
          │                                   │
┌─────────▼───────────┐            ┌──────────▼───────────┐
│ Mark initial load   │            │ Save new log         │
│ complete            │            │ position             │
└─────────────────────┘            └──────────────────────┘
```

## Replication Structure

CDC is configured using `mode: change-capture` in a standard Sling replication file. CDC-specific options go under `change_capture_options`:

```yaml
source: MY_SOURCE
target: MY_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    run_max_events: 10000       # max change events per run
    run_max_duration: 10m       # max wall-clock time per run
    soft_delete: false          # keep deleted rows marked with _sling_synced_op='D'
    snapshot_start: now         # 'now' or 'beginning' for the very first run
    snapshot_chunk_size: 100000 # rows per chunk during initial snapshot
    snapshot_run_duration: 30m  # cap time spent on initial snapshot per run
    # replay_from: "2025-01-01T00:00:00Z"  # rewind to re-process from a point in time (beta)
    # slot_level: shared        # 'shared' (one slot for all streams) or 'stream' (one per table)
    retry_attempts: 3           # retries on transient failures
    retry_delay: 5s             # delay between retries

streams:
  my_database.users:

  my_database.orders:
    change_capture_options:
      soft_delete: true         # per-stream override: preserve deleted rows
      run_max_events: 50000     # higher event budget for a busy table
      run_max_duration: 5m      # shorter run window

  my_database.products:
    change_capture_options:
      snapshot_start: beginning         # capture full history from earliest log position
      snapshot_chunk_size: 50000        # smaller chunks for a wide table
      snapshot_run_duration: 30m        # resume snapshot across runs
```

Options set in `defaults.change_capture_options` apply to all streams. Per-stream `change_capture_options` override the defaults.

## Options Reference

<table data-full-width="false"><thead><tr><th>Key</th><th>Description</th></tr></thead><tbody><tr><td><code>run_max_events</code></td><td>Maximum number of change events to process per run. When this limit is reached, Sling saves the position and exits. Default is <code>10000</code>.</td></tr><tr><td><code>run_max_duration</code></td><td>Maximum duration per run (e.g., <code>30s</code>, <code>10m</code>, <code>1h</code>). If no events arrive within this window, the run completes with zero changes. Default is <code>10m</code>.</td></tr><tr><td><code>soft_delete</code></td><td>When <code>true</code>, DELETE events mark the row with <code>_sling_synced_op = 'D'</code> and update <code>_sling_synced_at</code> instead of removing the row. Default is <code>false</code>.</td></tr><tr><td><code>snapshot_start</code></td><td>Where to start reading the transaction log on the very first run. Default is <code>now</code>. Use <code>beginning</code> to read from the earliest available log position.</td></tr><tr><td><code>snapshot_chunk_size</code></td><td>Number of rows per chunk during the initial snapshot. Default is <code>100000</code>.</td></tr><tr><td><code>snapshot_run_duration</code></td><td>Maximum time to spend on the initial snapshot per run (e.g., <code>30m</code>, <code>1h</code>). When the budget is exhausted, Sling exits cleanly after the current chunk and resumes on the next run. Default: no limit.</td></tr><tr><td><code>replay_from</code></td><td>Rewind the CDC position to re-process changes from an earlier point. Accepts source-specific formats (e.g., RFC 3339 timestamp, binlog position, GTID set). Applied once per unique value.</td></tr><tr><td><code>slot_level</code></td><td>Controls how the source's replication reader is scoped across the streams in a replication. <code>shared</code> uses a single replication slot (PostgreSQL) or binlog reader (MySQL) for all streams in the group, so every stream converges to the same unified transaction-log position — giving a point-in-time-consistent view across tables and reading the log only once. <code>stream</code> uses one slot/reader per table (independent positions). Defaults to <code>shared</code> on sources that support a shared reader (<strong>PostgreSQL</strong>, <strong>MySQL</strong>/MariaDB) and <code>stream</code> on all others. Setting <code>shared</code> on a source without a shared reader (e.g. SQL Server, Oracle, MongoDB) is ignored and falls back to per-stream.</td></tr><tr><td><code>change_feed</code></td><td>Names the pre-provisioned, DBA-managed server-side CDC object Sling reads from. Engine-neutral: it maps to a PostgreSQL <strong>publication</strong>, a SQL Server <strong>capture instance</strong>, or an Oracle GoldenGate <strong>Data Stream</strong>. Sling never creates this object — a DBA provisions it once so the Sling role can run with least privilege (read-only). If omitted, PostgreSQL/SQL Server expect the conventional per-stream object and error with the exact DDL to run if it is missing; Oracle falls back to the <code>gg_stream</code> connection property. MySQL/MariaDB/MongoDB ignore it (their change stream has no named server object). See the per-source <a href="/pages/HTYVG0TPFEYYPhPKw32W">setup</a> guides.</td></tr><tr><td><code>retry_attempts</code></td><td>Number of retry attempts on transient failures. Default is <code>3</code>.</td></tr><tr><td><code>retry_delay</code></td><td>Delay between retries. Default is <code>5s</code>.</td></tr></tbody></table>

## CDC Setup & Permissions (DBA-owned)

Sling **reads** from change data capture; it does not provision it. Server-side CDC objects — a PostgreSQL publication, a SQL Server capture instance, or an Oracle GoldenGate Data Stream — are created once by a database administrator. This keeps the Sling connection role **least-privilege and read-only**: it never runs `ALTER TABLE`, `CREATE PUBLICATION`, or `sp_cdc_enable_*` against your source, so it works on managed databases (Cloud SQL, RDS, Azure SQL) where superuser/owner rights are unavailable.

The engine-neutral [`change_feed`](#options-reference) option names the pre-provisioned object to read from:

```yaml
defaults:
  mode: change-capture
  change_capture_options:
    change_feed: my_cdc_object   # publication / capture instance / GG stream
```

| Source                                            | `change_feed` maps to               | Who provisions it                                    |
| ------------------------------------------------- | ----------------------------------- | ---------------------------------------------------- |
| [PostgreSQL](/concepts/change-capture/postgres)   | Logical replication **publication** | DBA (`CREATE PUBLICATION … FOR TABLE …`)             |
| [SQL Server](/concepts/change-capture/sql-server) | CDC **capture instance**            | DBA (`sys.sp_cdc_enable_db` / `sp_cdc_enable_table`) |
| [Oracle](/concepts/change-capture/oracle)         | GoldenGate **Data Stream**          | DBA (also settable via `gg_stream`)                  |
| [MySQL](/concepts/change-capture/mysql) / MariaDB | binary log (no named object)        | server config (`binlog_format=ROW`)                  |
| [MongoDB](/concepts/change-capture/mongodb)       | change stream (no named object)     | replica set / oplog enabled                          |

If a required object or grant is missing, Sling stops with a clear error containing the **exact DDL** a DBA should run. The Sling role itself needs only the read-level privileges for the engine (e.g. PostgreSQL: `REPLICATION` + `SELECT`; SQL Server: membership in the capture instance's gating role + `SELECT`). See each per-source guide for the precise setup script.

## CDC Metadata Columns

Sling adds three metadata columns to every CDC-managed target table:

| Column             | Type          | Description                                                                                                                                                       |
| ------------------ | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_sling_synced_at` | `timestamptz` | Timestamp when the row was last synced.                                                                                                                           |
| `_sling_synced_op` | `varchar`     | The operation type: `S` (snapshot), `I` (insert), `U` (update), `D` (delete). When `soft_delete: true`, deleted rows are preserved with `_sling_synced_op = 'D'`. |
| `_sling_cdc_seq`   | `bigint`      | Monotonically increasing sequence number for ordering events within and across runs.                                                                              |

## State Management

CDC state is stored in the connection specified by the `SLING_STATE` environment variable. This tracks:

* The current transaction log position
* Whether the initial snapshot is complete
* Checkpoint progress for in-progress snapshots
* Total rows captured

```bash
# Store state in a PostgreSQL table
export SLING_STATE='MY_POSTGRES/sling_state'

# Or store state in a file
export SLING_STATE='MY_AWS/sling_state'
```

{% hint style="warning" %}
The `SLING_STATE` connection must be configured before running CDC replications. Without it, Sling cannot track positions and each run would repeat the initial snapshot. See location string details [here](/sling-cli/environment#location-string).
{% endhint %}

## Scheduling

CDC is designed to be run repeatedly on a schedule. Each run processes a bounded batch of changes (controlled by `run_max_events` and `run_max_duration`) and exits.

Configure the replication in the [Sling Platform](/sling-platform/platform) UI with a schedule interval. The platform handles orchestration, monitoring, and alerting automatically.

![Sling Platform UI](/files/33ReA95FxIhUTrOLvgSy)

## Comparison with Other Modes

| Feature          | `change-capture`    | `incremental`           | `full-refresh`  |
| ---------------- | ------------------- | ----------------------- | --------------- |
| Captures inserts | Yes                 | Yes                     | Yes             |
| Captures updates | Yes                 | Yes (with `update_key`) | Yes             |
| Captures deletes | Yes                 | With `delete_missing`   | Yes             |
| Reads from       | Transaction log     | Table query             | Table query     |
| State tracking   | Log position        | Max update\_key value   | None            |
| Source load      | Minimal (reads log) | Queries table           | Full table scan |
| Initial setup    | Automatic snapshot  | Manual first load       | N/A             |


# MySQL / MariaDB

CDC source setup for MySQL and MariaDB

Sling supports Change Data Capture from MySQL and MariaDB by reading the binary log (binlog). Each run reads row-level inserts, updates, and deletes from the binlog and merges them into the target table.

For general CDC concepts, the two-phase process, and all available options, see the [Change Capture overview](/concepts/change-capture).

## Prerequisites

Ensure your MySQL source has binary logging enabled with row-based format:

```sql
-- Verify binlog is enabled and using ROW format
SHOW VARIABLES LIKE 'log_bin';          -- Must be ON
SHOW VARIABLES LIKE 'binlog_format';    -- Must be ROW
SHOW VARIABLES LIKE 'binlog_row_image'; -- Should be FULL
```

The MySQL user must have the `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges:

```sql
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'sling_user'@'%';
GRANT SELECT ON my_database.* TO 'sling_user'@'%';
```

{% hint style="info" %}
MariaDB uses the same binlog mechanism as MySQL. The same prerequisites apply.
{% endhint %}

## Quick Start

```bash
# Source MySQL
sling conns set MY_MYSQL type=mysql host=mysql.example.com user=sling_user password=secret database=my_database

# Target PostgreSQL
sling conns set MY_POSTGRES type=postgres host=pg.example.com user=postgres password=secret database=analytics

# State store (required for CDC)
export SLING_STATE='MY_POSTGRES/sling_state'
```

```yaml
# replication.yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  my_database.customers:
  my_database.orders:
```

```bash
# First run: performs the initial snapshot
sling run -d replication.yaml

# Subsequent runs: captures and applies changes
sling run -d replication.yaml
```

## Examples

### Large Tables with Custom Chunk Size

For very large tables, adjust the chunk size to control memory usage and checkpointing frequency during the initial snapshot.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000  # 50k rows per chunk

streams:
  my_database.transactions:
    # This 10M-row table will be loaded in ~200 chunks
    # If interrupted, it resumes from the last completed chunk
```

### Time-Bounded Snapshots for Very Large Tables

For tables with hundreds of millions of rows, the initial snapshot can take hours. Use `snapshot_run_duration` to cap how long each run spends on the snapshot. The next run automatically resumes from the last completed chunk.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000
    snapshot_run_duration: 30m  # spend at most 30 minutes per run on the snapshot

streams:
  my_database.huge_events:
    # 500M rows — will take multiple runs to complete the initial load
    # Each run processes ~30 minutes worth of chunks, then exits cleanly
```

### High-Throughput Workloads

For tables with heavy write activity, increase `run_max_events` and `run_max_duration` so each run captures more changes.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: warehouse.{stream_table}
  change_capture_options:
    run_max_events: 50000  # Process up to 50k events per run
    run_max_duration: 60s  # Wait up to 60 seconds for events

streams:
  my_database.click_events:
  my_database.page_views:
```

### Soft Deletes

Keep deleted rows in the target instead of physically removing them. Deleted rows are marked with `_sling_synced_op = 'D'`. Useful for audit trails or when downstream queries need to detect deletions.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    soft_delete: true

streams:
  my_database.customers:
  my_database.subscriptions:
```

When a row is deleted in MySQL, the target row is preserved with `_sling_synced_op` set to `'D'` and `_sling_synced_at` updated to the current timestamp. A subsequent re-insert of the same primary key restores the row with the appropriate operation type.

### Mixed Streams with Per-Stream Overrides

Different tables can have different CDC options.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  # High-volume table: larger batches
  my_database.events:
    change_capture_options:
      run_max_events: 100000
      run_max_duration: 2m

  # Audit table: keep soft deletes
  my_database.user_accounts:
    change_capture_options:
      soft_delete: true

  # Standard table: uses defaults
  my_database.products:
```

### Replay / Backfill from a Point in Time

If target data becomes inconsistent, you can replay changes from an earlier position. The `replay_from` value is applied exactly once per unique value.

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    replay_from: "2025-06-01T00:00:00Z"  # Re-process all changes since June 1

streams:
  my_database.orders:
```

After the replay run completes, remove or change the `replay_from` value. Leaving it unchanged has no effect (it is only applied once).

## Replay Formats

The `replay_from` option accepts these MySQL-specific position formats:

* **RFC 3339 timestamp**: `2025-06-01T00:00:00Z`
* **Binlog file:offset**: `mysql-bin.000003:12345`
* **GTID set**: `cc51b500-0cda-11f1-9e7c-0242ac110002:1-100`

## Binlog Retention

Ensure binlog retention is long enough to cover the maximum gap between CDC runs. If MySQL purges binlogs that Sling hasn't read yet, the run will fail.

```sql
-- Check current retention
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
-- or for older MySQL versions:
SHOW VARIABLES LIKE 'expire_logs_days';

-- Set retention to 7 days (MySQL 8.0+)
SET GLOBAL binlog_expire_logs_seconds = 604800;
```

## Troubleshooting

### "CDC not supported for \<type>"

Ensure your source connection is configured as `type=mysql` or `type=mariadb`.

### Initial snapshot keeps restarting

Ensure `SLING_STATE` is configured. Without state persistence, Sling cannot track that the snapshot completed and will restart it on every run.

### Missing changes after snapshot

This should not happen — Sling captures the binlog position before the snapshot begins. If you suspect missing data, check that the binlog retention period is long enough to cover the snapshot duration:

```sql
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
-- or for older MySQL versions:
SHOW VARIABLES LIKE 'expire_logs_days';
```

### "could not resolve replay\_from position"

The `replay_from` value must be in one of the formats listed in the Replay Formats section above. Ensure the value is valid and the binlog position still exists on the server.

### Binlog purged before CDC could read

If MySQL purges binlogs that Sling hasn't read yet, the run will fail. Increase binlog retention to be longer than the maximum gap between CDC runs:

```sql
-- Set retention to 7 days (MySQL 8.0+)
SET GLOBAL binlog_expire_logs_seconds = 604800;
```


# PostgreSQL

CDC source setup for PostgreSQL

Sling supports Change Data Capture from PostgreSQL by reading the Write-Ahead Log (WAL) via logical replication. Each run reads row-level inserts, updates, and deletes from the WAL and merges them into the target table.

For general CDC concepts, the two-phase process, and all available options, see the [Change Capture overview](/concepts/change-capture).

## Prerequisites

Ensure your PostgreSQL instance has logical replication enabled:

```sql
-- Verify WAL level is set to logical
SHOW wal_level;  -- Must be 'logical'
```

If `wal_level` is not `logical`, update it and restart PostgreSQL:

```sql
ALTER SYSTEM SET wal_level = logical;
-- Restart PostgreSQL for the change to take effect
```

Sling connects with a **least-privilege, read-only** role: it needs only the `REPLICATION` attribute and `SELECT` on the source tables. Sling does not run any DDL against your source — the publication and replica identity are provisioned once by a DBA (see [CDC Setup](#cdc-setup-dba-owned) below).

```sql
-- Grant replication privilege (needed to create/read the replication slot)
ALTER ROLE sling_user REPLICATION;

-- Grant read access to source tables (needed for the initial snapshot)
GRANT SELECT ON ALL TABLES IN SCHEMA public TO sling_user;
```

{% hint style="info" %}
The Sling role does **not** need `ALTER` or ownership on the tables. Replica identity and the publication are set up by a DBA, not by Sling. This is what makes CDC work on managed PostgreSQL (Cloud SQL, RDS, Azure) where superuser/owner rights are unavailable.
{% endhint %}

{% hint style="info" %}
Ensure `pg_hba.conf` allows replication connections for the user and database. For example:

```
host    replication     sling_user      0.0.0.0/0       md5
```

Reload PostgreSQL after editing `pg_hba.conf`.
{% endhint %}

## CDC Setup (DBA-owned)

Sling reads from a pre-provisioned **publication**; it never creates one. A DBA (or a role that owns the tables) runs this setup once, then Sling references it via the [`change_feed`](/concepts/change-capture#options-reference) option.

**1. Create a publication scoped to the tables you want to capture:**

```sql
CREATE PUBLICATION sling_cdc_pub FOR TABLE public.customers, public.orders;
```

Scope the publication to exactly the streams you replicate — avoid `FOR ALL TABLES` unless you truly want every table in scope (it also requires superuser).

**2. Ensure each captured table has a CDC-usable replica identity.** PostgreSQL only includes old-row data in UPDATE and DELETE WAL messages when the table has a replica identity — without one, those changes carry no key and cannot be applied to the target. Tables with a **primary key** already satisfy this under the default replica identity (PostgreSQL logs the key — sufficient for a PK-based merge), so no DDL is needed.

Tables **without** a primary key need one set explicitly. Either form works:

```sql
-- Option A: log the full old row. Always works, but increases WAL volume.
ALTER TABLE public.events REPLICA IDENTITY FULL;

-- Option B: log only a key, via an existing unique index on NOT NULL column(s).
-- Cheaper than FULL. The index must be unique, valid, non-partial and
-- non-deferrable, over NOT NULL columns.
CREATE UNIQUE INDEX events_id_uq ON public.events (id);
ALTER TABLE public.events REPLICA IDENTITY USING INDEX events_id_uq;
```

Sling accepts either, and validates it before streaming. Note that `REPLICA IDENTITY NOTHING` is never usable for CDC — including on tables that have a primary key.

**3. Point Sling at the publication** with `change_feed`:

```yaml
defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    change_feed: sling_cdc_pub   # the DBA-provisioned publication

streams:
  public.customers:
  public.orders:
```

Sling validates the publication exists and covers each configured stream, then creates and manages its own replication slot (which needs only the `REPLICATION` attribute). If the publication is missing, does not cover a table, or a table lacks a usable replica identity, Sling stops with a clear error containing the exact DDL for a DBA to run.

{% hint style="info" %}
If `change_feed` is omitted, Sling expects a publication named after its derived slot (`sling_cdc_<hash>`) and errors with the `CREATE PUBLICATION` DDL if it is not found. Setting `change_feed` explicitly is recommended so the publication name is stable and readable.
{% endhint %}

## Quick Start

```bash
# Source PostgreSQL
sling conns set MY_PG_SOURCE type=postgres host=source.example.com user=sling_user password=secret database=my_database

# Target PostgreSQL
sling conns set MY_PG_TARGET type=postgres host=target.example.com user=postgres password=secret database=analytics

# State store (required for CDC)
export SLING_STATE='MY_PG_TARGET/sling_state'
```

```yaml
# replication.yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  public.customers:
  public.orders:

# or declare state here
env:
  SLING_STATE: MY_PG_TARGET/sling_state
```

```bash
# First run: performs the initial snapshot
sling run -d replication.yaml

# Subsequent runs: captures and applies changes
sling run -d replication.yaml
```

## Examples

### Large Tables with Custom Chunk Size

For very large tables, adjust the chunk size to control memory usage and checkpointing frequency during the initial snapshot.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000  # 50k rows per chunk

streams:
  public.transactions:
    # This 10M-row table will be loaded in ~200 chunks
    # If interrupted, it resumes from the last completed chunk
```

### Time-Bounded Snapshots for Very Large Tables

For tables with hundreds of millions of rows, the initial snapshot can take hours. Use `snapshot_run_duration` to cap how long each run spends on the snapshot. The next run automatically resumes from the last completed chunk.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000
    snapshot_run_duration: 30m  # spend at most 30 minutes per run on the snapshot

streams:
  public.huge_events:
    # 500M rows — will take multiple runs to complete the initial load
    # Each run processes ~30 minutes worth of chunks, then exits cleanly
```

### High-Throughput Workloads

For tables with heavy write activity, increase `run_max_events` and `run_max_duration` so each run captures more changes.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: warehouse.{stream_table}
  change_capture_options:
    run_max_events: 50000  # Process up to 50k events per run
    run_max_duration: 60s  # Wait up to 60 seconds for events

streams:
  public.click_events:
  public.page_views:
```

### Soft Deletes

Keep deleted rows in the target instead of physically removing them. Deleted rows are marked with `_sling_synced_op = 'D'`. Useful for audit trails or when downstream queries need to detect deletions.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    soft_delete: true

streams:
  public.customers:
  public.subscriptions:
```

When a row is deleted in the source, the target row is preserved with `_sling_synced_op` set to `'D'` and `_sling_synced_at` updated to the current timestamp. A subsequent re-insert of the same primary key restores the row with the appropriate operation type.

### Mixed Streams with Per-Stream Overrides

Different tables can have different CDC options.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  # High-volume table: larger batches
  public.events:
    change_capture_options:
      run_max_events: 100000
      run_max_duration: 2m

  # Audit table: keep soft deletes
  public.user_accounts:
    change_capture_options:
      soft_delete: true

  # Standard table: uses defaults
  public.products:
```

### Replay / Backfill from a Point in Time

If target data becomes inconsistent, you can replay changes from an earlier position. The `replay_from` value is applied exactly once per unique value.

```yaml
source: MY_PG_SOURCE
target: MY_PG_TARGET

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    replay_from: "0/16B3748"  # Re-process all changes from this WAL position

streams:
  public.orders:
```

After the replay run completes, remove or change the `replay_from` value. Leaving it unchanged has no effect (it is only applied once).

## Replay Formats

The `replay_from` option accepts these PostgreSQL-specific position formats:

* **RFC 3339 timestamp**: `2025-06-01T00:00:00Z` — resolves to the replication slot's `restart_lsn` (best-effort; PostgreSQL WAL does not support direct timestamp seeking)
* **PostgreSQL LSN**: `0/16B3748` — a Write-Ahead Log position in hex format

## Replication Slots and Publications

The **publication** is DBA-provisioned (see [CDC Setup](#cdc-setup-dba-owned)) and shared by all streams that reference the same `change_feed`. The **replication slot** is created and managed by Sling — it needs only the `REPLICATION` attribute, never ownership. How the slot is scoped depends on the [`slot_level`](/concepts/change-capture#options-reference) option, which defaults to `shared` for PostgreSQL.

### `slot_level: shared` (default)

All streams in the replication share **one** replication slot (and read from the one `change_feed` publication):

* **Replication slot**: Created once with a deterministic name (`sling_cdc_<hash>`) derived from the host, port, and database — independent of the table list. Every stream reads from this single slot, so they all converge to the same unified WAL position (LSN), giving a point-in-time-consistent view across tables. The WAL is decoded only once per run regardless of how many tables are captured.
* **Publication**: The one named by `change_feed`. It must cover every configured stream; Sling validates this and errors with `ALTER PUBLICATION … ADD TABLE` if a table is missing.

Because the slot is shared across streams, it is **not** dropped automatically when you remove a single stream — removing it would break the other streams reading from it. To tear down a shared slot manually:

```sql
SELECT pg_drop_replication_slot('sling_cdc_<hash>');
-- The publication is DBA-owned; drop it only if no stream still uses it:
-- DROP PUBLICATION sling_cdc_pub;
```

### `slot_level: stream`

Each captured table gets its **own** slot (all still reading from the DBA's publication):

* **Replication slot**: Named `sling_cdc_<hash>` where the hash is derived from the host, port, and that table's name, so each stream is fully isolated and advances independently.
* **Publication**: The `change_feed` publication, which must include the table.

Per-table slots are cleaned up automatically when you remove the corresponding CDC stream. Use this mode when you want each table to advance fully independently rather than sharing a unified position.

{% hint style="info" %}
Cleanup drops **only** the Sling-managed replication slot. The publication and each table's replica identity are DBA-owned, so Sling leaves them exactly as it found them — deactivating CDC never issues DDL against your tables.
{% endhint %}

## WAL Retention

Replication slots prevent PostgreSQL from recycling WAL segments that haven't been consumed yet. If CDC runs are paused for an extended period, WAL can accumulate and consume significant disk space.

To protect against unbounded WAL growth, set a safety limit:

```sql
-- Cap WAL retained per slot to 10 GB (PostgreSQL 13+)
ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';
SELECT pg_reload_conf();
```

{% hint style="warning" %}
If a replication slot exceeds `max_slot_wal_keep_size`, PostgreSQL may invalidate the slot. Sling will detect this and automatically recreate the slot with a fresh initial snapshot on the next run.
{% endhint %}

## Troubleshooting

### "wal\_level must be 'logical'"

The PostgreSQL instance is not configured for logical replication. Set `wal_level` and restart:

```sql
ALTER SYSTEM SET wal_level = logical;
-- Then restart PostgreSQL
```

### "current user does not have REPLICATION privilege"

Grant the `REPLICATION` attribute to the CDC user:

```sql
ALTER ROLE sling_user REPLICATION;
```

### "could not open replication connection"

Ensure `pg_hba.conf` allows replication connections for the user and database. Add a line like:

```
host    replication     sling_user      0.0.0.0/0       md5
```

Then reload PostgreSQL configuration.

### "CDC publication … does not exist"

The publication named by `change_feed` (or the derived default) has not been created. Sling does not create it — a DBA must, once:

```sql
CREATE PUBLICATION sling_cdc_pub FOR TABLE public.customers, public.orders;
```

Then set `change_feed: sling_cdc_pub` in `change_capture_options`.

### "table … is not a member of publication"

A configured stream is not covered by the publication. A DBA adds it:

```sql
ALTER PUBLICATION sling_cdc_pub ADD TABLE public.orders;
```

### "table … has no CDC-usable replica identity"

PostgreSQL is not logging a key for UPDATE/DELETE on the table, so those changes cannot be applied to the target. The error reports the table's current `relreplident` value. Causes:

| Cause                                                                                                              | Fix (run once, by a table owner)                                              |
| ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| No primary key, replica identity left at `DEFAULT` (`d`)                                                           | `ALTER TABLE … REPLICA IDENTITY FULL;` or nominate a unique index (see below) |
| Replica identity explicitly set to `NOTHING` (`n`) — applies even if the table has a primary key                   | `ALTER TABLE … REPLICA IDENTITY DEFAULT;` (PK tables) or `FULL`               |
| `USING INDEX` (`i`) whose nominated index was dropped, invalidated, or is partial/deferrable/over nullable columns | Re-nominate a usable index, or switch to `FULL`                               |

```sql
-- Log the full old row:
ALTER TABLE public.events REPLICA IDENTITY FULL;

-- Or log only a key, using a unique index on NOT NULL column(s):
ALTER TABLE public.events REPLICA IDENTITY USING INDEX events_id_uq;
```

Tables that have a primary key and are left at the default replica identity do **not** need any of this — PostgreSQL already logs the key for a PK-based merge.

### "replication slot is already active"

Another consumer (another Sling process or a different tool) is connected to the same replication slot. Ensure only one CDC process is running per stream group at a time.

### Initial snapshot keeps restarting

Ensure `SLING_STATE` is configured. Without state persistence, Sling cannot track that the snapshot completed and will restart it on every run.

### Large WAL lag warning

If Sling warns about WAL lag exceeding 5 GB, the CDC consumer is falling behind. Increase the run frequency or raise `run_max_events` / `run_max_duration` to process more changes per run. Consider setting `max_slot_wal_keep_size` as a safety net to prevent unbounded disk usage.


# SQL Server

CDC source setup for SQL Server

Sling supports Change Data Capture from SQL Server by reading the native CDC change tables. SQL Server CDC is query-based: the SQL Server Agent captures row-level changes from the transaction log into system-managed change tables, and Sling reads those tables on each run using `cdc.fn_cdc_get_all_changes_*()`.

For general CDC concepts, the two-phase process, and all available options, see the [Change Capture overview](/concepts/change-capture).

## Prerequisites

### 1. Supported Editions

SQL Server CDC is available on **Enterprise**, **Standard** (2016 SP1+), and **Developer** editions. **Express edition does not support CDC.**

For Azure:

* **Azure SQL Database**: CDC is supported on **S3 (Standard) tier or higher**. Basic/S0/S1/S2 tiers are not recommended.
* **Azure SQL Managed Instance**: Full CDC support (same as on-premises).

### 2. Enable CDC on the Database

{% hint style="info" %}
CDC setup is **DBA-owned**. Sling reads from CDC change tables but never enables CDC or creates capture instances against your source — so the Sling role can stay least-privilege and read-only. A DBA runs the steps below once.
{% endhint %}

```sql
USE my_database;
GO
EXEC sys.sp_cdc_enable_db;
GO
```

Verify CDC is enabled:

```sql
SELECT name, is_cdc_enabled FROM sys.databases WHERE name = DB_NAME();
-- is_cdc_enabled should be 1
```

### 3. SQL Server Agent Must Be Running

On-premises SQL Server requires SQL Server Agent to be running. When CDC is enabled, SQL Server creates two Agent jobs:

* **`cdc.<dbname>_capture`** — reads the transaction log and populates change tables.
* **`cdc.<dbname>_cleanup`** — purges expired change data (default retention: 3 days).

If Agent is stopped, changes accumulate in the transaction log but are **not** captured into change tables.

```sql
-- Check if the CDC capture job exists
SELECT * FROM msdb.dbo.cdc_jobs WHERE job_type = 'capture';
```

{% hint style="info" %}
Azure SQL Database does not use SQL Server Agent — it has a built-in CDC scheduler that runs automatically. No action is needed.
{% endhint %}

### 4. User Permissions

Because CDC setup is done by a DBA (not Sling), the Sling login is **read-only**. It needs only:

* **`SELECT`** on the source tables (for the initial snapshot), and
* **`SELECT`** on the `cdc` schema change tables — granted via the capture instance's gating role (`@role_name`) if one is set, or directly.

```sql
-- Create a read-only CDC user
CREATE LOGIN sling_user WITH PASSWORD = 'YourStrongPassword!';
USE my_database;
CREATE USER sling_user FOR LOGIN sling_user;

-- Read access to the source tables and the CDC change tables
GRANT SELECT ON SCHEMA::dbo TO sling_user;
GRANT SELECT ON SCHEMA::cdc TO sling_user;
```

{% hint style="info" %}
The Sling user does **not** need `db_owner`. That role is only required for the one-time `sp_cdc_enable_db` / `sp_cdc_enable_table` setup, which a DBA performs — not Sling. This keeps CDC workable on locked-down and managed instances.
{% endhint %}

### 5. Enable CDC on Tables

A DBA enables CDC on each table you want to capture. This creates a **capture instance** (default name `<schema>_<table>`) — the object Sling reads from.

```sql
EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name   = N'my_table',
    @role_name     = NULL,          -- or a gating role the Sling user is a member of
    @supports_net_changes = 1;
GO
```

Verify table-level CDC:

```sql
-- List all CDC-enabled tables
SELECT name, is_tracked_by_cdc FROM sys.tables WHERE is_tracked_by_cdc = 1;

-- List all capture instances
EXEC sys.sp_cdc_help_change_data_capture;
```

By default Sling reads the conventional `<schema>_<table>` capture instance for each stream. If a table has a differently-named or secondary capture instance, point Sling at it with [`change_feed`](/concepts/change-capture#options-reference):

```yaml
defaults:
  mode: change-capture
  change_capture_options:
    change_feed: dbo_my_table_v2   # explicit capture instance name

streams:
  dbo.my_table:
```

If a configured table is not tracked by CDC (or the named `change_feed` capture instance does not exist), Sling stops with a clear error containing the exact `sp_cdc_enable_table` DDL for a DBA to run.

## Quick Start

```bash
# Source SQL Server
sling conns set MY_MSSQL type=sqlserver host=sqlserver.example.com user=sling_user password='YourStrongPassword!' database=my_database

# Target PostgreSQL
sling conns set MY_POSTGRES type=postgres host=pg.example.com user=postgres password=secret database=analytics

# State store (required for CDC)
export SLING_STATE='MY_POSTGRES/sling_state'
```

```yaml
# replication.yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  dbo.customers:
  dbo.orders:
```

```bash
# First run: performs the initial snapshot
sling run -r replication.yaml

# Subsequent runs: captures and applies changes
sling run -r replication.yaml
```

## How It Works

Unlike MySQL (binlog streaming) and PostgreSQL (WAL logical replication), SQL Server CDC is **query-based**:

1. The SQL Server Agent continuously reads the transaction log and writes row-level changes into CDC change tables (`cdc.dbo_<table>_CT`).
2. On each run, Sling queries `cdc.fn_cdc_get_all_changes_<capture_instance>()` to read changes between the last saved LSN and the current max LSN.
3. Changes are merged into the target table using the standard CDC merge strategy.
4. The new LSN position is saved in the state store.

Because CDC is query-based, each run reads **all available changes** in the LSN range rather than streaming. This means:

* There is no persistent connection or replication slot (unlike PostgreSQL).
* The CDC Agent must have time to process the transaction log before Sling can see the changes. There is typically a few seconds of latency between a DML commit and the change appearing in the change table.

## Examples

### Large Tables with Custom Chunk Size

For very large tables, adjust the chunk size to control memory usage and checkpointing frequency during the initial snapshot.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000  # 50k rows per chunk

streams:
  dbo.transactions:
    # This 10M-row table will be loaded in ~200 chunks
    # If interrupted, it resumes from the last completed chunk
```

### Time-Bounded Snapshots for Very Large Tables

For tables with hundreds of millions of rows, the initial snapshot can take hours. Use `snapshot_run_duration` to cap how long each run spends on the snapshot. The next run automatically resumes from the last completed chunk.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000
    snapshot_run_duration: 30m  # spend at most 30 minutes per run on the snapshot

streams:
  dbo.huge_events:
    # 500M rows — will take multiple runs to complete the initial load
    # Each run processes ~30 minutes worth of chunks, then exits cleanly
```

### High-Throughput Workloads

For tables with heavy write activity, increase `run_max_events` and `run_max_duration` so each run captures more changes.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: warehouse.{stream_table}
  change_capture_options:
    run_max_events: 50000  # Process up to 50k events per run
    run_max_duration: 60s  # Wait up to 60 seconds for events

streams:
  dbo.click_events:
  dbo.page_views:
```

### Soft Deletes

Keep deleted rows in the target instead of physically removing them. Deleted rows are marked with `_sling_synced_op = 'D'`. Useful for audit trails or when downstream queries need to detect deletions.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    soft_delete: true

streams:
  dbo.customers:
  dbo.subscriptions:
```

When a row is deleted in the source, the target row is preserved with `_sling_synced_op` set to `'D'` and `_sling_synced_at` updated to the current timestamp. A subsequent re-insert of the same primary key restores the row with the appropriate operation type.

### Mixed Streams with Per-Stream Overrides

Different tables can have different CDC options.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  # High-volume table: larger batches
  dbo.events:
    change_capture_options:
      run_max_events: 100000
      run_max_duration: 2m

  # Audit table: keep soft deletes
  dbo.user_accounts:
    change_capture_options:
      soft_delete: true

  # Standard table: uses defaults
  dbo.products:
```

### Replay / Backfill from a Point in Time

If target data becomes inconsistent, you can replay changes from an earlier position. The `replay_from` value is applied exactly once per unique value.

```yaml
source: MY_MSSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    replay_from: "2025-06-01T00:00:00Z"  # Re-process all changes since June 1

streams:
  dbo.orders:
```

After the replay run completes, remove or change the `replay_from` value. Leaving it unchanged has no effect (it is only applied once).

## Replay Formats

The `replay_from` option accepts these SQL Server-specific position formats:

* **RFC 3339 timestamp**: `2025-06-01T00:00:00Z` — resolved to an LSN using `sys.fn_cdc_map_time_to_lsn()`

## CDC Retention

SQL Server's CDC cleanup job purges change data older than the retention period (default: **3 days / 4320 minutes**). Ensure retention is long enough to cover the maximum gap between CDC runs.

```sql
-- Check current retention (in minutes)
SELECT retention FROM msdb.dbo.cdc_jobs WHERE job_type = 'cleanup';

-- Increase retention to 7 days (10080 minutes)
EXEC sys.sp_cdc_change_job
    @job_type = N'cleanup',
    @retention = 10080;
GO

-- Restart the cleanup job for the change to take effect
EXEC sys.sp_cdc_stop_job @job_type = N'cleanup';
EXEC sys.sp_cdc_start_job @job_type = N'cleanup';
```

{% hint style="warning" %}
If CDC data is cleaned up before Sling reads it, the run will fail with a "CDC data cleaned up past checkpoint position" error. Increase retention or run CDC more frequently.
{% endhint %}

## Transaction Log Considerations

CDC prevents the transaction log from being truncated until the capture job has processed all changes. If the CDC Agent falls behind (e.g., Agent is stopped or under heavy load), the transaction log can grow significantly.

Monitor transaction log usage:

```sql
DBCC SQLPERF(LOGSPACE);
```

To tune the capture job throughput:

```sql
EXEC sys.sp_cdc_change_job
    @job_type = N'capture',
    @maxtrans = 1000,       -- max transactions per scan cycle (default: 500)
    @maxscans = 20,          -- max scan cycles per polling interval (default: 10)
    @pollinginterval = 5;    -- seconds between polling cycles (default: 5)
```

## Troubleshooting

### "Change Data Capture is not enabled on this database"

A DBA must enable CDC on the database once (Sling does not do this):

```sql
USE my_database;
EXEC sys.sp_cdc_enable_db;
```

### "table … is not tracked by SQL Server CDC"

A DBA must enable CDC on the table, which creates its capture instance. The error message includes the exact DDL:

```sql
EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name   = N'my_table',
    @role_name     = NULL,
    @supports_net_changes = 1;
```

If a capture instance already exists under a non-default name, set `change_feed: <capture_instance>` in `change_capture_options` instead.

### "sys.fn\_cdc\_get\_max\_lsn() returned NULL"

The SQL Server Agent has not completed its first capture scan. Sling retries automatically for up to 30 seconds, but if the error persists:

1. Verify SQL Server Agent is running.
2. Manually trigger a scan: `EXEC sys.sp_cdc_scan;`
3. Check for errors in the capture job: `SELECT * FROM msdb.dbo.cdc_jobs WHERE job_type = 'capture';`

### "CDC data cleaned up past checkpoint position"

The CDC cleanup job purged change data that Sling hasn't read yet. Increase retention:

```sql
EXEC sys.sp_cdc_change_job @job_type = N'cleanup', @retention = 10080;
EXEC sys.sp_cdc_stop_job @job_type = N'cleanup';
EXEC sys.sp_cdc_start_job @job_type = N'cleanup';
```

### Initial snapshot keeps restarting

Ensure `SLING_STATE` is configured. Without state persistence, Sling cannot track that the snapshot completed and will restart it on every run.

### No CDC capture job found warning

SQL Server Agent may not be running. On Windows, start it from SQL Server Configuration Manager or Services. On Linux (Docker), ensure the `MSSQL_AGENT_ENABLED` environment variable is set:

```bash
docker run -e MSSQL_AGENT_ENABLED=true -e ACCEPT_EULA=Y ...
```

### Schema changes on CDC-enabled tables

When you `ALTER TABLE` on a CDC-enabled table (e.g., adding a column), the existing CDC capture instance continues tracking the original column set. New columns are **not** automatically captured.

To capture the new schema, create a second capture instance (SQL Server allows up to 2 per table) and then disable the old one:

```sql
-- Create new capture instance with updated schema
EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name = N'my_table',
    @capture_instance = N'dbo_my_table_v2',
    @role_name = NULL,
    @supports_net_changes = 1;

-- Disable old capture instance
EXEC sys.sp_cdc_disable_table
    @source_schema = N'dbo',
    @source_name = N'my_table',
    @capture_instance = N'dbo_my_table';
```

{% hint style="info" %}
A fresh CDC run after re-enabling the capture instance will use the new schema. If Sling's saved checkpoint references the old capture instance, it will automatically detect the change and start from the new instance's minimum LSN.
{% endhint %}


# Oracle

CDC source setup for Oracle

Sling supports Change Data Capture from Oracle by connecting to Oracle GoldenGate Data Streams over WebSocket. Each run reads row-level inserts, updates, and deletes from the GoldenGate Data Stream and merges them into the target table. Sling uses GoldenGate Data Streams — not LogMiner — because Data Streams provide a reliable, low-overhead, real-time change feed without requiring direct access to redo logs from the client.

For general CDC concepts, the two-phase process, and all available options, see the [Change Capture overview](/concepts/change-capture).

## Prerequisites

### 1. Oracle Database Requirements

Oracle Database 19c or later is required. The following settings must be configured on the source database:

```sql
-- Verify ARCHIVELOG mode is enabled
SELECT LOG_MODE FROM V$DATABASE;  -- Must be ARCHIVELOG

-- Enable ARCHIVELOG mode (if not already enabled)
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;

-- Enable force logging (recommended)
ALTER DATABASE FORCE LOGGING;

-- Enable supplemental logging
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

-- Enable GoldenGate replication parameter
ALTER SYSTEM SET enable_goldengate_replication = TRUE SCOPE=BOTH;
```

Verify the configuration:

```sql
SELECT SUPPLEMENTAL_LOG_DATA_MIN, FORCE_LOGGING FROM V$DATABASE;
-- Both should return YES
```

### 2. GoldenGate Requirements

Oracle GoldenGate **23ai or later** is required. The Data Streams feature — which Sling uses to consume changes over WebSocket — was introduced in GoldenGate 23ai.

Before running Sling, you must have:

1. A GoldenGate deployment connected to the source Oracle database.
2. An **Extract** process configured to read the source database's redo logs.
3. A **Data Stream** provisioned and attached to the Extract's trail file.

{% hint style="info" %}
**GoldenGate Free** is available at no cost for Oracle databases under 20 GB. It includes the Data Streams feature and is sufficient for development and small workloads. For production use with larger databases, a commercial GoldenGate license is required.
{% endhint %}

{% hint style="info" %}
Configuring GoldenGate Extract and Data Streams is done through the GoldenGate Admin Console or REST API. Refer to the [Oracle GoldenGate documentation](https://docs.oracle.com/en/middleware/goldengate/) for detailed setup instructions.
{% endhint %}

### 3. GoldenGate Connection Properties

The following properties are added to the Oracle connection definition and are only used when `mode: change-capture` is active:

<table data-full-width="false"><thead><tr><th>Property</th><th>Description</th></tr></thead><tbody><tr><td><code>gg_host</code> <strong>(required)</strong></td><td>GoldenGate REST/WebSocket hostname or IP address.</td></tr><tr><td><code>gg_port</code></td><td>GoldenGate REST/WebSocket port. Default is <code>443</code>.</td></tr><tr><td><code>gg_user</code></td><td>GoldenGate REST API user. Default is <code>oggadmin</code>.</td></tr><tr><td><code>gg_password</code> <strong>(required)</strong></td><td>GoldenGate REST API password.</td></tr><tr><td><code>gg_stream</code> <strong>(required)</strong></td><td>Name of the pre-provisioned Data Stream in GoldenGate. Can also be set per-replication via the engine-neutral <a href="/pages/99wN7w0524C5hkUJxIsM#options-reference"><code>change_feed</code></a> option, which takes precedence over this property.</td></tr><tr><td><code>gg_tls_skip_verify</code></td><td>Skip TLS certificate verification. Default is <code>false</code>. Set to <code>true</code> for self-signed certificates.</td></tr><tr><td><code>gg_tls_ca_cert</code></td><td>Path to a custom CA certificate file for TLS verification.</td></tr></tbody></table>

{% hint style="info" %}
The Data Stream name is the Oracle equivalent of the engine-neutral [`change_feed`](/concepts/change-capture#options-reference) CDC option (PostgreSQL publication / SQL Server capture instance). You may set it either on the connection as `gg_stream` or per-replication as `change_feed` under `change_capture_options`; if both are set, `change_feed` wins. Either way it must name a Data Stream that already exists — Sling reads it, it does not create it.
{% endhint %}

### 4. User Permissions

The Oracle database user used by Sling needs `SELECT` access on the source tables (used during the initial snapshot):

```sql
GRANT SELECT ON HR.EMPLOYEES TO sling_user;
GRANT SELECT ON HR.DEPARTMENTS TO sling_user;

-- Or grant schema-wide access
GRANT SELECT ANY TABLE TO sling_user;
```

{% hint style="info" %}
The GoldenGate Extract process typically runs under a dedicated GoldenGate admin user with broader privileges. The Sling connection user only needs `SELECT` on the source tables for the initial snapshot — change events are consumed from the GoldenGate Data Stream, not directly from the database.
{% endhint %}

## Quick Start

```bash
# Source Oracle (with GoldenGate CDC properties)
sling conns set MY_ORACLE type=oracle host=oracle.example.com user=sling_user password=secret sid=ORCL \
  gg_host=gg.example.com gg_password=gg_secret gg_stream=my_data_stream

# Target PostgreSQL
sling conns set MY_POSTGRES type=postgres host=pg.example.com user=postgres password=secret database=analytics

# State store (required for CDC)
export SLING_STATE='MY_POSTGRES/sling_state'
```

```yaml
# replication.yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  HR.EMPLOYEES:
  HR.DEPARTMENTS:
```

```bash
# First run: performs the initial snapshot
sling run -r replication.yaml

# Subsequent runs: captures and applies changes
sling run -r replication.yaml
```

## How It Works

Unlike MySQL (binlog streaming) and PostgreSQL (WAL logical replication), Oracle CDC uses an external middleware layer:

1. The GoldenGate **Extract** process reads the Oracle redo logs and writes change records to a trail file.
2. The GoldenGate **Data Stream** exposes the trail file as a WebSocket endpoint, streaming row-level change events in real time.
3. On each run, Sling connects to the Data Stream, reads change events from the last saved position, and merges them into the target table.
4. The GoldenGate position is saved in the state store.

Because Sling connects to GoldenGate (not directly to Oracle redo logs), the Oracle database sees **no additional load** from CDC reads. All redo log parsing is handled by the GoldenGate Extract process.

## Examples

### Large Tables with Custom Chunk Size

For very large tables, adjust the chunk size to control memory usage and checkpointing frequency during the initial snapshot.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000  # 50k rows per chunk

streams:
  HR.TRANSACTIONS:
    # This 10M-row table will be loaded in ~200 chunks
    # If interrupted, it resumes from the last completed chunk
```

### Time-Bounded Snapshots for Very Large Tables

For tables with hundreds of millions of rows, the initial snapshot can take hours. Use `snapshot_run_duration` to cap how long each run spends on the snapshot. The next run automatically resumes from the last completed chunk.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000
    snapshot_run_duration: 30m  # spend at most 30 minutes per run on the snapshot

streams:
  HR.HUGE_EVENTS:
    # 500M rows — will take multiple runs to complete the initial load
    # Each run processes ~30 minutes worth of chunks, then exits cleanly
```

### High-Throughput Workloads

For tables with heavy write activity, increase `run_max_events` and `run_max_duration` so each run captures more changes.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: warehouse.{stream_table}
  change_capture_options:
    run_max_events: 50000  # Process up to 50k events per run
    run_max_duration: 60s  # Wait up to 60 seconds for events

streams:
  HR.CLICK_EVENTS:
  HR.PAGE_VIEWS:
```

### Soft Deletes

Keep deleted rows in the target instead of physically removing them. Deleted rows are marked with `_sling_synced_op = 'D'`. Useful for audit trails or when downstream queries need to detect deletions.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    soft_delete: true

streams:
  HR.CUSTOMERS:
  HR.SUBSCRIPTIONS:
```

When a row is deleted in Oracle, the target row is preserved with `_sling_synced_op` set to `'D'` and `_sling_synced_at` updated to the current timestamp. A subsequent re-insert of the same primary key restores the row with the appropriate operation type.

### Mixed Streams with Per-Stream Overrides

Different tables can have different CDC options.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}

streams:
  # High-volume table: larger batches
  HR.EVENTS:
    change_capture_options:
      run_max_events: 100000
      run_max_duration: 2m

  # Audit table: keep soft deletes
  HR.USER_ACCOUNTS:
    change_capture_options:
      soft_delete: true

  # Standard table: uses defaults
  HR.PRODUCTS:
```

### Replay / Backfill from a Point in Time

If target data becomes inconsistent, you can replay changes from an earlier position. The `replay_from` value is applied exactly once per unique value.

```yaml
source: MY_ORACLE
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    replay_from: "2025-06-01T00:00:00Z"  # Re-process all changes since June 1

streams:
  HR.ORDERS:
```

After the replay run completes, remove or change the `replay_from` value. Leaving it unchanged has no effect (it is only applied once).

## Replay Formats

The `replay_from` option accepts these Oracle-specific position formats:

* **RFC 3339 timestamp**: `2025-06-01T00:00:00Z`
* **GoldenGate position**: A GoldenGate position string as reported by the Data Stream (e.g., the position value from the state store)

## Why GoldenGate (Not LogMiner)

Sling uses GoldenGate Data Streams instead of LogMiner for Oracle CDC. Here's why:

* **LogMiner is resource-intensive**: LogMiner parses redo logs via SQL sessions on the source database, consuming CPU and memory on the production system. GoldenGate Extract offloads this work to a separate process.
* **LogMiner has version-specific limitations**: The `CONTINUOUS_MINE` option — which enabled continuous redo log reading — was deprecated in Oracle 19c. Without it, LogMiner requires manual log file management.
* **LogMiner is single-threaded**: It does not scale well for high-volume, low-latency CDC workloads.
* **GoldenGate Data Streams are purpose-built**: They provide a real-time, WebSocket-based change feed with built-in position tracking, designed specifically for CDC consumers.
* **GoldenGate Free is available**: For databases under 20 GB, [GoldenGate Free](https://www.oracle.com/integration/goldengate/free/) can be used at no cost, making it accessible for development and small deployments.

## GoldenGate Setup Overview

Setting up GoldenGate is a one-time process. Here is a condensed checklist:

1. **Install GoldenGate 23ai** (or later). The quickest method is Docker:

   ```bash
   docker login container-registry.oracle.com
   docker pull container-registry.oracle.com/goldengate/goldengate-free:latest

   docker run --name ogg-free -p 443:443 \
     -e OGG_ADMIN_PWD='YourPassword1!' \
     container-registry.oracle.com/goldengate/goldengate-free:latest
   ```
2. **Connect GoldenGate to the source Oracle database** via the GoldenGate Admin Console at `https://localhost:443`.
3. **Create an Extract** process that reads the source database's redo logs.
4. **Create a Data Stream** in the GoldenGate Console under Distribution Service > Data Streams, and point it to the Extract's trail file.
5. **Note the Data Stream name** — this becomes the `gg_stream` connection property in Sling.
6. **Ensure the GoldenGate REST API is accessible** from the machine running Sling (port 443 by default).

For detailed instructions, see:

* [Oracle GoldenGate Free Getting Started Guide](https://docs.oracle.com/en/middleware/goldengate/free/21/uggfe/get-started-goldengate-free.html)
* [Data Streams Components](https://docs.oracle.com/en/database/goldengate/core/26/coredoc/distribute-datastream-componentsofoggds.html)

## Troubleshooting

### "CDC not supported for \<type>"

Ensure your source connection is configured as `type=oracle` and includes the required `gg_host`, `gg_password`, and `gg_stream` properties.

### "could not connect to GoldenGate Data Stream"

Verify `gg_host` and `gg_port` are correct and that the GoldenGate REST API is reachable from the machine running Sling. Check firewall rules and TLS settings. For self-signed certificates, set `gg_tls_skip_verify: true` for testing.

### "GoldenGate Data Stream not found"

The configured Data Stream name (`gg_stream`, or `change_feed` if set) must match an existing Data Stream in GoldenGate. Verify the stream exists in the GoldenGate Admin Console under Distribution Service > Data Streams.

### "authentication failed for GoldenGate"

Check `gg_user` and `gg_password`. The default GoldenGate admin user is `oggadmin`. Ensure the password meets Oracle's complexity requirements.

### Initial snapshot keeps restarting

Ensure `SLING_STATE` is configured. Without state persistence, Sling cannot track that the snapshot completed and will restart it on every run.

### "could not resolve replay\_from position"

The `replay_from` value must be a valid RFC 3339 timestamp or GoldenGate position string. Ensure the position is still available in the GoldenGate trail files.


# MongoDB

CDC source setup for MongoDB

Sling supports Change Data Capture from MongoDB by using Change Streams, which are built on MongoDB's oplog (operations log). Each run reads document-level inserts, updates, replaces, and deletes from the change stream and merges them into the target table.

For general CDC concepts, the two-phase process, and all available options, see the [Change Capture overview](/concepts/change-capture).

## Prerequisites

### 1. Replica Set Required

MongoDB Change Streams require a replica set. **Standalone MongoDB instances are not supported.** A single-node replica set is sufficient for development and testing.

To convert a standalone instance to a single-node replica set:

1. Add the following to your `mongod.conf`:

   ```yaml
   replication:
     replSetName: "rs0"
   ```
2. Restart `mongod`, then initialize the replica set:

   ```javascript
   // In mongosh
   rs.initiate()
   ```
3. Verify the replica set is running:

   ```javascript
   rs.status()
   ```

{% hint style="warning" %}
Change Streams are not available on standalone MongoDB instances. If your MongoDB is standalone, convert it to a single-node replica set before enabling CDC.
{% endhint %}

{% hint style="info" %}
**MongoDB Atlas**: Change Streams work out of the box on M10+ dedicated clusters (which are always deployed as replica sets). No special setup is needed — just use the standard Atlas connection string. Note that Atlas Flex Clusters do not support Change Streams.
{% endhint %}

### 2. User Permissions

The MongoDB user needs the `read` role on the source database. This role includes both the `find` and `changeStream` privileges required for CDC:

```javascript
db.createUser({
  user: "sling_user",
  pwd: "secret",
  roles: [
    { role: "read", db: "my_database" }
  ]
})
```

To watch multiple databases, grant the `read` role on each, or use `readAnyDatabase` on the `admin` database for deployment-wide access.

### 3. Post-Images (Recommended)

MongoDB 6.0+ supports Change Stream post-images, which provide the complete document state after every update. This ensures full row data for UPDATE events without an additional database read.

To enable post-images on a collection:

```javascript
db.runCommand({
  collMod: "my_collection",
  changeStreamPreAndPostImages: { enabled: true }
})
```

{% hint style="info" %}
Post-images are optional. On MongoDB versions below 6.0, or on collections without post-images enabled, Sling falls back to `updateLookup` mode, which performs a separate read to fetch the current document on each UPDATE event. For most workloads this is sufficient, but enabling post-images provides stronger consistency guarantees and avoids the extra read.
{% endhint %}

### 4. Version Requirements

| Feature                                      | Minimum Version |
| -------------------------------------------- | --------------- |
| Change Streams (collection-level)            | MongoDB 3.6     |
| Change Streams (database-level)              | MongoDB 4.0     |
| Post-images (`changeStreamPreAndPostImages`) | MongoDB 6.0     |

Sling requires **MongoDB 4.0 or later** for database-level change stream watching.

## Quick Start

```bash
# Source MongoDB (no special CDC properties needed — uses standard connection)
sling conns set MY_MONGO type=mongodb host=mongo.example.com user=sling_user password=secret port=27017

# Target PostgreSQL
sling conns set MY_POSTGRES type=postgres host=pg.example.com user=postgres password=secret database=analytics

# State store (required for CDC)
export SLING_STATE='MY_POSTGRES/sling_state'
```

```yaml
# replication.yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: public.{stream_table}

streams:
  my_database.customers:
  my_database.orders:
```

```bash
# First run: performs the initial snapshot
sling run -r replication.yaml

# Subsequent runs: captures and applies changes
sling run -r replication.yaml
```

{% hint style="info" %}
No special CDC connection properties are needed for MongoDB. Sling uses the standard MongoDB connection to open Change Streams. The `primary_key` defaults to `[_id]` since every MongoDB document has an `_id` field.
{% endhint %}

## Examples

### Large Tables with Custom Chunk Size

For very large collections, adjust the chunk size to control memory usage and checkpointing frequency during the initial snapshot.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000  # 50k documents per chunk

streams:
  my_database.transactions:
    # This 10M-document collection will be loaded in ~200 chunks
    # If interrupted, it resumes from the last completed chunk
```

### Time-Bounded Snapshots for Very Large Tables

For collections with hundreds of millions of documents, the initial snapshot can take hours. Use `snapshot_run_duration` to cap how long each run spends on the snapshot. The next run automatically resumes from the last completed chunk.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: analytics.{stream_table}
  change_capture_options:
    snapshot_chunk_size: 50000
    snapshot_run_duration: 30m  # spend at most 30 minutes per run on the snapshot

streams:
  my_database.huge_events:
    # 500M documents — will take multiple runs to complete the initial load
    # Each run processes ~30 minutes worth of chunks, then exits cleanly
```

### High-Throughput Workloads

For collections with heavy write activity, increase `run_max_events` and `run_max_duration` so each run captures more changes.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: warehouse.{stream_table}
  change_capture_options:
    run_max_events: 50000  # Process up to 50k events per run
    run_max_duration: 60s  # Wait up to 60 seconds for events

streams:
  my_database.click_events:
  my_database.page_views:
```

### Soft Deletes

Keep deleted documents in the target instead of physically removing them. Deleted rows are marked with `_sling_synced_op = 'D'`. Useful for audit trails or when downstream queries need to detect deletions.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: public.{stream_table}
  change_capture_options:
    soft_delete: true

streams:
  my_database.customers:
  my_database.subscriptions:
```

When a document is deleted in MongoDB, the target row is preserved with `_sling_synced_op` set to `'D'` and `_sling_synced_at` updated to the current timestamp. A subsequent re-insert of the same `_id` restores the row with the appropriate operation type.

### Mixed Streams with Per-Stream Overrides

Different collections can have different CDC options.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: public.{stream_table}

streams:
  # High-volume collection: larger batches
  my_database.events:
    change_capture_options:
      run_max_events: 100000
      run_max_duration: 2m

  # Audit collection: keep soft deletes
  my_database.user_accounts:
    change_capture_options:
      soft_delete: true

  # Standard collection: uses defaults
  my_database.products:
```

### Replay / Backfill from a Point in Time

If target data becomes inconsistent, you can replay changes from an earlier position. The `replay_from` value is applied exactly once per unique value.

```yaml
source: MY_MONGO
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [_id]
  object: public.{stream_table}
  change_capture_options:
    replay_from: "2025-06-01T00:00:00Z"  # Re-process all changes since June 1

streams:
  my_database.orders:
```

After the replay run completes, remove or change the `replay_from` value. Leaving it unchanged has no effect (it is only applied once).

## Replay Formats

The `replay_from` option accepts this MongoDB-specific position format:

* **RFC 3339 timestamp**: `2025-06-01T00:00:00Z` — resolved to a MongoDB `operationTime` (ClusterTime)

## Oplog Retention

MongoDB's oplog is a capped collection. If the oplog wraps around and removes entries that Sling hasn't processed, the Change Stream will be invalidated.

Check the current oplog size and retention window:

```javascript
db.getReplicationInfo()
```

Ensure the oplog window is longer than the maximum gap between CDC runs. To resize the oplog:

```javascript
// Resize oplog to 10 GB (MongoDB 4.0+)
db.adminCommand({ replSetResizeOplog: 1, size: 10240 })
```

{% hint style="warning" %}
If the oplog wraps past Sling's saved position, the next run will detect the invalidated Change Stream and automatically perform a fresh initial snapshot.
{% endhint %}

## Document Flattening

MongoDB documents are nested JSON. Sling flattens nested documents into columns using dot notation — for example, a field `address.city` in a nested document becomes the column `address__city` in the target table. Arrays and deeply nested objects beyond the flattening depth are serialized as JSON strings.

## Troubleshooting

### "CDC not supported for \<type>"

Ensure your source connection is configured as `type=mongodb`.

### "change stream not supported"

MongoDB Change Streams require a replica set. Standalone instances are not supported. Convert to a single-node replica set:

```javascript
// In mongod.conf, add:
// replication:
//   replSetName: "rs0"

// Then restart mongod and initialize:
rs.initiate()
```

### "not authorized to run changeStream"

The MongoDB user needs the `read` role on the source database, which includes the `changeStream` privilege:

```javascript
db.grantRolesToUser("sling_user", [
  { role: "read", db: "my_database" }
])
```

### Initial snapshot keeps restarting

Ensure `SLING_STATE` is configured. Without state persistence, Sling cannot track that the snapshot completed and will restart it on every run.

### "could not resolve replay\_from position"

The `replay_from` value must be a valid RFC 3339 timestamp.

### Missing fields in UPDATE events

Enable post-images (MongoDB 6.0+) for complete document state on updates:

```javascript
db.runCommand({
  collMod: "my_collection",
  changeStreamPreAndPostImages: { enabled: true }
})
```

Without post-images, Sling uses `updateLookup` which reads the current document state. This works for most cases but may miss intermediate values in high-write scenarios.

### "change stream invalidated"

The oplog wrapped past the saved resume position. Sling will automatically re-snapshot on the next run. To prevent this, increase oplog size or run CDC more frequently:

```javascript
// Resize oplog to 10 GB
db.adminCommand({ replSetResizeOplog: 1, size: 10240 })
```


# Hooks / Steps

Execute custom actions throughout your replication or pipeline

Hooks are powerful mechanisms in Sling that allow you to execute custom actions before (pre-hooks) or after (post-hooks) a replication stream, as well as at the start or end of the replication parent cycle (before first stream and/or after last stream). They enable you to extend and customize your data pipeline with various operations such as data validation, notifications, file management, and custom processing.

{% hint style="success" %}
Hooks are the same as Steps, when using sling in [Pipeline](/concepts/pipeline) mode.

Furthermore, Sling Hooks integrate seamlessly with the [Sling VSCode Extension](/sling-cli/vscode). The extension provides schema validation, auto-completion, hover documentation, and diagnostics for your hooks configurations, making it easier to author and debug complex workflows.
{% endhint %}

Some typical operations include:

## Stream Level

**Pre-Hooks**: Execute before the replication stream run starts

* Validate prerequisites
* Download necessary files
* Set up configurations
* Perform cleanup operations

**Post-Hooks**: Execute after the replication stream run completes

* Validate results
* Send notifications
* Upload processed files
* Clean up temporary files
* Log completion status

**Pre/Post-Merge-Hooks**: Execute in transaction session, before/after data is loaded/merged into final table

* Set specific session setting and configuration
* Alter table holding the temporary data, prior to merge
* Run Specific SQL queries on other tables

## Available Hook Types

| Hook Type   | Description                                                | Documentation                                   |
| ----------- | ---------------------------------------------------------- | ----------------------------------------------- |
| Check       | Validate conditions and control flow                       | [Check Hook](/concepts/hooks/check)             |
| Command     | Run any command/process                                    | [Command Hook](/concepts/hooks/command)         |
| Copy        | Transfer files between local or remote storage connections | [Copy Hook](/concepts/hooks/copy)               |
| Delete      | Remove files from local or remote storage connections      | [Delete Hook](/concepts/hooks/delete)           |
| Group       | Run sequences of steps or loop over values                 | [Group Hook](/concepts/hooks/group)             |
| HTTP        | Make HTTP requests to external services                    | [HTTP Hook](/concepts/hooks/http)               |
| Inspect     | Inspect a file or folder                                   | [Inspect Hook](/concepts/hooks/inspect)         |
| List        | List files in folder                                       | [List Hook](/concepts/hooks/list)               |
| Log         | Output custom messages and create audit trails             | [Log Hook](/concepts/hooks/log)                 |
| Query       | Execute SQL queries against any defined connection         | [Query Hook](/concepts/hooks/query)             |
| Replication | Run a Replication                                          | [Replication Hook](/concepts/hooks/replication) |
| Routine     | Execute reusable step sequences from external files        | [Routine Hook](/concepts/hooks/routine)         |
| Store       | Store values for later in-process access                   | [Store Hook](/concepts/hooks/store)             |
| Read        | Read contents of files from storage connections            | [Read Step](/concepts/hooks/read)               |
| Write       | Write content to files in storage connections              | [Write Step](/concepts/hooks/write)             |

## Hook Configuration

Hooks can be configured in two locations:

* At the `defaults` level (applies to all streams)
* At the individual `stream` level (overrides defaults)

**Stream level Hooks**

We can use the following structure to decare hooks with the `hooks` key, under the `defaults` branch or under any stream branch.

```yaml
defaults:
  ...

streams:
  my_stream:
    hooks:
      # Prior to stream beginning
      pre:
        - type: log
          # hook configuration...

      # Inside session/transaction, before merge, right after BEGIN
      pre_merge:
        - type: query
          # hook configuration...

      # Inside session/transaction, after merge, right before COMMIT
      post_merge:
        - type: query
          # hook configuration...

      # After stream finishes
      post:
        - type: http
          # hook configuration...
```

**Replication level Hooks**

We can also define hooks to run at the replication file level, meaning before any of the streams run and/or after all the streams have ran. For replication level hooks, we must declare the `start` and `end` hooks at the root of the YAML configuration.

```yaml
# replication level hooks need to be set at the root of the YAML
hooks:
  start:
    - type: query
      # hook configuration...

  end:
    - type: http
      # hook configuration...

defaults:
  ...

streams:
  ...
```

## Common Hook Properties

All hook types share some common properties:

| Property     | Description                                                                       | Required                 |
| ------------ | --------------------------------------------------------------------------------- | ------------------------ |
| `type`       | The type of hook (`query`/ `http`/ `check`/ `copy` / `delete`/ `log` / `inspect`) | Yes                      |
| `if`         | Optional condition to determine if the hook should execute                        | No                       |
| `id`         | a specify identifier to refer to the hook output data.                            | No                       |
| `on_failure` | What to do if the hook fails (`abort`/ `warn`/ `quiet`/`skip`/`break`)            | No (defaults to `abort`) |

## Variables Available

* `runtime_state` - Contains all state variables available
* `state.*` - All hooks output state information (keyed by hook id)
* `store.*` - All stored values from previous hooks
* `env.*` - All variables defined in the `env`
* `timestamp.*` - Various timestamp parts information
* `execution.*` - Replication run level information
* `source.*` - Source connection information
* `target.*` - Target connection information
* `stream.*` - Current source stream info
* `object.*` - Current target object info
* `runs.*` - All runs information (keyed by stream run id)
* `run.*` - Current stream run information

### Nested Fields

<details>

<summary><code>timestamp.*</code> Fields</summary>

| Field       | Type     | Description                        | Example                            |
| ----------- | -------- | ---------------------------------- | ---------------------------------- |
| `timestamp` | datetime | Full timestamp object              | `2025-01-19T08:27:31.473303-05:00` |
| `unix`      | integer  | Unix epoch timestamp               | `1737286051`                       |
| `file_name` | string   | Timestamp formatted for file names | `2025_01_19_082731`                |
| `rfc3339`   | string   | RFC3339 formatted timestamp        | `2025-01-19T08:27:31-05:00`        |
| `date`      | string   | Date only                          | `2025-01-19`                       |
| `datetime`  | string   | Date and time                      | `2025-01-19 08:27:31`              |
| `YYYY`      | string   | Four-digit year                    | `2025`                             |
| `YY`        | string   | Two-digit year                     | `25`                               |
| `MMM`       | string   | Three-letter month abbreviation    | `Jan`                              |
| `MM`        | string   | Two-digit month                    | `01`                               |
| `DD`        | string   | Two-digit day                      | `19`                               |
| `DDD`       | string   | Three-letter day abbreviation      | `Sun`                              |
| `HH`        | string   | Two-digit hour (24-hour format)    | `08`                               |

</details>

<details>

<summary><code>execution.*</code> Fields</summary>

| Field              | Type        | Description                                | Example                            |
| ------------------ | ----------- | ------------------------------------------ | ---------------------------------- |
| `id`               | string      | Unique execution identifier                | `2rxeplXz2UqdIML1NncvWKNQuwD`      |
| `file_path`        | string      | Path to the replication configuration file | `/path/to/replication.yaml`        |
| `file_name`        | string      | Name to the replication configuration file | `replication.yaml`                 |
| `total_bytes`      | integer     | Total bytes processed across all runs      | `6050`                             |
| `total_rows`       | integer     | Total rows processed across all runs       | `34`                               |
| `status.count`     | integer     | Total number of streams                    | `1`                                |
| `status.success`   | integer     | Number of successful streams               | `1`                                |
| `status.running`   | integer     | Number of running streams                  | `0`                                |
| `status.skipped`   | integer     | Number of skipped streams                  | `0`                                |
| `status.cancelled` | integer     | Number of cancelled streams                | `0`                                |
| `status.warning`   | integer     | Number of streams with warnings            | `0`                                |
| `status.error`     | integer     | Number of errored streams                  | `0`                                |
| `start_time`       | datetime    | Execution start time                       | `2025-01-19T08:27:22.988403-05:00` |
| `end_time`         | datetime    | Execution end time                         | `2025-01-19T08:27:31.472684-05:00` |
| `duration`         | integer     | Execution duration in seconds              | `8`                                |
| `error`            | string/null | Error message if execution failed          | `null`                             |

</details>

<details>

<summary><code>source.*</code> / <code>target.*</code> Connection Fields</summary>

| Field       | Type   | Description          | Example (Source) | Example (Target) |
| ----------- | ------ | -------------------- | ---------------- | ---------------- |
| `name`      | string | Connection name      | `aws_s3`         | `postgres`       |
| `type`      | string | Connection type      | `s3`             | `postgres`       |
| `kind`      | string | Connection kind      | `file`           | `database`       |
| `bucket`    | string | S3/GCS bucket name   | `my-bucket-1`    | \`\`             |
| `container` | string | Azure container name | \`\`             | \`\`             |
| `database`  | string | Database name        | \`\`             | `postgres`       |
| `instance`  | string | Database instance    | \`\`             | \`\`             |
| `schema`    | string | Default schema       | \`\`             | `public`         |

</details>

<details>

<summary><code>stream.*</code> Fields</summary>

| Field          | Type   | Description                        | Example                                                                                           |
| -------------- | ------ | ---------------------------------- | ------------------------------------------------------------------------------------------------- |
| `file_folder`  | string | Parent folder of the file          | `update_dt_year=2018`                                                                             |
| `file_name`    | string | Name of the file                   | `update_dt_month=11`                                                                              |
| `file_ext`     | string | File extension                     | `parquet`                                                                                         |
| `file_path`    | string | Full file path                     | `test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11`                   |
| `name`         | string | Stream name pattern                | `test/public_test1k_postgres_pg_parquet/{part_year}/{part_month}/`                                |
| `description`  | string | Stream description (if provided)   | \`\`                                                                                              |
| `schema`       | string | Schema name (for database sources) | \`\`                                                                                              |
| `schema_lower` | string | Schema name in lowercase           | \`\`                                                                                              |
| `schema_upper` | string | Schema name in uppercase           | \`\`                                                                                              |
| `table`        | string | Table name (for database sources)  | \`\`                                                                                              |
| `table_lower`  | string | Table name in lowercase            | \`\`                                                                                              |
| `table_upper`  | string | Table name in uppercase            | \`\`                                                                                              |
| `full_name`    | string | Full stream identifier             | `s3://my-bucket-1/test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11/` |

</details>

<details>

<summary><code>object.*</code> Fields</summary>

| Field            | Type   | Description               | Example                                     |
| ---------------- | ------ | ------------------------- | ------------------------------------------- |
| `schema`         | string | Target schema name        | `public`                                    |
| `table`          | string | Target table name         | `test1k_postgres_pg_parquet`                |
| `name`           | string | Quoted object name        | `"public"."test1k_postgres_pg_parquet"`     |
| `full_name`      | string | Full quoted object name   | `"public"."test1k_postgres_pg_parquet"`     |
| `temp_schema`    | string | Temporary schema name     | `public`                                    |
| `temp_table`     | string | Temporary table name      | `test1k_postgres_pg_parquet_tmp`            |
| `temp_full_name` | string | Full temporary table name | `"public"."test1k_postgres_pg_parquet_tmp"` |

</details>

<details>

<summary><code>run.*</code> Fields</summary>

| Field                   | Type        | Description                                  | Example                                 |
| ----------------------- | ----------- | -------------------------------------------- | --------------------------------------- |
| `id`                    | string      | Run identifier                               | `test_public_test1k`                    |
| `stream.*`              | object      | Stream information (see stream fields above) | `{...}`                                 |
| `object.*`              | object      | Object information (see object fields above) | `{...}`                                 |
| `total_bytes`           | integer     | Total bytes processed in this run            | `6050`                                  |
| `total_rows`            | integer     | Total rows processed in this run             | `34`                                    |
| `status`                | string      | Run status                                   | `success`                               |
| `start_time`            | datetime    | Run start time                               | `2025-01-19T08:27:22.988403-05:00`      |
| `end_time`              | datetime    | Run end time                                 | `2025-01-19T08:27:31.472684-05:00`      |
| `duration`              | integer     | Run duration in seconds                      | `8`                                     |
| `incremental_value`     | any         | The incremental value used                   | `2025-01-19T08:27:31.472684-05:00`      |
| `range`                 | string      | The start/end range values used              | `2025-01-01,2025-02-01`                 |
| `error`                 | string/null | Error message if run failed                  | `null`                                  |
| `config.mode`           | string      | Replication mode                             | `incremental`                           |
| `config.object`         | string      | Target object                                | `"public"."test1k_postgres_pg_parquet"` |
| `config.primary_key`    | array       | Primary key columns                          | `["id"]`                                |
| `config.update_key`     | string      | Update key column                            | `update_dt`                             |
| `config.source_options` | object      | Source-specific options                      | `{}`                                    |
| `config.target_options` | object      | Target-specific options                      | `{}`                                    |

</details>

### `runtime_state` Payload

The best way to view any available variables is to print the `runtime_state` variable.

For example, using the `log` hook as shown below will print all available variables.

```yaml
- type: log
  message: '{runtime_state}'
```

Shows something like below JSON payload.

<details>

<summary><code>runtime_state</code> Payload</summary>

```json
{
  "state": {
    "end-01": {},
    "start-01": {
      "level": "info",
      "message": "{...}",
      "status": "success"
    },
    "start-02": {
      "path": "sling-state/test/r.19",
      "status": "success"
    }
  },
  "store": {
    "my_key": "my_value"
  },
  "env": {
    "RESET": "true",
    "SLING_STATE": "aws_s3/sling-state/test/r.19"
  },
  "timestamp": {
    "timestamp": "2025-01-19T08:27:31.473303-05:00",
    "unix": 1737286051,
    "file_name": "2025_01_19_082731",
    "rfc3339": "2025-01-19T08:27:31-05:00",
    "date": "2025-01-19",
    "datetime": "2025-01-19 08:27:31",
    "YYYY": "2025",
    "YY": "25",
    "MMM": "Jan",
    "MM": "01",
    "DD": "19",
    "DDD": "Sun",
    "HH": "08"
  },
  "source": {
    "name": "aws_s3",
    "type": "s3",
    "kind": "file",
    "bucket": "my-bucket-1",
    "container": "",
    "database": "",
    "instance": "",
    "schema": ""
  },
  "target": {
    "name": "postgres",
    "type": "postgres",
    "kind": "database",
    "bucket": "",
    "container": "",
    "database": "postgres",
    "instance": "",
    "schema": "public"
  },
  "stream": {
    "file_folder": "update_dt_year=2018",
    "file_name": "update_dt_month=11",
    "file_ext": "parquet",
    "file_path": "test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11",
    "name": "test/public_test1k_postgres_pg_parquet/{part_year}/{part_month}/",
    "schema": "",
    "schema_lower": "",
    "schema_upper": "",
    "table": "",
    "table_lower": "",
    "table_upper": "",
    "full_name": "s3://my-bucket-1/test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11/"
  },
  "object": {
    "schema": "public",
    "table": "test1k_postgres_pg_parquet",
    "name": "\"public\".\"test1k_postgres_pg_parquet\"",
    "full_name": "\"public\".\"test1k_postgres_pg_parquet\"",
    "temp_schema": "public",
    "temp_table": "test1k_postgres_pg_parquet_tmp",
    "temp_full_name": "\"public\".\"test1k_postgres_pg_parquet_tmp\""
  },
  "runs": {
    "test_public_test1k": {
      "id": "test_public_test1k",
      "stream": {
        "file_folder": "update_dt_year=2018",
        "file_name": "update_dt_month=11",
        "file_ext": "parquet",
        "file_path": "test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11",
        "name": "test/public_test1k_postgres_pg_parquet/{part_year}/{part_month}/",
        "schema": "",
        "schema_lower": "",
        "schema_upper": "",
        "table": "",
        "table_lower": "",
        "table_upper": "",
        "full_name": "s3://my-bucket-1/test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11/"
      },
      "object": {
        "schema": "public",
        "table": "test1k_postgres_pg_parquet",
        "name": "\"public\".\"test1k_postgres_pg_parquet\"",
        "full_name": "\"public\".\"test1k_postgres_pg_parquet\"",
        "temp_schema": "public",
        "temp_table": "test1k_postgres_pg_parquet_tmp",
        "temp_full_name": "\"public\".\"test1k_postgres_pg_parquet_tmp\""
      },
      "total_bytes": 6050,
      "total_rows": 34,
      "status": "success",
      "start_time": "2025-01-19T08:27:22.988403-05:00",
      "end_time": "2025-01-19T08:27:31.472684-05:00",
      "duration": 8,
      "error": null,
      "config": {
        "mode": "incremental",
        "object": "\"public\".\"test1k_postgres_pg_parquet\"",
        "primary_key": [
          "id"
        ],
        "update_key": "update_dt",
        "source_options": {},
        "target_options": {},
        "single": false,
        "hooks": {}
      }
    }
  },
  "execution": {
    "id": "2rxeplXz2UqdIML1NncvWKNQuwD",
    "string": "/path/to/replication.yaml",
    "total_bytes": 6050,
    "total_rows": 34,
    "status": {
      "count": 1,
      "success": 1,
      "running": 0,
      "skipped": 0,
      "cancelled": 0,
      "warning": 0,
      "error": 0
    },
    "start_time": "2025-01-19T08:27:22.988403-05:00",
    "end_time": "2025-01-19T08:27:31.472684-05:00",
    "duration": 8,
    "error": null
  },
  "run": {
    "id": "test_public_test1k",
    "stream": {
      "file_folder": "update_dt_year=2018",
      "file_name": "update_dt_month=11",
      "file_ext": "parquet",
      "file_path": "test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11",
      "name": "test/public_test1k_postgres_pg_parquet/{part_year}/{part_month}/",
      "schema": "",
      "schema_lower": "",
      "schema_upper": "",
      "table": "",
      "table_lower": "",
      "table_upper": "",
      "full_name": "s3://my-bucket-1/test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11/"
    },
    "object": {
      "schema": "public",
      "table": "test1k_postgres_pg_parquet",
      "name": "\"public\".\"test1k_postgres_pg_parquet\"",
      "full_name": "\"public\".\"test1k_postgres_pg_parquet\"",
      "temp_schema": "public",
      "temp_table": "test1k_postgres_pg_parquet_tmp",
      "temp_full_name": "\"public\".\"test1k_postgres_pg_parquet_tmp\""
    },
    "total_bytes": 6050,
    "total_rows": 34,
    "status": "success",
    "start_time": "2025-01-19T08:27:22.988403-05:00",
    "end_time": "2025-01-19T08:27:31.472684-05:00",
    "duration": 8,
    "error": null,
    "config": {
      "mode": "incremental",
      "object": "\"public\".\"test1k_postgres_pg_parquet\"",
      "primary_key": [
        "id"
      ],
      "update_key": "update_dt",
      "source_options": {},
      "target_options": {},
      "single": false,
      "hooks": {}
    }
  }
}
```

</details>

Furthermore, we can access any data-point using a [`jmespath` expression](https://jmespath.org/):

* `state["start-02"].status` - Gets the status of a hook (returns `success`)
* `store.my_key` - Gets a stored value from the store (returns `my_value`)
* `run.total_rows` - Gets the number of rows processed in the current run (returns `34`)
* `run.duration` - Gets the duration of the current run in seconds (returns `8`)
* `timestamp.unix` - The epoch/unix timestamp (returns `1737286051`)
* `source.bucket` - Gets the source S3 bucket name (returns `my-bucket-1`)
* `target.database` - Gets the target database name (returns `postgres`)
* `run.config.primary_key[0]` - Gets the first primary key column (returns `id`)
* `stream.file_path` - Gets the current stream's file path (returns `test/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11`)
* `stream.file_ext` - Gets the file extension (returns `parquet`)
* `stream.schema_lower` - Gets the stream schema name in lowercase
* `stream.table_upper` - Gets the stream table name in uppercase
* `object.temp_full_name` - Gets the temporary table full name (returns `"public"."test1k_postgres_pg_parquet_tmp"`)
* `execution.status.error` - Gets the count of errored streams (returns `0`)
* `execution.total_bytes` - Gets the total bytes processed across all runs (returns `6050`)
* `runs["test_public_test1k"].status` - Gets the status of a specific run by ID (returns `success`)

## Complete Example

```yaml
# replication level hooks need to be set at the root of the YAML
hooks:
  # runs in order before replication starts.
  start:
      - type: query
        query: select ....
        id: my_query # can use `{state.my_query.result[0].col1}` later
        on_failure: abort
      
  # runs in order after all streams have completed.
  end:
      # check for any errors. if errored, do not proceed (break)
      - type: check
        check: execution.status.error == 0
        on_failure: break
      
      - type: query
        query: update ....
        into: result # can use `{result.col1}` later
        on_failure: abort

defaults:
  hooks:
    pre:
      - type: query
        connection: source_db
        query: "UPDATE status SET running = true"
        on_failure: abort

    post:
      - type: check
        check: "run.total_rows > 0"
        on_failure: warn
      
      - type: http
        if: run.status == "success"
        url: "https://api.example.com/webhook"
        method: POST
        payload: | # my_query.result will serialize into an array of objects
          {
            "status": "complete",
            "name": "{ my_query.result[0].name }"
            "records": {my_query.result}}
          }

streams:
  public.users:
    hooks:
      # runs in order before stream run
      pre:
        - type: query
          query: update ....
          on_failure: abort

      # runs in order after stream run
      post:
        - type: http
          url: https://my.webhook/path
          method: POST
          payload: |
            {"result": "{run.status}"}
```

## Best Practices

1. **Error Handling**: Specify appropriate `on_failure` behavior. The default value is `abort`.
2. **Validation**: Use `check` hooks to validate prerequisites and results
3. **Logging**: Implement `log` hooks for better observability
4. **Cleanup**: Use `delete` hooks to manage temporary / old files
5. **Modularity**: Break down complex operations into multiple hooks
6. **Conditions**: Use `if` conditions to control hook execution
7. **Environment Awareness**: Consider different environments in hook configurations


# Check

Check hooks allow you to validate conditions and control the flow of your replication process. They are useful for implementing data quality checks, validating prerequisites, and ensuring business rules are met.

## Configuration

```yaml
- type: check
  check: "run.total_rows > threshold"  # Required: The condition to evaluate
  failure_message: '{run.total_rows} is below threshold'  # Optional: the message to use as an error
  vars:                       # Optional: Local variables for the check
    threshold: 1000
    min_date: "2023-01-01"
  on_failure: abort          # Optional: abort/warn/quiet/skip
  id: my_id                  # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property         | Required | Description                                           |
| ---------------- | -------- | ----------------------------------------------------- |
| check            | Yes      | The condition to evaluate                             |
| failure\_message | No       | A Message to use as the error if check fails          |
| vars             | No       | Map of scoped variables that can be used in the check |
| on\_failure      | No       | What to do if the check fails (abort/warn/quiet/skip) |

## Output

When the check hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
failure_message: message  # The rendered message
result: true     # The result of the check evaluation (true/false)
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.check}` - the compiled expresion to check
* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.result}` - Boolean result of the check

## Examples

### Basic Row Count Validation

Ensure that the replication processed a minimum number of rows:

```yaml
hooks:
  post:
    - type: check
      check: "run.total_rows >= min_rows"
      vars:
        min_rows: 100
      on_failure: abort
```

### Multiple Condition Check

Validate multiple conditions before starting replication:

```yaml
hooks:
  pre:
    - type: check
      check: |
        run.stream.schema != '' && 
        run.object.schema != '' && 
        timestamp.hour >= 1 && 
        timestamp.hour <= 23
      on_failure: abort
```

### Data Quality Threshold Check

Verify that the error rate in processed data is below a threshold:

```yaml
hooks:
  post:
    - type: check
      check: |
        state.quality_check.result.error_rate <= max_error_rate
      vars:
        max_error_rate: 0.01  # 1% error rate threshold
      on_failure: warn
```

### Time Window Validation

Ensure replication runs within specific time windows:

```yaml
hooks:
  pre:
    - type: check
      check: |
        (timestamp.hour >= start_hour && 
         timestamp.hour <= end_hour) ||
        (timestamp.day_name in allowed_days)
      vars:
        start_hour: 20  # 8 PM
        end_hour: 6    # 6 AM
        allowed_days: ["Saturday", "Sunday"]
      on_failure: skip
```

### Complex Business Rule Validation

Implement complex business rules with multiple conditions:

```yaml
hooks:
  post:
    - type: check
      check: |
        (run.total_rows >= min_rows && 
         run.total_rows <= max_rows) &&
        (run.duration <= max_duration) &&
        (state.data_check.result.null_percentage <= max_null_percent)
      vars:
        min_rows: 1000
        max_rows: 1000000
        max_duration: 3600  # 1 hour
        max_null_percent: 5
      on_failure: abort
```

### Environment-Based Validation

Apply different validation rules based on the environment:

```yaml
hooks:
  pre:
    - type: check
      check: |
        (env.ENVIRONMENT == 'production' AND 
         run.stream.name IN prod_allowed_streams) OR
        (env.ENVIRONMENT != 'production')
      vars:
        prod_allowed_streams: ["customers", "orders", "products"]
      on_failure: abort
```

### Resource Usage Check

Validate system resource availability before proceeding:

```yaml
hooks:
  pre:
    - type: check
      check: |
        state.resource_check.result.available_disk_space >= min_disk_space &&
        state.resource_check.result.available_memory >= min_memory
      vars:
        min_disk_space: 10737418240  # 10GB in bytes
        min_memory: 4294967296      # 4GB in bytes
      on_failure: warn
```

### Data Freshness Check

Ensure source data is fresh enough before replication:

```yaml
hooks:
  pre:
    - type: check
      check: |
        state.freshness_check.result.last_update_time >= 
        timestamp.unix - max_age_seconds
      vars:
        max_age_seconds: 3600  # 1 hour
      on_failure: skip
```


# Command

Command hooks allow you to execute system commands or scripts as part of your replication workflow. This is particularly useful for running data processing scripts, triggering external processes, or performing system-level operations.

## Configuration

```yaml
- type: command
  command: ["executable", "arg1", "arg2"]  # Required: Command and arguments as array or string
  working_dir: /path/to/dir  # Optional: Directory to run the command in (default: Sling's current directory)
  print: true      # Optional: Print command output to console (default: true)
  capture: true    # Optional: Capture command output in hook result (default: true)
  timeout: 300     # Optional: Command timeout in seconds (default: no timeout)
  env:              # Optional: Environment variables for the command
    ENV_VAR1: "value1"
    ENV_VAR2: "value2"
  ssh_conn: MY_SSH  # Optional: Run the command remotely over an SSH connection
  on_failure: abort # Optional: abort/warn/quiet/skip
  id: my_id         # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property     | Required | Description                                                                                                                                                              |
| ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| command      | Yes      | String or Array containing the command and its arguments                                                                                                                 |
| working\_dir | No       | Working directory (`cwd`) to run the command in. Defaults to the directory Sling is running from. Supports expressions and is created automatically if it does not exist |
| print        | No       | Whether to print command output to console (default: true)                                                                                                               |
| capture      | No       | Whether to capture command output in hook result (default: true)                                                                                                         |
| timeout      | No       | Command timeout in seconds. If 0 or not specified, no timeout is applied                                                                                                 |
| env          | No       | Map of environment variables to set for the command                                                                                                                      |
| ssh\_conn    | No       | Name of an SSH connection to execute the command on a remote host                                                                                                        |
| on\_failure  | No       | What to do if the command fails (abort/warn/quiet/skip)                                                                                                                  |

### Working Directory

By default the command inherits the directory Sling is running from, which is not always where your scripts or relative paths live. Use `working_dir` to set the command's `cwd` explicitly:

```yaml
hooks:
  pre:
    - type: command
      command: ./run-etl.sh --stream "{run.stream.name}"
      working_dir: /opt/etl/scripts
```

Notes:

* The value supports expressions, so it can be built dynamically — e.g. `working_dir: "/data/{run.stream.name}"` or `working_dir: "{env.PROJECT_DIR}/scripts"`.
* If the directory does not exist, Sling creates it before running the command.
* When combined with `ssh_conn`, the command is prefixed with `cd <working_dir> &&` on the remote host.

## Output

When the command hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
binary: "/path/to/executable"  # The binary that was executed
arguments: ["arg1", "arg2"]  # The arguments passed to the command
start: "2024-01-01T00:00:00Z"  # Command start time
end: "2024-01-01T00:00:01Z"  # Command end time
timeout: 300  # Only present if timeout was specified
output:  # Only present if capture: true
  stdout: "Standard output text"
  stderr: "Standard error text"
  combined: "Combined output text"
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.binary}` - The binary that was executed
* `{state.hook_id.arguments}` - The arguments passed to the command
* `{state.hook_id.start}` - Command start time
* `{state.hook_id.end}` - Command end time
* `{state.hook_id.timeout}` - Timeout value (if specified)
* `{state.hook_id.output.stdout}` - Standard output (if capture: true)
* `{state.hook_id.output.stderr}` - Standard error (if capture: true)
* `{state.hook_id.output.combined}` - Combined output (if capture: true)

## Examples

### Run Data Processing Script

Execute a Python script to process data before replication:

```yaml
hooks:
  pre:
    - type: command
      command: python scripts/process_data.py --stream "{run.stream.name}"
      timeout: 600  # 10 minute timeout
      env:
        PYTHONPATH: "/path/to/libs"
        DATA_DIR: "{env.data_directory}"
      print: true
      on_failure: abort
```

### Run a CLI Tool from Its Own Directory

Run a custom CLI tool that expects to be invoked from its project directory (so relative paths and config files resolve correctly):

```yaml
hooks:
  pre:
    - type: command
      command: ["./my-cli", "extract", "--config", "config.yaml"]
      working_dir: /opt/my-tool
      capture: true
      print: true
      on_failure: abort
```

### System Cleanup

Clean up temporary files after processing:

```yaml
hooks:
  post:
    - type: command
      command: ["rm", "-rf", "/tmp/processed/{run.stream.name}/*"]
      on_failure: warn
```

### Run Data Quality Checks

Execute a data quality checking script and capture its output:

```yaml
hooks:
  post:
    - type: command
      command: [
        "python",
        "scripts/quality_check.py",
        "--table", "{run.object.full_name}",
        "--date", "{timestamp.date}"
      ]
      timeout: 1800  # 30 minute timeout
      capture: true
      env:
        DB_CONNECTION: "{target.connection_string}"
      on_failure: warn
```

### Conditional Command Execution

Run commands based on environment or conditions:

```yaml
hooks:
  post:
    - type: command
      if: env.PRODUCTION == "true"
      command: notify-admin --stream "{run.stream.name}" --status {run.status}
      print: true
      env:
        NOTIFY_TOKEN: "{env.notification_token}"
```

### Run Shell Script

Execute a shell script with parameters:

```yaml
hooks:
  pre:
    - type: command
      command: [
        "bash",
        "scripts/prepare_environment.sh",
        "{target.environment}",
        "{run.stream.name}"
      ]
      print: true
      capture: true
      on_failure: abort
```

### Generate Reports

Run a report generation tool after successful replication:

```yaml
hooks:
  post:
    - type: command
      if: run.status == "success"
      command: [
        "report-generator",
        "--input", "{run.object.full_name}",
        "--output", "reports/{run.stream.name}_{timestamp.date}.pdf"
      ]
      timeout: 900  # 15 minute timeout
      env:
        REPORT_TEMPLATE: "templates/standard.tpl"
        OUTPUT_DIR: "/var/reports"
```


# Copy

Copy hooks allow you to transfer files between storage locations. This is particularly useful for moving files between different storage systems, making backups, or archiving data.

## Configuration

```yaml
- type: copy
  from: "connection1/path/to/source"   # Required: Source Location
  to: "connection2/path/to/dest"       # Required: Destination Location
  single_file: true       # Optional: true/false. Force treat source as single file (no need to list the source location)
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id      # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property     | Required | Description                                                                                                                                                                                                                  |
| ------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| from         | Yes      | The source [location](/sling-cli/environment#location-string) string. Contains connection name and path.                                                                                                                     |
| to           | Yes      | The destination [location](/sling-cli/environment#location-string) string. Contains connection name and path.                                                                                                                |
| single\_file | No       | Boolean flag to specify whether to treat the source as a single file. If `true`, copies as a single file. If `false`, uses recursive copy for directories. If not specified, automatically detects based on the source path. |
| on\_failure  | No       | What to do if the copy fails (abort/warn/quiet/skip)                                                                                                                                                                         |

## Output

When the copy hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
from_uri: "s3://bucket/path/to/file"  # The normalized URI of the source file
from_path: "/path/to/source/file"  # The source path
to_uri: "gcs://bucket/path/to/file"  # The normalized URI of the destination file
to_path: "/path/to/dest/file"  # The destination path
bytes_written: 1024  # Number of bytes written
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.from_uri}` - The normalized URI of the source file
* `{state.hook_id.from_path}` - The source path
* `{state.hook_id.to_uri}` - The normalized URI of the destination file
* `{state.hook_id.to_path}` - The destination path
* `{state.hook_id.bytes_written}` - Number of bytes written

## Examples

### Archive Files Between Cloud Storage

Archive files between different cloud storage providers:

```yaml
hooks:
  post:
    - type: copy
      if: run.status == "success"
      from: "aws_s3/{run.object.file_path}"
      to: "gcs/archives/{target.name}/{timestamp.YYYY}/{timestamp.MM}/{run.object.file_path}"
      on_failure: warn
```

### Upload A Local DuckDB Database into S3

Copy a local database file into Amazon S3 after writing to it:

```yaml
hooks:
  end:
    - type: copy
      from: "{target.instance}"
      to: "aws_s3/duckdb/{env.FOLDER}/backup.db"
      on_failure: warn
```

### Copy Multiple Files Pattern

Copy multiple files matching a pattern:

```yaml
hooks:
  post:
    - type: copy
      from: "local//tmp/exports/*.parquet"
      to: "gcs/data-lake/raw/{run.stream.name}/"
      on_failure: abort
```

### Force Single File Copy

Force treating the source as a single file, even if it could be interpreted as a pattern:

```yaml
hooks:
  post:
    - type: copy
      from: "s3/data/file.json"
      to: "local//backup/file.json"
      single_file: true
      on_failure: abort
```

### Force Directory Copy

Force recursive directory copying:

```yaml
hooks:
  post:
    - type: copy
      from: "local//data/exports"
      to: "s3/backup/exports"
      single_file: false
      on_failure: warn
```


# Delete

Delete hooks allow you to remove files or directories from local or remote storage locations. This is useful for cleanup operations, removing temporary files, or maintaining storage quotas.

## Configuration

```yaml
- type: delete
  location: "aws_s3/path/to/file"   # Required: Location string
  recursive: false        # Optional: true/false. Default is false
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                       |
| ----------- | -------- | ------------------------------------------------------------------------------------------------- |
| location    | Yes      | The [location](/sling-cli/environment#location-string) string. Contains connection name and path. |
| recursive   | No       | Whether to delete recursively                                                                     |
| path        | Yes      | The path to the file or directory to delete                                                       |
| on\_failure | No       | What to do if the deletion fails (abort/warn/quiet/skip)                                          |

## Output

When the delete hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
path: "path/to/file"  # The path that was deleted
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.path}` - The path that was deleted

## Examples

### Clean Up Temporary Files

Remove temporary files after successful processing:

```yaml
hooks:
  post:
    - type: delete
      location: "local//tmp/processed/{run.stream.name}_{timestamp.date}/*"
      on_failure: warn
```

### Clean Up Staging Area

Remove processed files from a staging area after successful replication:

```yaml
hooks:
  post:
    - type: delete
      if: run.status == "success"
      location: "azure_blob/staging/{target.environment}/{run.stream.name}/"
      recursive: true
      on_failure: warn
```

### Remove Failed Processing Artifacts

Clean up artifacts from failed processing attempts:

```yaml
hooks:
  post:
    - type: delete
      if: run.status == "error"
      location: "gcs/failed-jobs/{timestamp.date}/{run.stream.name}/"
      on_failure: quiet
```

### Clean Up Log Files

Remove old log files after successful processing:

```yaml
hooks:
  post:
    - type: delete
      location: "local//var/log/sling/{run.stream.name}/*.log"
      on_failure: quiet
```


# Group

Group hooks allow you to execute a sequence of steps, optionally in a loop. This is particularly useful for running multiple operations together, iterating over datasets, or creating reusable step templates.

## Configuration

```yaml
- type: group
  steps:                # Required: Array of steps to execute
    - type: log
      message: "Step 1"
    - type: http
      url: "https://api.example.com"
  loop: [1, 2, 3]      # Optional: Array or jmespath expression for iteration
  env:                 # Optional: Environment variables for all steps
    ENV_VAR1: "value1"
    ENV_VAR2: "value2"
  on_failure: abort    # Optional: abort/warn/quiet/skip
  id: my_id           # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                            |
| ----------- | -------- | ------------------------------------------------------ |
| steps       | Yes      | Array of step configurations to execute                |
| loop        | No       | Array or jmespath expression defining iteration values |
| env         | No       | Map of environment variables available to all steps    |
| on\_failure | No       | What to do if any step fails (abort/warn/quiet/skip)   |

## Output

When the group hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
loop_values: 5   # The rendered values if loop was provided
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.loop_values}` - Values used in loop (if provided)

Within loop iterations, steps can access:

* `{loop.index}` - Current iteration index (0-based)
* `{loop.value}` - Current value from the loop array/expression

## Examples

### Basic Step Group

Execute multiple steps in sequence:

```yaml
- type: group
  id: initialization
  steps:
    - type: log
      message: "Starting initialization"
    - type: query
      connection: target_db
      query: "CREATE SCHEMA IF NOT EXISTS processed"
    - type: log
      message: "Initialization complete"
```

### Loop Over Array

Iterate over a fixed array of values:

```yaml
- type: group
  id: process_regions
  loop: ["us-east", "us-west", "eu-central"]
  steps:
    - type: log
      message: "Processing region: {loop.value}"
    - type: query
      connection: target_db
      query: |
        UPDATE region_status 
        SET last_checked = CURRENT_TIMESTAMP
        WHERE region_name = '{loop.value}'
```

### Loop Over Previous Hook Results

Use results from a previous hook as loop input:

```yaml
- type: list
  id: file_list
  location: aws_s3/data/daily/
  only: files    # return only files

- type: group
  id: process_files
  loop: state.file_list.result
  steps:
    - type: log
      message: "Processing file: {loop.value.name}"

    - type: download
      connection: aws_s3
      remote_path: "data/daily/{loop.value.name}"
```

### Conditional Step Execution in Loop

Execute steps conditionally within a loop:

```yaml
- type: group
  id: validate_tables
  loop: ["users", "orders", "products"]
  steps:
    - type: query
      connection: source_db
      id: get_count
      query: |
        SELECT COUNT(*) as row_count 
        FROM {loop.value}
    - type: log
      if: get_count.result[0].row_count > 1000
      message: "Table {loop.value} has {get_count.result[0].row_count} rows"
```

### Environment Variables in Group

Share environment variables across steps:

```yaml
- type: group
  id: backup_process
  env:
    BACKUP_DATE: "{timestamp.date}"
    BACKUP_PATH: "backups/{target.environment}"
  steps:
    - type: log
      message: "Starting backup for {env.BACKUP_DATE}"
    - type: copy
      from: "source_storage/{run.stream.name}"
      to: "{env.BACKUP_PATH}/{env.BACKUP_DATE}/{run.stream.name}"
```

### Nested Groups

Create hierarchical step organization:

```yaml
- type: group
  id: main_process
  loop: ["staging", "production"]
  steps:
    - type: log
      message: "Processing environment: {loop.value}"
    - type: group
      steps:
        - type: query
          connection: "{loop.value}_db"
          query: "VACUUM ANALYZE my_table"
        - type: log
          message: "Maintenance complete for {loop.value}"
```


# Http

HTTP hooks enable you to make HTTP requests to external services as part of your replication workflow. This is particularly useful for integrating with external APIs, sending notifications, or triggering other systems.

## Configuration

```yaml
- type: http
  url: "https://api.example.com/webhook"  # Required
  method: GET                             # Optional: GET/POST/PUT/DELETE (default: GET)
  payload: '{"status": "{run.status}"}'   # Optional: Request body (also accepts YAML object)
  timeout: 10                             # Optional: Request timeout seconds (default is 30sec)
  headers:                                # Optional: Request headers
    Authorization: "Bearer token"
  auth:                                   # Optional: Authentication configuration
    type: basic                           # basic, aws-sigv4, or hmac
    username: "myuser"
    password: "mypass"
  proxy: "http://user:pass@proxy.example.com:8080"  # Optional: HTTP/HTTPS proxy URL
  write_to: local/path/to/response.json   # Optional: Save response to a file
  on_failure: abort                       # Optional: abort/warn/quiet/skip
  id: my_id                               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                                                                             |
| ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| url         | Yes      | The URL to send the request to                                                                                                                          |
| method      | No       | HTTP method (GET/POST/PUT/DELETE). Defaults to GET                                                                                                      |
| payload     | No       | The request body (for POST/PUT requests)                                                                                                                |
| headers     | No       | Map of HTTP headers to include in the request                                                                                                           |
| auth        | No       | Authentication configuration (see [Authentication](#authentication) section)                                                                            |
| proxy       | No       | HTTP/HTTPS proxy URL (e.g., `http://user:pass@proxy.example.com:8080`)                                                                                  |
| write\_to   | No       | [Location](/sling-cli/environment#location-string) to save the response bytes to a file (e.g., `local/path/to/response.json`, `s3/folder/response.csv`) |
| on\_failure | No       | What to do if the request fails (abort/warn/quiet/skip)                                                                                                 |

## Output

When the HTTP hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
request:  # Details of the request made
  method: "GET"  # HTTP method used
  url: "https://api.example.com/webhook"  # The URL called
  headers:  # Headers sent with the request
    Authorization: "Bearer token"
  payload: '{"status": "success"}'  # The request body sent
response:  # Details of the response received
  headers:  # Response headers
    Content-Type: "application/json"
    # ... other response headers
  status: "200 OK"  # HTTP status message
  status_code: 200  # HTTP status code
  size: 123456  # Response bytes size 
  text: "Response body as text"  # Raw response body
  json:  # Parsed JSON response (if response is JSON)
    key: "value"
    # ... rest of JSON structure
written_to:  # Details of saved response file (if write_to is specified)
  location: "local/path/to/response.json"  # The write_to location
  uri: "file:///absolute/path/to/response.json"  # Absolute URI of saved file
  bytes_written: 1234  # Number of bytes written to file
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.request.method}` - HTTP method used
* `{state.hook_id.request.url}` - The URL called
* `{state.hook_id.request.headers}` - Request headers
* `{state.hook_id.request.payload}` - Request body
* `{state.hook_id.response.status}` - HTTP status message
* `{state.hook_id.response.status_code}` - HTTP status code
* `{state.hook_id.response.text}` - Raw response body
* `{state.hook_id.response.json}` - Parsed JSON response
* `{state.hook_id.response.size}` - Size of response in bytes
* `{state.hook_id.written_to.location}` - Location where response was saved
* `{state.hook_id.written_to.uri}` - Absolute URI of saved response file

## Examples

### Slack Notification

Send a notification to Slack after replication completes:

```yaml
hooks:
  post:
    - type: http
      url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
      method: POST
      payload: |
        {
          "text": "Replication Status Update",
          "blocks": [
            {
              "type": "section",
              "text": {
                "type": "mrkdwn",
                "text": "*Stream:* {run.stream.name}\n*Status:* {run.status}\n*Rows Processed:* {run.total_rows}\n*Duration:* {run.duration} seconds"
              }
            }
          ]
        }
      on_failure: warn
```

### Microsoft Teams Alert

Send an alert to Microsoft Teams when replication fails:

```yaml
hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://your-teams-webhook-url"
      method: POST
      payload: |
        {
          "@type": "MessageCard",
          "@context": "http://schema.org/extensions",
          "themeColor": "FF0000",
          "summary": "Replication Failed",
          "sections": [{
            "activityTitle": "⚠️ Replication Failed",
            "facts": [
              {
                "name": "Stream",
                "value": "{run.stream.name}"
              },
              {
                "name": "Target Table",
                "value": "{run.object.full_name}"
              },
              {
                "name": "Start Time",
                "value": "{run.start_time}"
              },
              {
                "name": "End Time",
                "value": "{run.end_time}"
              }
            ]
          }]
        }
```

### REST API Integration

Fetch configuration from an external API before starting replication:

```yaml
hooks:
  pre:
    - type: http
      url: "https://api.company.com/v1/config/{run.stream.name}"
      method: GET
      headers:
        Authorization: "Bearer {source.api_key}"
        Content-Type: "application/json"
```

### API with Basic Authentication

Call an API endpoint that requires basic authentication:

```yaml
hooks:
  post:
    - type: http
      url: "https://api.example.com/v1/notify"
      method: POST
      auth:
        type: basic
        username: "{env.API_USER}"
        password: "{env.API_PASS}"
      payload: |
        {
          "event": "replication_complete",
          "stream": "{run.stream.name}",
          "rows": {run.total_rows}
        }
```

### API with HMAC Signature

Call an API that requires HMAC signature authentication:

```yaml
hooks:
  post:
    - type: http
      url: "https://partner-api.example.com/webhooks/events"
      method: POST
      auth:
        type: hmac
        algorithm: sha256
        secret: "{env.PARTNER_HMAC_SECRET}"
        signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
        request_headers:
          X-Signature: "sha256={signature}"
          X-Timestamp: "{unix_time}"
      payload: |
        {
          "event_type": "replication.complete",
          "stream": "{run.stream.name}",
          "status": "{run.status}",
          "rows_processed": {run.total_rows},
          "timestamp": "{run.end_time}"
        }
```

### Trigger External Workflow

Trigger an external workflow system after successful replication:

```yaml
hooks:
  post:
    - type: http
      if: run.status == "success"
      url: "https://api.workflow-system.com/v1/triggers"
      method: POST
      headers:
        Authorization: "ApiKey {target.api_key}"
      payload: |
        {
          "workflow_id": "data-quality-check",
          "parameters": {
            "table_name": "{run.object.full_name}",
            "row_count": {run.total_rows},
            "execution_date": "{timestamp.date}"
          }
        }
```

### Data Quality Service Integration

Send data quality metrics to a monitoring service:

```yaml
hooks:
  post:
    - type: http
      url: "https://metrics-api.company.com/v1/metrics"
      method: POST
      headers:
        X-API-Key: "{target.metrics_api_key}"
      payload: |
        {
          "metric_type": "data_quality",
          "timestamp": "{run.end_time}",
          "metrics": {
            "table_name": "{run.object.full_name}",
            "record_count": {run.total_rows},
            "processing_time_seconds": {run.duration},
            "bytes_processed": {run.total_bytes}
          },
          "tags": {
            "environment": "{env.environment}",
            "stream": "{run.stream.name}"
          }
        }
```

### Error Tracking Integration

Send error details to an error tracking service when replication fails:

```yaml
hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://api.errortrackers.com/v1/errors"
      method: POST
      headers:
        Authorization: "Bearer {target.error_tracking_token}"
      payload: |
        {
          "error": {
            "name": "Replication Failed",
            "environment": "{target.environment}",
            "metadata": {
              "stream": "{run.stream.name}",
              "target_table": "{run.object.full_name}",
              "start_time": "{run.start_time}",
              "end_time": "{run.end_time}",
              "rows_processed": {run.total_rows}
            }
          }
        }
      on_failure: warn
```

### Download and Save API Response

Fetch data from an API and save the response to a file:

```yaml
hooks:
  pre:
    - type: http
      id: fetch_config
      url: "https://api.example.com/v1/config"
      method: GET
      headers:
        Authorization: "Bearer {env.API_TOKEN}"
      write_to: local/configs/api_response.json
    
    # Use the saved response location in a subsequent hook
    - type: log
      message: "API response saved to {state.fetch_config.written_to.uri} ({state.fetch_config.written_to.bytes_written} bytes)"
```

## Authentication

The HTTP hook supports multiple authentication methods through the `auth` property:

### Basic Authentication

Use HTTP Basic Authentication with username and password:

```yaml
- type: http
  url: "https://api.example.com/endpoint"
  auth:
    type: basic
    username: "{env.API_USERNAME}"
    password: "{env.API_PASSWORD}"
```

### AWS Signature V4

Sign requests using AWS Signature Version 4 (for AWS services):

```yaml
- type: http
  url: "https://my-service.us-east-1.amazonaws.com/endpoint"
  auth:
    type: aws-sigv4
    aws_service: execute-api
    aws_region: us-east-1
    aws_access_key_id: "{env.AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "{env.AWS_SECRET_ACCESS_KEY}"
    # Optional:
    # aws_session_token: "{env.AWS_SESSION_TOKEN}"
    # aws_profile: "my-profile"
```

**Note**: AWS credentials can also be loaded from environment variables or AWS profiles if not explicitly provided.

### HMAC Signature

Sign requests using HMAC (Hash-based Message Authentication Code) for custom API authentication:

```yaml
- type: http
  url: "https://api.example.com/endpoint"
  auth:
    type: hmac
    algorithm: sha256  # or sha512 (default: sha256)
    secret: "{env.HMAC_SECRET}"
    signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
    request_headers:
      X-Signature: "{signature}"
      X-Timestamp: "{unix_time}"
      # Optional nonce:
      # X-Nonce: "{nonce}"
    # nonce_length: 16  # Optional: generate random nonce (in bytes)
```

**Available template variables for `signing_string` and `request_headers`:**

* `{http_method}` - HTTP method (GET, POST, etc.)
* `{http_path}` - Request path including query string
* `{http_body_md5}` - MD5 hash of request body (hex)
* `{http_body_sha1}` - SHA1 hash of request body (hex)
* `{http_body_sha256}` - SHA256 hash of request body (hex)
* `{http_body_sha512}` - SHA512 hash of request body (hex)
* `{http_body_raw}` - Raw request body as string
* `{http_query}` - Canonical query string (sorted, URL-encoded)
* `{http_headers}` - Canonical headers string (lowercase, sorted)
* `{unix_time}` - Current Unix timestamp (seconds)
* `{unix_time_ms}` - Current Unix timestamp (milliseconds)
* `{date_iso}` - Current date in ISO 8601 format (RFC3339)
* `{date_rfc1123}` - Current date in RFC1123 format
* `{nonce}` - Random nonce (if `nonce_length` is set)
* `{signature}` - The computed HMAC signature (available in `request_headers`)

**Example with complete HMAC flow:**

```yaml
- type: http
  url: "https://api.partner.com/v1/orders"
  method: POST
  payload: '{"order_id": "12345"}'
  auth:
    type: hmac
    algorithm: sha256
    secret: "{env.PARTNER_API_SECRET}"
    signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
    nonce_length: 16
    request_headers:
      Authorization: "HMAC-SHA256 {signature}"
      X-Timestamp: "{unix_time}"
      X-Nonce: "{nonce}"
```

### Bearer Token (via Headers)

For Bearer token authentication, use the `headers` property instead of `auth`:

```yaml
- type: http
  url: "https://api.example.com/endpoint"
  headers:
    Authorization: "Bearer {env.API_TOKEN}"
```

## Proxy Configuration

If your network requires routing HTTP requests through a proxy server, you can configure it using the `proxy` property:

```yaml
- type: http
  url: "https://api.example.com/endpoint"
  proxy: "http://proxy.company.com:8080"
```

### Proxy with Authentication

For proxies that require authentication, include credentials in the proxy URL:

```yaml
- type: http
  url: "https://api.example.com/endpoint"
  proxy: "http://{env.PROXY_USER}:{env.PROXY_PASS}@proxy.company.com:8080"
```

The proxy URL format is: `http://[username:password@]host:port`

**Notes:**

* Both HTTP and HTTPS proxies are supported
* Authentication credentials in the proxy URL are automatically handled
* The proxy applies to all HTTP/HTTPS requests made by this hook


# Inspect

The inspect hook allows you to retrieve metadata about files, directories, or database objects from any supported connection. This is particularly useful for validating existence, checking properties, or monitoring changes.

## Configuration

```yaml
- type: inspect
  location: connection_name/path_or_object  # Required: Location string (database or storage)
  recursive: true/false                    # Optional: For files, get nested file stats
  on_failure: abort                        # Optional: abort/warn/quiet/skip
  id: my_id                               # Optional: Generated if not provided
```

## Properties

| Property    | Required | Description                                                                                                                                                                                      |
| ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| location    | Yes      | The [location](/sling-cli/environment#location-string) string. Can be a **database location** (e.g., `postgres/public.table_name`) or **storage location** (e.g., `aws_s3/bucket/path/file.txt`) |
| recursive   | No       | For file systems: whether to get total count/size of nested files                                                                                                                                |
| on\_failure | No       | What to do if the inspection fails (abort/warn/quiet/skip)                                                                                                                                       |

### Location Examples

* **Database**: `postgres/public.users`, `mysql_db/analytics.events`, `snowflake/DATABASE.SCHEMA.TABLE`
* **Storage**: `aws_s3/bucket/data/file.csv`, `local//tmp/data/file.json`, `gcs/bucket/folder/`

## Output

### File System Objects

When inspecting files or directories, the hook returns:

```yaml
status: success                          # Status of the hook execution
exists: true                            # Whether the path exists
path: "path/to/file"                    # The normalized path
name: "file"                           # The name of the file/directory
uri: "s3://bucket/path/to/file"        # The full URI
is_dir: false                          # Whether the path is a directory
size: 1024                             # File size in bytes
node_count: 5                          # Total nodes (if recursive)
folder_count: 2                        # Total folders (if recursive)
file_count: 3                          # Total files (if recursive)
created_at: "2023-01-01T00:00:00Z"     # Creation timestamp if available
created_at_unix: 1672531200            # Creation unix timestamp if available
updated_at: "2023-01-02T00:00:00Z"     # Last modified timestamp if available
updated_at_unix: 1672617600            # Last modified unix timestamp if available
```

### Database Objects

When inspecting database tables, the hook returns:

```yaml
status: success                         # Status of the hook execution
exists: true                           # Whether the table exists
database: "my_database"                # Database name
schema: "public"                       # Schema name
name: "my_table"                       # Table name
fdqn: "my_database.public.my_table"    # Fully qualified name
columns:                               # Array of column details
  - name: "id"
    type: "integer"
    db_type: "int4"
    position: 1
  - name: "name"
    type: "string"
    db_type: "varchar"
    position: 2
    precision: 255
column_names: ["id", "name"]           # Array of column names
column_map:                            # Map of columns by name
  id:
    name: "id"
    type: "integer"
    db_type: "int4"
    position: 1
  name:
    name: "name"
    type: "string"
    db_type: "varchar"
    position: 2
    precision: 255
```

## Accessing Output Data

You can access these values in subsequent hooks using JMESPath syntax:

### File System Data

* `{state.hook_id.exists}` - Whether the path exists
* `{state.hook_id.size}` - File size in bytes
* `{state.hook_id.is_dir}` - Whether it's a directory
* `{state.hook_id.created_at}` - Creation timestamp
* `{state.hook_id.updated_at}` - Last modified timestamp

### Database Data

* `{state.hook_id.exists}` - Whether the table exists
* `{state.hook_id.column_names}` - Array of column names
* `{state.hook_id.column_map.column_name.type}` - Specific column type
* `{state.hook_id.fdqn}` - Fully qualified table name

## Examples

### Database Examples

#### Verify Table Exists

Check if a required table exists:

```yaml
steps:
  - type: inspect
    id: table_check
    location: postgres/public.customer_data

  - type: check
    check: state.table_check.exists == true
    failure_message: "Required table does not exist"
    on_failure: abort
```

#### Validate Table Schema

Ensure required columns exist and check column properties:

```yaml
steps:
  - type: inspect
    id: schema_check
    location: postgres/public.test_table

  - type: log
    message: |
      Table inspection results:
      - Exists: {state.schema_check.exists}
      - Database: {state.schema_check.database}
      - Schema: {state.schema_check.schema}
      - Name: {state.schema_check.name}
      - FDQN: {state.schema_check.fdqn}
      - Column Count: {length(state.schema_check.columns)}

  - type: check
    check: contains(state.schema_check.column_names, 'user_id') && contains(state.schema_check.column_names, 'event_timestamp')
    failure_message: "Table missing required columns"
    on_failure: abort

  - type: check
    check: state.schema_check.columns[2].precision == 10
    failure_message: "Column precision mismatch"
```

#### Check Column Types

Validate specific column data types:

```yaml
steps:
  - type: inspect
    id: column_check
    location: mysql_db/analytics.user_events

  - type: check
    check: state.column_check.column_map.created_at.type == 'datetime'
    failure_message: "created_at column must be datetime type"
    on_failure: warn
```

### File System Examples

#### Verify File Existence

Check if a required file exists before processing:

```yaml
steps:
  - type: inspect
    id: config_check
    location: aws_s3/data/config/{run.stream.name}.json

  - type: check
    check: state.config_check.exists == true
    failure_message: "Config file not found for stream {run.stream.name}"
    on_failure: abort
```

#### Check File Properties

Ensure a file meets requirements:

```yaml
steps:
  - type: inspect
    id: file_check
    location: local//tmp/data/input.csv

  - type: log
    message: |
      File inspection results:
      - Exists: {state.file_check.exists}
      - Path: {state.file_check.path}
      - Name: {state.file_check.name}
      - Size: {state.file_check.size}
      - Is Directory: {state.file_check.is_dir}

  - type: check
    check: state.file_check.size > 0
    failure_message: "Input file is empty"
    on_failure: abort

  - type: check
    check: state.file_check.name == "input.csv"
    failure_message: "Wrong file name"
```

#### Directory Inspection with Recursive Stats

Get total file count and size in a directory:

```yaml
steps:
  - type: inspect
    id: dir_stats
    location: aws_s3/bucket/data/
    recursive: true

  - type: log
    message: |
      Directory inspection results (recursive):
      - Total Size: {state.dir_stats.size}
      - Node Count: {state.dir_stats.node_count}
      - File Count: {state.dir_stats.file_count}
      - Folder Count: {state.dir_stats.folder_count}

  - type: check
    check: state.dir_stats.file_count >= 1
    failure_message: "Directory should contain at least one file"
```

#### File Age Check

Skip processing if file is too old:

```yaml
steps:
  - type: inspect
    id: age_check
    location: local//tmp/data/input.csv

  - type: check
    check: state.age_check.updated_at_unix >= (timestamp.unix - 86400)
    failure_message: "File is older than 24 hours"
    on_failure: skip
```

## Notes

* **Database Locations**: Use the format `connection_name/database.schema.table` or `connection_name/schema.table` depending on your database
* **Storage Locations**: Use the format `connection_name/path/to/file` or `connection_name/path/to/directory/`
* **File Systems**: Not all filesystems provide all metadata fields
* **File Systems**: Timestamps may be zero if not supported by the filesystem
* **File Systems**: Directory sizes are typically reported as 0 unless `recursive: true`
* **Databases**: Column precision and scale are only populated for decimal/numeric types
* **General**: The hook will not fail if the path/object is invalid; it returns `exists: false`
* **General**: Use appropriate connection types (file connections for files, database connections for tables)


# List

List hooks allow you to retrieve file and directory listings from any supported filesystem connection. This is particularly useful for discovering files, validating directory contents, or preparing for batch operations.

## Configuration

```yaml
- type: list
  location: "aws_s3/path/to/directory"  # Required: Location string
  recursive: false      # Optional: List files/folders recursively (default: false)
  only: files | folders  # Optional: List only files or only folders
  into: my_variable    # Optional: Store results in store or env (e.g., "file_list" or "env.FILE_LIST")
  on_failure: abort    # Optional: abort/warn/quiet/skip
  id: my_id           # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                                                                                                             |
| ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| location    | Yes      | The [location](/sling-cli/environment#location-string) string. Contains connection name and path.                                                                                       |
| recursive   | No       | Whether to list files recursively in subdirectories (default: false)                                                                                                                    |
| only        | No       | Filter to list only "files" or only "folders"                                                                                                                                           |
| into        | No       | Store the result array in the replication store or environment variables. Use `store.variable_name` or just `variable_name` for store, or `env.VARIABLE_NAME` for environment variables |
| on\_failure | No       | What to do if the listing fails (abort/warn/quiet/skip)                                                                                                                                 |

## Output

When the list hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
result:  # Array of file/directory entries
  - name: "file1.txt"  # Name of the file/directory
    path: "path/to/file1.txt"  # Full path
    location: "my_conn/path/to/file1.txt"  # Location string
    uri: "s3://bucket/path/to/file1.txt"  # Full URI
    is_file: true  # Whether entry is a file
    is_dir: false  # Whether entry is a directory
    size: 1024  # Size in bytes
    created_at: "2023-01-01T00:00:00Z"  # Creation timestamp if available
    created_at_unix: 1672531200  # Creation unix timestamp if available
    updated_at: "2023-01-02T00:00:00Z"  # Last modified timestamp if available
    updated_at_unix: 1672617600  # Last modified unix timestamp if available
path: "path/to/directory"  # The listed path
connection: "aws_s3"  # The connection used
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.result}` - Array of file/directory entries
* `{state.hook_id.path}` - The listed path
* `{state.hook_id.connection}` - The connection used

## Examples

### Process Files in Directory

List files and process them in a group:

```yaml
hooks:
  pre:
    - type: list
      id: file_list
      location: "aws_s3/data/{run.stream.name}/"
      recursive: true

    - type: group
      loop: state.file_list.result
      steps:
        - type: log
          if: loop.value.is_file
          message: "Processing file: {loop.value.name}"
```

### Archive Old Files

List and archive files older than a certain date:

```yaml
hooks:
  post:
    - type: list
      id: old_files
      location: "gcs/temp/{run.stream.name}/"
      recursive: true

    - type: group
      loop: state.old_files.result
      steps:
        - type: copy
          if: loop.value.updated_at_unix < timestamp.unix - 7*24*60*60  # 7 days old
          from: "{loop.value.location}"
          to: "gcs/archive/{timestamp.year}/{timestamp.month}/{loop.value.name}"
```

### Size-based Processing

Process files based on their size:

```yaml
hooks:
  pre:
    - type: list
      id: large_files
      location: "aws_s3/uploads/"

    - type: group
      loop: state.large_files.result
      steps:
        - type: log
          if: loop.value.size > 1024*1024  # > 1MB
          message: "Large file detected: {loop.value.name} ({loop.value.size} bytes)"
```

### Store Results for Later Use

Use the `into` parameter to store list results in the store or environment variables for use across pipeline steps:

```yaml
steps:
  # List files and store in replication store
  - type: list
    location: "s3/data/inbox/"
    recursive: true
    only: files
    into: inbox_files  # Store in replication store

  # Use stored results in subsequent step
  - type: log
    message: "Found {len(store.inbox_files)} files in inbox"

  # Process each file
  - type: group
    loop: store.inbox_files
    steps:
      - type: log
        message: "Processing: {loop.value.name} ({loop.value.size} bytes)"
```

### Store Results as Environment Variable

Store list results as a JSON environment variable for use in subsequent pipeline steps or replications:

```yaml
steps:
  # List files and store as environment variable
  - type: list
    location: "local/exports/"
    only: files
    into: env.EXPORT_FILES  # Store as environment variable (JSON string)

  # The env var is now available in subsequent steps
  - type: log
    message: "Export files available: {env.EXPORT_FILES}"

  # Run replication that can access the env var
  - type: replication
    path: /path/to/replication.yaml
```

### Store File Paths for API Iteration

List files and use their paths to drive API endpoint iteration:

```yaml
steps:
  # List CSV files to process
  - type: list
    location: "s3/data/uploads/"
    recursive: false
    only: files
    into: upload_files

  # Store just the file paths
  - type: store
    key: env.FILE_PATHS
    value: >
      {join(map(store.upload_files, "location"), ",")}

  # Log the files that will be processed
  - type: log
    message: "Will process files: {env.FILE_PATHS}"

  # Run replication using file list
  - type: replication
    path: /path/to/process_files.yaml
```

### Combined with Query Results

Combine list results with database queries:

```yaml
steps:
  # List available data files
  - type: list
    location: "s3/data/raw/"
    recursive: true
    only: files
    into: raw_files

  # Query database for already processed files
  - type: query
    connection: MY_DB
    query: |
      SELECT filename
      FROM processed_files
      WHERE processed_date > CURRENT_DATE - INTERVAL '7 days'
    into: processed_files

  # Store count of new files to process
  - type: store
    key: new_file_count
    value: >
      {len(store.raw_files) - len(store.processed_files)}

  # Log processing status
  - type: log
    message: |
      File Processing Status:
      - Total raw files: {len(store.raw_files)}
      - Already processed: {len(store.processed_files)}
      - New files to process: {store.new_file_count}

  # Process only new files
  - type: group
    if: store.new_file_count > 0
    loop: store.raw_files
    steps:
      - type: log
        message: "Processing new file: {loop.value.name}"
```

## Notes

* Not all filesystems provide all metadata fields
* Timestamps may be zero if not supported by the filesystem
* Directory sizes are typically reported as 0
* The hook will not fail if the path doesn't exist or is empty
* When using `into`, the result array is stored directly without the wrapping object (no need to access `.result`)
* Use `into: "variable_name"` for replication store (accessible via `{store.variable_name}`)
* Use `into: "env.VARIABLE_NAME"` for environment variables (accessible via `{env.VARIABLE_NAME}`, stored as JSON string)


# Log

Log hooks allow you to output custom messages during the replication process. This is useful for debugging, monitoring, and creating audit trails of your data pipeline operations.

## Configuration

```yaml
- type: log
  message: "Custom log message"  # Required: The message to log
  level: info                    # Optional: Log level (info/warn/debug)
  on_failure: abort              # Optional: abort/warn/quiet/skip
```

## Properties

| Property    | Required | Description                                         |
| ----------- | -------- | --------------------------------------------------- |
| message     | Yes      | The message to log                                  |
| level       | No       | Log level (info/warn/debug). Defaults to `info`     |
| on\_failure | No       | What to do if logging fails (abort/warn/quiet/skip) |

## Output

When the log hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
level: "info"    # The log level used
message: "Custom log message"  # The message that was logged
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.level}` - The log level used
* `{state.hook_id.message}` - The message that was logged

## Examples

### Print out Runtime State

Log the basic status of a stream after completion:

```yaml
hooks:
  post:
    - type: log
      message: "runtime_state => {runtime_state}"

    - type: log
      message: "Stream {run.stream.name} completed with status {run.status}. Processed {run.total_rows} rows."
```

### Conditional Warning Log

Log a warning when row count is below threshold:

```yaml
hooks:
  post:
    - type: log
      if: "run.total_rows < 1000"
      message: "⚠️ Warning: Low record count for {run.stream.name}. Only processed {run.total_rows} rows."
      level: warn
```

### Debug Information

Log detailed information before starting the replication:

```yaml
hooks:
  pre:
    - type: log
      message: |
        Starting replication for:
        Stream: {run.stream.name}
        Source Schema: {run.stream.schema}
        Target Table: {run.object.full_name}
        Environment: {target.environment}
        Timestamp: {timestamp.datetime}
      level: debug
```

### Performance Metrics Logging

Log performance metrics after successful completion:

```yaml
hooks:
  post:
    - type: log
      if: run.status == "success"
      message: |
        Performance Metrics for {run.stream.name}:
        - Total Rows: {run.total_rows}
        - Total Bytes: {run.total_bytes}
        - Duration: {run.duration} seconds
        - Processing Rate: {run.total_rows / (run.duration)} rows/second
      level: info
```

### Error Context Logging

Log detailed context when errors occur:

```yaml
hooks:
  post:
    - type: log
      if: run.status == "error"
      message: |
        ❌ Error in stream {run.stream.name}:
        - Source: {source.type}
        - Target: {target.type}
        - Object: {run.object.full_name}
        - Environment: {target.environment}
        - Start Time: {run.start_time}
        - End Time: {run.end_time}
        Please check the logs for more details.
      level: warn
```

### Audit Trail Logging

Create a detailed audit trail of replication activities:

```yaml
hooks:
  pre:
    - type: log
      message: |
        🚀 Starting replication task:
        - Stream: {run.stream.name}
        - Mode: {run.mode}
        - Environment: {target.environment}
        - User: {source.user}
        - Start Time: {timestamp.datetime}
      level: info
  post:
    - type: log
      message: |
        📋 Replication task completed:
        - Stream: {run.stream.name}
        - Status: {run.status}
        - Rows Processed: {run.total_rows}
        - Duration: {run.end_time - run.start_time} seconds
        - End Time: {timestamp.datetime}
      level: info
```

### Data Quality Logging

Log data quality metrics after processing:

```yaml
hooks:
  post:
    - type: log
      message: |
        Data Quality Report for {run.stream.name}:
        - Total Records: {run.total_rows}
        - Null Rate: {state.quality_check.result.null_rate}%
        - Duplicate Rate: {state.quality_check.result.duplicate_rate}%
        - Invalid Format Rate: {state.quality_check.result.invalid_rate}%
      level: info
```

### 8. Environment-Specific Logging (Pre-Hook)

Adjust log verbosity based on environment:

```yaml
hooks:
  pre:
    - type: log
      if: 'target.environment == "production"'
      message: "⚠️ Running production replication for {run.stream.name}"
      level: warn
    - type: log
      if: 'target.environment != "production"'
      message: "Starting {target.environment} replication for {run.stream.name}"
      level: debug
```

### 9. Resource Usage Logging (Post-Hook)

Log resource usage statistics:

```yaml
hooks:
  post:
    - type: log
      message: |
        Resource Usage for {run.stream.name}:
        - Memory Used: {state.resource_check.result.memory_used}
        - CPU Usage: {state.resource_check.result.cpu_usage}%
        - Disk IO: {state.resource_check.result.disk_io_bytes} bytes
      level: debug
```

### 10. Milestone Logging (Post-Hook)

Log important milestones during processing:

```yaml
hooks:
  post:
    - type: log
      if: "run.total_rows >= 1000000"
      message: "🏆 Milestone: Processed over 1 million records in {run.stream.name}"
      level: info
    - type: log
      if: "run.total_bytes >= 1073741824"  # 1GB
      message: "🏆 Milestone: Processed over 1GB of data in {run.stream.name}"
      level: info
```


# Query

Query hooks allow you to execute SQL queries against any defined connection in your environment. This is particularly useful for pre and post-processing tasks, data validation, and maintaining metadata.

## Configuration

```yaml
- type: query
  connection: target_db   # Connection name (required)
  query:      "SELECT * FROM table"  # Required
  into:       "my_results"  # Optional: Store results in variable
  transient:  false       # Optional: Use transient connection
  transaction: true      # Optional: Use transaction, or set isolation level
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id:         my_id       # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                                                                                                                                                      |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| connection  | Yes      | The name of the connection to execute the query against                                                                                                                                                                          |
| query       | Yes      | The SQL query to execute (can also be the path of a local `.sql` file with `file://` prefix)                                                                                                                                     |
| into        | No       | Variable name to store the query results. If not specified, results are included in hook output.                                                                                                                                 |
| transient   | No       | Whether to use a transient connection (default: false)                                                                                                                                                                           |
| transaction | No       | Whether to use transaction, and optionally set isolation level. Ideal for multiple statements. Supported values: `default`, `true` ( same as `default`), `read_uncommitted`, `read_committed`, `repeatable_read`, `serializable` |
| on\_failure | No       | What to do if the query fails (abort/warn/quiet/skip)                                                                                                                                                                            |

## Output

When the query hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
query: "SELECT * FROM table"  # The executed query
connection: "target_db"  # The connection used
columns: ["column1", "column2", ...]  # List of column names from the result
result:  # Array of records from the query result (only if 'into' is not specified)
  - column1: value1
    column2: value2
  - column1: value3
    column2: value4
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.query}` - The executed query
* `{state.hook_id.connection}` - The connection used
* `{state.hook_id.columns}` - List of column names
* `{state.hook_id.result}` - Array of result records (only if 'into' is not specified)
* `{state.hook_id.result[0].column_name}` - Access specific values from the result
* `{store.variable_name}` - Stored results when using 'into' parameter

## Examples

### Store Query Results for Later Use

Execute a query and store results for use in subsequent hooks:

```yaml
hooks:
  pre:
    - type: query
      connection: source_db
      query: |
        SELECT 
          table_name,
          last_updated,
          row_count
        FROM metadata_table
        WHERE table_name = '{run.stream.name}'
      into: "table_metadata"
      
    - type: log
      message: "Last updated: {store.table_metadata[0].last_updated}, Row count: {store.table_metadata[0].row_count}"
      
```

### Transactional Session Settings

Use `pre_merge` and `post_merge` (formally `pre_sql` and `post_sql`) (available in v1.4.24+) for tight session settings. Below example uses the same connection session/transaction to run custom `SET` queries.

```yaml
source: mssql
target: mssql2022

streams:
  dbo.identity_source:
    object: dbo.identity_target

    hooks:
      pre_merge:
        - type: query
          connection: '{target.name}'
          query: "SET IDENTITY_INSERT {run.object.full_name} ON"

      post_merge:
        - type: query
          connection: '{target.name}'
          query: "SET IDENTITY_INSERT {run.object.full_name} OFF"
```

### Get Configuration Values

Retrieve configuration values from database and use them in other hooks:

```yaml
hooks:
  pre:
    - type: query
      connection: config_db
      query: |
        SELECT 
          config_key,
          config_value
        FROM application_config
        WHERE environment = '{env.ENV_NAME}'
      into: "app_config"
      
    - type: write
      to: "local/temp/runtime_config.json"
      content: "{store.app_config}"
```

### Conditional Processing Based on Query Results

Use query results to control subsequent hook execution:

```yaml
hooks:
  pre:
    - type: query
      connection: target_db
      query: |
        SELECT COUNT(*) as record_count
        FROM {run.object.full_name}
        WHERE DATE(created_at) = CURRENT_DATE
      into: "daily_count"
      
    - type: query
      connection: target_db
      if: "store.daily_count[0].record_count > 1000"
      query: |
        DELETE FROM {run.object.full_name}
        WHERE DATE(created_at) = CURRENT_DATE
        ORDER BY created_at
        LIMIT 500
```

### Update Status Table

Track when a replication starts by updating a status table:

```yaml
hooks:
  pre:
    - type: query
      connection: target_db
      query: |
        INSERT INTO replication_status (
          stream_name, 
          start_time, 
          status
        ) VALUES (
          '{run.stream.name}',
          '{run.start_time}',
          'RUNNING'
        )

    - type: query
      connection: target_db
      query: file://path/to/file.sql
```

### Data Quality Check

Verify data quality after loading and raise an alert if issues are found:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      query: |
        WITH quality_check AS (
          SELECT 
            COUNT(*) as invalid_records
          FROM {run.object.full_name}
          WHERE email IS NULL 
            OR LENGTH(email) < 5 
            OR email NOT LIKE '%@%.%'
        )
        INSERT INTO data_quality_alerts (
          table_name,
          check_time,
          invalid_count,
          total_count
        )
        SELECT 
          '{run.object.full_name}',
          CURRENT_TIMESTAMP,
          invalid_records,
          {run.total_rows}
        FROM quality_check
        WHERE invalid_records > 0
```

### Cleanup Old Data

Clean up old data before loading new data in incremental mode:

```yaml
hooks:
  pre:
    - type: query
      connection: target_db
      query: |
        DELETE FROM {run.object.full_name}
        WHERE created_date < DATEADD(day, -90, CURRENT_DATE)
      on_failure: warn
```

### Update Metadata

Update a metadata table after successful load:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      if: run.status == "success"
      query: |
        MERGE INTO table_metadata t
        USING (
          SELECT 
            '{run.object.full_name}' as table_name,
            {run.total_rows} as total_rows,
            '{run.end_time}' as last_updated
        ) s
        ON t.table_name = s.table_name
        WHEN MATCHED THEN
          UPDATE SET 
            total_rows = s.total_rows,
            last_updated = s.last_updated
        WHEN NOT MATCHED THEN
          INSERT (table_name, total_rows, last_updated)
          VALUES (s.table_name, s.total_rows, s.last_updated)
```

### Validate and Rollback

Check data consistency and rollback if issues are found:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      query: |
        DO $$
        DECLARE
          duplicate_count INT;
        BEGIN
          SELECT COUNT(*) INTO duplicate_count
          FROM (
            SELECT customer_id, COUNT(*)
            FROM {run.object.full_name}
            GROUP BY customer_id
            HAVING COUNT(*) > 1
          ) t;
          
          IF duplicate_count > 0 THEN
            RAISE EXCEPTION 'Found % duplicate customer records', duplicate_count;
          END IF;
        END $$;
      on_failure: abort
```

### Aggregate Statistics

Calculate and store aggregated statistics after data load:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      if: run.status == "success"
      query: |
        INSERT INTO sales_summary (
          date,
          total_sales,
          avg_order_value,
          order_count
        )
        SELECT
          DATE(order_date),
          SUM(amount),
          AVG(amount),
          COUNT(*)
        FROM {run.object.full_name}
        WHERE order_date >= CURRENT_DATE - INTERVAL '1 day'
        GROUP BY DATE(order_date)
        ON CONFLICT (date) DO UPDATE
        SET
          total_sales = EXCLUDED.total_sales,
          avg_order_value = EXCLUDED.avg_order_value,
          order_count = EXCLUDED.order_count
```

### Transaction Support

Execute multiple operations within a transaction with specific isolation level:

```yaml
hooks:
  pre:
    - type: query
      connection: target_db
      transaction: repeatable_read  # Ensures consistent reads within transaction
      query: |
        -- Multiple operations executed as a single transaction
        DELETE FROM staging_table WHERE process_date < CURRENT_DATE - 7;
        
        INSERT INTO staging_table (id, name, process_date)
        SELECT id, name, CURRENT_DATE
        FROM source_table
        WHERE status = 'active';
        
        UPDATE process_log 
        SET last_run = CURRENT_TIMESTAMP, 
            records_processed = (SELECT COUNT(*) FROM staging_table)
        WHERE process_name = 'daily_staging';
```

When using transactions, if any statement fails, all operations within the transaction will be rolled back automatically.


# Replication

Replication hooks allow you to trigger nested replication tasks within your workflow. This is particularly useful for orchestrating complex data pipelines, running dependent replications, or managing multi-step data transformations.

## Configuration

```yaml
- type: replication
  path: "path/to/replication.yaml"  # Required: Path to replication configuration file
  mode: "full-refresh"          # Optional: Override replication mode
  range: "2021-01-01,2022-01-01"    # Optional: Override backfill range for incremental mode
  streams: ["stream1", "stream2"]   # Optional: List of specific streams to run
  env:                              # Optional: Environment variables for the replication
    ENV_VAR1: "value1"
    ENV_VAR2: "value2"
  on_failure: abort                 # Optional: abort/warn/quiet/skip
  id: my_id                         # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                               |
| ----------- | -------- | ------------------------------------------------------------------------- |
| path        | Yes      | Path to the replication configuration file                                |
| mode        | No       | Override the replication mode (full/incremental)                          |
| range       | No       | Override the backfill range for incremental mode                          |
| streams     | No       | List of specific streams to run. If not provided, all streams will be run |
| env         | No       | Map of environment variables to set for the replication                   |
| on\_failure | No       | What to do if the replication fails (abort/warn/quiet/skip)               |

## Output

When the replication hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
path: "path/to/replication.yaml"  # The replication configuration file path
range: "2021-01-01,2022-01-01"  # The backfill range used (if specified)
streams: ["stream1", "stream2"]  # The streams that were processed
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.path}` - The replication configuration file path
* `{state.hook_id.range}` - The backfill range used
* `{state.hook_id.streams}` - The streams that were processed

## Examples

### Run Dependent Replication

Execute a dependent replication after successful processing:

```yaml
hooks:
  post:
    - type: replication
      if: run.status == "success"
      path: "configs/dependent_replication.yaml"
      on_failure: abort
```

### Environment-Specific Replication

Run different replication configurations based on environment:

```yaml
hooks:
  post:
    - type: replication
      path: "configs/{target.environment}/transform.yaml"
      streams: ["{run.stream.name}_processed"]
      env:
        SOURCE_SCHEMA: "{run.stream.schema}"
        TARGET_TABLE: "{run.object.name}_final"
      on_failure: warn
```

### Conditional Stream Selection

Select streams based on runtime conditions:

```yaml
hooks:
  post:
    - type: replication
      path: "configs/conditional_replication.yaml"
      if: run.total_rows > 1000
      streams:
        - "{run.stream.name}_analytics"
        - "{run.stream.name}_archive"
      env:
        PROCESS_DATE: "{timestamp.date}"
        SOURCE_TABLE: "{run.object.full_name}"
```


# Routine

Routine hooks allow you to execute reusable step sequences loaded from external YAML files. This is particularly useful for creating shared templates, standardizing common workflows, and maintaining DRY (Don't Repeat Yourself) principles across your data pipelines.

## Configuration

```yaml
- type: routine
  routine: "my_routine_name"    # Required: Name of the routine to execute
  params:                       # Optional: Parameters to pass to the routine
    param1: "value1"
    param2: "value2"
  env:                         # Optional: Environment variables for all steps
    ENV_VAR1: "value1"
    ENV_VAR2: "value2"
  on_failure: abort            # Optional: abort/warn/quiet/skip
  id: my_id                    # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                        |
| ----------- | -------- | ------------------------------------------------------------------ |
| routine     | Yes      | Name of the routine to execute (must exist in a routine file)      |
| params      | No       | Map of parameters to pass to the routine steps                     |
| env         | No       | Map of environment variables available to all steps in the routine |
| on\_failure | No       | What to do if any step fails (abort/warn/quiet/skip)               |

## Routine File Structure

Routines are loaded from YAML files in the directory specified by the `SLING_ROUTINES_DIR` environment variable. Each file can contain multiple named routines:

```yaml
# Example: /path/to/routines/common_tasks.yaml
routines:
  # required_params: [table_name, connection]
  validate_and_log:
    - type: log
      message: "Starting validation for {params.table_name}"
    - type: query
      connection: "{params.connection}"
      query: "SELECT COUNT(*) as count FROM {params.table_name}"
      id: count_check
    - type: log
      message: "Table has {count_check.result[0].count} rows"

  cleanup_temp_tables:
    - type: log
      message: "Cleaning up temporary tables"
    - type: query
      connection: target_db
      query: "DROP TABLE IF EXISTS temp_staging"
    - type: query
      connection: target_db
      query: "DROP TABLE IF EXISTS temp_backup"
```

### Required Parameters

You can specify required parameters for a routine using a comment above the routine name. This ensures that all necessary parameters are provided when the routine is called:

```yaml
routines:
  # required_params: [table_name, connection, backup_path]
  backup_table:
    - type: log
      message: "Backing up {params.table_name} to {params.backup_path}"
    - type: copy
      from: "{params.connection}/{params.table_name}"
      to: "{params.backup_path}/{params.table_name}"
```

If a required parameter is not provided when calling the routine, the execution will fail with an error message listing the missing parameters:

```yaml
# This will fail because 'backup_path' is missing
- type: routine
  routine: "backup_table"
  params:
    table_name: "customers"
    connection: "prod_db"
    # Missing: backup_path
```

Error message:

```
routine (backup_table) requires params that were not provided: ["backup_path"]
```

**Notes about `required_params`:**

* Must be specified as a YAML comment directly above the routine name
* Use YAML array syntax: `# required_params: [param1, param2, param3]`
* Parameter names should match exactly what you reference in the routine steps
* Validation happens before the routine steps are executed
* Helps prevent runtime errors and provides clear feedback about missing configuration

Now call in a replication or pipeline:

```yaml
# replication
hooks:
  post:
    - type: routine
      routine: "validate_and_log"
      params:
        table_name: "{run.object.name}"
        connection: "{target.name}"
      on_failure: abort

env:
  SLING_ROUTINES_DIR: /path/to/routines
---

# pipeline
steps:
  - type: routine
    routine: cleanup_temp_tables

env:
  SLING_ROUTINES_DIR: /path/to/routines
```

**Important Notes:**

* The `SLING_ROUTINES_DIR` environment variable must be set
* Only `.yaml` and `.yml` files are considered
* Routine names must be unique across all files in the directory
* The directory is scanned recursively, so feel free to create sub-folders

You can set the env var in the `env` section of a replication or pipeline, or as a regular environment variables before running sling:

```bash
# Mac/Unix
export SLING_ROUTINES_DIR=path/to/dir

# Windows Powershell
$env:SLING_ROUTINES_DIR = 'C:/path/to/dir'
```

## Output

When the routine hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success      # Status of the hook execution
routine: "my_routine"  # The routine name that was executed
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.routine}` - The routine name that was executed

### Returning Custom Output from Routines

Routines can return custom output values using the [`store`](/concepts/hooks/store) hook with the `output.` prefix. Any key starting with `output.` will be accessible to the caller after the routine completes:

**In the routine definition:**

```yaml
# File: /path/to/routines/calculations.yaml
routines:
  # required_params: [input_value]
  calculate_metrics:
    - type: log
      message: "Processing input: {params.input_value}"
      id: process_log
    
    - type: query
      connection: "{params.connection}"
      query: "SELECT COUNT(*) as total_count FROM {params.table_name}"
      id: count_query
    
    # Store output values that will be returned to caller
    - type: store
      map:
        output.log_message: '{state.process_log.message}'
        output.row_count: '{state.count_query.result[0].total_count}'
        output.timestamp: '{timestamp.iso}'
```

**Calling the routine and accessing output:**

```yaml
steps:
  - type: routine
    id: metrics_run
    routine: "calculate_metrics"
    params:
      input_value: "test_data"
      connection: "postgres"
      table_name: "public.customers"
  
  # Access the custom output values
  - type: log
    message: "Routine returned: {state.metrics_run.log_message}"
  
  - type: log
    message: "Total rows: {state.metrics_run.row_count}"
  
  - type: check
    check: state.metrics_run.row_count > 0
    failure_message: "Expected rows to be greater than 0"
  
  # Use output values in subsequent steps
  - type: http
    url: "https://api.example.com/metrics"
    method: POST
    payload:
      row_count: "{state.metrics_run.row_count}"
      processed_at: "{state.metrics_run.timestamp}"
```

**Key Points:**

* Use `store` hook with `key: output.<name>` to define return values
* Access via `{state.routine_id.<name>}` after the routine completes
* Any data type can be returned (strings, numbers, objects, arrays)
* Multiple output values can be set using multiple `store` hooks
* Output values are only accessible if the routine completes successfully

Within routine steps, you can access:

* `{params.param_name}` - Parameters passed to the routine
* `{env.ENV_VAR}` - Environment variables set for the routine

## Examples

### Basic Routine Usage

Execute a simple routine without parameters:

```yaml
# In your pipeline/replication hooks:
hooks:
  post:
    - type: routine
      routine: "cleanup_temp_tables"
      on_failure: warn
```

### Routine with Parameters

Pass dynamic parameters to a reusable routine:

```yaml
# In your pipeline/replication hooks:
hooks:
  post:
    - type: routine
      routine: "validate_and_log"
      params:
        table_name: "{run.object.name}"
        connection: "{target.name}"
      on_failure: abort
```

### Routine with Environment Variables

Set environment variables for all steps in the routine:

```yaml
hooks:
  post:
    - type: routine
      routine: "backup_and_archive"
      params:
        source_table: "{run.stream.name}"
      env:
        BACKUP_DATE: "{timestamp.date}"
        ENVIRONMENT: "{target.environment}"
      on_failure: abort
```

### Conditional Routine Execution

Execute a routine based on conditions:

```yaml
hooks:
  post:
    - type: routine
      if: run.total_rows > 10000
      routine: "large_table_optimization"
      params:
        table_name: "{run.object.name}"
        row_count: "{run.total_rows}"
```

### Chaining Routines

Execute multiple routines in sequence:

```yaml
hooks:
  post:
    - type: routine
      id: validation
      routine: "validate_data_quality"
      params:
        table_name: "{run.object.name}"

    - type: routine
      if: validation.status == "success"
      routine: "send_success_notification"
      params:
        table_name: "{run.object.name}"
        validated_at: "{timestamp.iso}"
```

### Routine with Output Values

Create a routine that returns custom values for use in subsequent steps:

**Define the routine:**

```yaml
# File: /path/to/routines/data_quality.yaml
routines:
  # required_params: [connection, table_name]
  check_data_quality:
    - type: query
      connection: "{params.connection}"
      query: "SELECT COUNT(*) as total_rows FROM {params.table_name}"
      id: count_check
    
    - type: query
      connection: "{params.connection}"
      query: "SELECT COUNT(*) as null_rows FROM {params.table_name} WHERE primary_key IS NULL"
      id: null_check
    
    # Calculate quality score
    - type: store
      key: output.total_rows
      value: '{state.count_check.result[0].total_rows}'
    
    - type: store
      key: output.null_rows
      value: '{state.null_check.result[0].null_rows}'
    
    - type: store
      key: output.quality_passed
      value: '{state.null_check.result[0].null_rows == 0}'
    
    - type: log
      message: "Quality check: {state.output.total_rows} total rows, {state.output.null_rows} null rows"
```

**Use the routine and its outputs:**

```yaml
steps:
  - type: routine
    id: quality_check
    routine: "check_data_quality"
    params:
      connection: "prod_db"
      table_name: "customers"
  
  # Use the output values in conditional logic
  - type: log
    if: quality_check.quality_passed == true
    message: "✓ Quality check passed! Processed {quality_check.total_rows} rows"
  
  - type: log
    if: quality_check.quality_passed == false
    message: "✗ Quality check failed! Found {quality_check.null_rows} rows with null primary keys"
  
  # Fail the pipeline if quality check didn't pass
  - type: check
    check: state.quality_check.quality_passed == true
    failure_message: "Data quality check failed: {state.quality_check.null_rows} rows have null primary keys"
  
  # Send notification with quality metrics
  - type: http
    url: "https://api.slack.com/webhooks/your-webhook"
    method: POST
    payload:
      text: "Data quality report for customers table"
      blocks:
        - type: section
          text:
            type: mrkdwn
            text: |
              *Total Rows:* {state.quality_check.total_rows}
              *Null Rows:* {state.quality_check.null_rows}
              *Status:* {state.quality_check.quality_passed}
```

### Complex Routine Example

Create a comprehensive routine file for data processing:

```yaml
# File: /path/to/routines/data_processing.yaml
routines:
  # required_params: [table_name, target_connection, replication_config, stream_name]
  full_table_refresh:
    - type: log
      message: "Starting full refresh for {params.table_name}"

    - type: query
      connection: "{params.target_connection}"
      query: "TRUNCATE TABLE {params.table_name}"
      id: truncate_step

    - type: replication
      path: "{params.replication_config}"
      streams: ["{params.stream_name}"]
      mode: "full-refresh"
      id: load_step

    - type: query
      connection: "{params.target_connection}"
      query: |
        UPDATE metadata.refresh_log
        SET last_refresh = CURRENT_TIMESTAMP,
            row_count = (SELECT COUNT(*) FROM {params.table_name})
        WHERE table_name = '{params.table_name}'
      id: update_metadata

    - type: log
      message: "Refresh complete: {update_metadata.result[0].row_count} rows loaded"

  # required_params: [target_connection, replication_config, stream_name, table_name, primary_key]
  incremental_with_dedup:
    - type: log
      message: "Starting incremental load with deduplication"

    - type: query
      connection: "{params.target_connection}"
      query: "CREATE TABLE IF NOT EXISTS {params.table_name}_staging LIKE {params.table_name}"

    - type: replication
      path: "{params.replication_config}"
      streams: ["{params.stream_name}"]
      mode: "incremental"
      id: incremental_load

    - type: query
      connection: "{params.target_connection}"
      query: |
        DELETE FROM {params.table_name}
        WHERE {params.primary_key} IN (
          SELECT {params.primary_key} FROM {params.table_name}_staging
        )

    - type: query
      connection: "{params.target_connection}"
      query: |
        INSERT INTO {params.table_name}
        SELECT * FROM {params.table_name}_staging

    - type: query
      connection: "{params.target_connection}"
      query: "DROP TABLE {params.table_name}_staging"
```

Usage:

```yaml
hooks:
  post:
    - type: routine
      routine: "full_table_refresh"
      params:
        table_name: "customers"
        target_connection: "{env.PROD_DB}"
        replication_config: "configs/customers.yaml"
        stream_name: "public.customers"
```


# Store

Store hooks allow you to store and manage values in memory during execution, creating a shared key-value store that can be used by subsequent hooks and replications. This hook is particularly useful for storing processing results, creating reusable variables, or maintaining state across your workflow.

The store hook supports two storage targets:

* **Replication Store** (default): Values stored with `store.*` or plain keys persist only within the current replication execution
* **Environment Variables**: Values stored with `env.*` prefix become actual environment variables, available to all subsequent steps, replications, and even API spec rendering (authentication blocks, dynamic endpoints, etc.)

## Configuration

**Single Key/Value:**

```yaml
- type: store
  key: my_key             # Required: Key to store the value under
  value: some value       # Required (unless deleting): Value to store
  delete: false           # Optional: Set to true to delete the key
```

**Multiple Key/Values:**

```yaml
- type: store
  map:                    # Set multiple key/value pairs at once
    key1: value1
    key2: value2
    env.API_KEY: "abc123"
  delete: false           # Optional: Set to true to delete all keys in the map (and unset env var)
```

## Properties

| Property    | Required | Description                                                                                                                                                                                  |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| key         | Yes\*    | The key name to store the value under. Use `env.KEY_NAME` prefix to set environment variables, or `store.key_name` (or just `key_name`) for replication store. \*Required unless using `map` |
| value       | Yes\*    | Value to store at the specified key. For environment variables, the value is automatically converted to a string. \*Required unless using `map` or deleting                                  |
| map         | No       | A map of key/value pairs to set multiple values at once. Use this instead of `key` and `value` when setting multiple store values. Supports both `env.*` and `store.*` keys                  |
| delete      | No       | When set to `true`, deletes the key(s) and their value(s) instead of storing. Works with both `key`/`value` and `map` approaches                                                             |
| on\_failure | No       | What to do if the operation fails (abort/warn/quiet/skip)                                                                                                                                    |

## Output

When the store hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success     # Status of the hook execution
operation: "set"    # The operation that was performed (set/delete)
path: "my_key"      # The key that was manipulated
value: {...}        # The value that was set (only for 'set' operation)
```

You can access these values in subsequent hooks using the following syntax:

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.operation}` - The operation performed (set/delete)
* `{state.hook_id.path}` - The key that was manipulated
* `{state.hook_id.value}` - The value that was set (only for 'set' operation)

## Accessing Stored Values

### Replication Store Values

To access values stored in the replication store, use the `store` namespace:

```yaml
{store.my_key}  # Access a value stored with key "my_key"
```

For example, if you stored a complex object, you can access nested properties:

```yaml
{store.user.name}        # Access the "name" property of an object stored under "user"
{store.metrics.count}    # Access the "count" property of an object stored under "metrics"
```

### Environment Variable Values

To access values stored as environment variables, use the `env` namespace:

```yaml
{env.MY_API_KEY}     # Access environment variable MY_API_KEY
{env.DATABASE_URL}   # Access environment variable DATABASE_URL
```

Environment variables set via `env.*` prefix are:

* Available across all subsequent pipeline steps and replications
* Accessible in API specs during rendering (authentication, dynamic endpoints, etc.)
* Accessible in connection configurations
* Visible to child processes

## Storage Target Comparison

| Feature               | Replication Store (`store.*`)           | Environment Variables (`env.*`)     |
| --------------------- | --------------------------------------- | ----------------------------------- |
| Scope                 | Current replication only                | All subsequent steps/replications   |
| API Spec Rendering    | ❌ Not available during spec compilation | ✅ Available before spec compilation |
| Complex Objects       | ✅ Supports nested objects/arrays        | ❌ String values only                |
| Use in Authentication | ❌ Cannot use in auth blocks             | ✅ Can use in auth/dynamic blocks    |
| Performance           | Faster (in-memory)                      | Slightly slower (OS env vars)       |
| Best For              | Temporary data, complex objects         | Configuration, credentials, tokens  |

## Examples

### Setting a Simple Value

Store a simple value for later use:

```yaml
hooks:
  pre:
    - type: store
      key: processing_mode
      value: "batch"
    
    # Later, use the stored value
    - type: log
      message: "Processing in {store.processing_mode} mode"
```

### Storing Complex Values

Store an object with multiple properties:

```yaml
hooks:
  pre:
    - type: store
      key: database_config
      value:
        max_connections: 10
        timeout: 30
        retry: true
    
    # Later, access the stored values
    - type: log
      message: "Max connections: {store.database_config.max_connections}"
```

### Using Variables in Values

Set values using variables from the runtime state:

```yaml
hooks:
  post:
    - type: store
      key: execution_summary
      value:
        stream: "{run.stream.name}"
        status: "{run.status}"
        processed_rows: "{run.total_rows}"
        timestamp: "{timestamp.datetime}"
    
    # Later, use the stored summary
    - type: log
      message: "Execution summary - Stream: {store.execution_summary.stream}, Status: {store.execution_summary.status}"
```

### Deleting Stored Values

Remove values that are no longer needed:

```yaml
hooks:
  post:
    - type: store
      key: temp_data
      delete: true
```

### Conditional Storage

Store values based on conditions:

```yaml
hooks:
  post:
    - type: store
      if: "run.status == 'success' && run.total_rows > 0"
      key: validation_result
      value: "passed"
    
    - type: store
      if: "run.status == 'error' || run.total_rows == 0"
      key: validation_result
      value: "failed"
    
    # Later, use the conditional value
    - type: log
      message: "Validation result: {store.validation_result}"
```

### Storing Results from Other Hooks

Store and access results from other hooks:

```yaml
hooks:
  pre:
    - type: query
      id: count_query
      connection: postgres
      query: "SELECT COUNT(*) as row_count FROM my_table"
    
    - type: store
      key: initial_count
      value: "{state.count_query.result[0].row_count}"
    
    # Later, use the stored count
    - type: log
      message: "Initial count: {store.initial_count}"
```

### Using Stored Values in Subsequent Hooks

Show how stored values can be used in various hooks:

```yaml
hooks:
  pre:
    # Store a configuration
    - type: store
      key: config
      value:
        table_name: "customers"
        batch_size: 1000
  
  post:
    # Use in query
    - type: query
      connection: postgres
      query: "SELECT COUNT(*) FROM {store.config.table_name}"
    
    # Use in HTTP hook
    - type: http
      url: "https://api.example.com/metrics"
      method: POST
      payload: |
        {
          "table": "{store.config.table_name}",
          "batch_size": {store.config.batch_size},
          "rows_processed": {run.total_rows}
        }
    
    # Use in log message
    - type: log
      message: "Processed {run.total_rows} rows from {store.config.table_name} in batches of {store.config.batch_size}"
```

### Building Dynamic Values

Combine stored values to build more complex structures:

```yaml
hooks:
  pre:
    - type: store
      key: user
      value:
        name: "John Doe"
        role: "admin"
    
    - type: store
      key: settings
      value:
        theme: "dark"
        language: "en"
    
    # Later, create a combined report using stored values
    - type: log
      message: |
        User: {store.user.name}
        Role: {store.user.role}
        Theme: {store.settings.theme}
        Language: {store.settings.language}
```

### Temporary Storage for Processing

Use store for intermediate processing:

```yaml
hooks:
  pre:
    # Extract date components
    - type: store
      key: date_parts
      value:
        year: "{timestamp.YYYY}"
        month: "{timestamp.MM}"
        day: "{timestamp.DD}"

    # Build a formatted path using the stored parts
    - type: store
      key: output_path
      value: "reports/{store.date_parts.year}/{store.date_parts.month}/daily_report_{store.date_parts.day}.csv"

    # Use the generated path
    - type: log
      message: "Will save report to: {store.output_path}"
```

## Setting Environment Variables

Use the `env.*` prefix to set environment variables that persist across pipeline steps and are available during API spec rendering.

### Basic Environment Variable

Set a simple environment variable:

```yaml
steps:
  - type: store
    key: env.API_KEY
    value: "sk_live_abc123xyz"

  - type: store
    key: env.ENVIRONMENT
    value: "production"

  # These env vars are now available in subsequent steps
  - type: log
    message: "Running in {env.ENVIRONMENT} environment"
```

### Dynamic API Authentication with Environment Variables

Set environment variables that are used in API spec authentication blocks:

**Pipeline Configuration:**

```yaml
steps:
  # Step 1: Query database to get API credentials
  - type: query
    connection: MY_CONFIG_DB
    query: |
      SELECT
        api_username,
        api_password,
        api_endpoint
      FROM api_credentials
      WHERE environment = 'production'
      LIMIT 1
    into: api_config

  # Step 2: Set credentials as environment variables
  - type: store
    key: env.API_USERNAME
    value: "{store.api_config[0].api_username}"

  - type: store
    key: env.API_PASSWORD
    value: "{store.api_config[0].api_password}"

  - type: store
    key: env.API_ENDPOINT
    value: "{store.api_config[0].api_endpoint}"

  # Step 3: Log configuration (without sensitive data)
  - type: log
    message: "API configured for endpoint: {env.API_ENDPOINT}"

  # Step 4: Run replication that uses these env vars in API spec
  - type: replication
    path: /path/to/api_replication.yaml
```

**API Spec (using environment variables):**

```yaml
name: "My API"

# Environment variables are available during spec rendering
authentication:
  type: basic
  username: "{env.API_USERNAME}"
  password: "{env.API_PASSWORD}"

defaults:
  state:
    base_url: "{env.API_ENDPOINT}"
  request:
    headers:
      X-Environment: "{env.ENVIRONMENT}"

endpoints:
  users:
    request:
      url: "{state.base_url}/users"
```

**Key Advantage:** Environment variables set via `env.*` hooks are available **before** API specs are compiled/rendered, making them usable in authentication blocks and dynamic endpoint definitions where `store.*` variables cannot be used.

### Setting Multiple Values at Once

Use the `map` property to set multiple store or environment variables in a single step:

```yaml
steps:
  # Set multiple environment variables at once
  - type: store
    map:
      env.API_USERNAME: "api_user_prod"
      env.API_PASSWORD: "secure_pass_123"
      env.API_ENDPOINT: "https://api.example.com"
      env.ENVIRONMENT: "production"

  # Set multiple store values at once
  - type: store
    map:
      retry_count: 3
      timeout: 30
      batch_size: 1000

  # Mix environment variables and store values
  - type: store
    map:
      env.DEPLOYMENT: "prod" # set env var
      cache_enabled: true
      max_connections: 10
      database_name: "production_db"

  # Use the stored values
  - type: log
    message: |
      Configuration:
      - Environment: {env.ENVIRONMENT}
      - Endpoint: {env.API_ENDPOINT}
      - Batch Size: {store.config.batch_size}
      - Max Connections: {store.max_connections}
```

### Dynamic Map from Query Results

Build a map dynamically from query results:

```yaml
steps:
  # Query database for configuration
  - type: query
    connection: MY_CONFIG_DB
    query: |
      SELECT
        'env.API_KEY' as key,
        api_key as value
      FROM api_config
      WHERE environment = 'production'
      UNION ALL
      SELECT
        'env.API_ENDPOINT' as key,
        api_endpoint as value
      FROM api_config
      WHERE environment = 'production'
    into: config_rows

  # Set all config values at once using map
  - type: store
    map:
      env.API_KEY: "{store.config_rows[0].value}"
      env.API_ENDPOINT: "{store.config_rows[1].value}"
```

### Deleting Environment Variables

Remove environment variables when no longer needed:

```yaml
steps:
  - type: store
    key: env.TEMP_TOKEN
    value: "temporary_value"

  # Use the token...

  - type: store
    key: env.TEMP_TOKEN
    delete: true  # Removes the environment variable
```

### Deleting Multiple Variables

Remove multiple keys at once:

```yaml
steps:
  # Set temporary variables
  - type: store
    map:
      env.TEMP_TOKEN: "token_123"
      env.TEMP_SESSION: "session_456"
      temp_data: "some_data"

  # Use the temporary variables...

  # Delete all temporary variables at once
  - type: store
    map:
      env.TEMP_TOKEN: ""
      env.TEMP_SESSION: ""
      temp_data: ""
    delete: true
```


# Read

Read hooks allow you to read the contents of files from any file-based storage connection. This is particularly useful for reading configuration files, processing text content, or incorporating file data into your workflow.

## Configuration

```yaml
- type: read
  from: "connection/path/to/file.txt"   # Required: Source Location
  into: "my_content"      # Optional: Store content in variable
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id      # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                                   |
| ----------- | -------- | ------------------------------------------------------------------------------------------------------------- |
| from        | Yes      | The source [location](/sling-cli/environment#location-string) string. Contains connection name and file path. |
| into        | No       | Variable name to store the file content. If not specified, content is included in hook output.                |
| on\_failure | No       | What to do if the read fails (abort/warn/quiet/skip)                                                          |

## Output

When the read hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
source_url: "s3://bucket/path/to/file.txt"  # The normalized URI of the source file
bytes_read: 1024  # Number of bytes read
content: "file content here"  # File content (only if 'into' is not specified)
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.source_url}` - The normalized URI of the source file
* `{state.hook_id.bytes_read}` - Number of bytes read
* `{state.hook_id.content}` - File content (only if 'into' is not specified)
* `{store.variable_name}` - Stored content when using 'into' parameter

## Examples

### Read Configuration File

Read a configuration file and use its content in subsequent hooks:

```yaml
hooks:
  pre:
    - type: read
      from: "s3/config/database-config.json"
      into: "db_config"
      
    - type: log
      message: "Database config: {store.db_config}"
```

### Read Query Template

Read a SQL query template from a file and use it in a query hook:

```yaml
hooks:
  pre:
    - type: read
      from: "local/queries/cleanup_template.sql"
      into: "cleanup_query"
      
    - type: query
      connection: target_db
      query: "{store.cleanup_query}"
```

### Process Text File Content

Read a text file and process its content:

```yaml
hooks:
  post:
    - type: read
      from: "gcs/reports/summary.txt"
      id: read_summary
      
    - type: log
      message: "Report summary: {state.read_summary.content}"
      
    - type: http
      url: "https://webhook.example.com/notify"
      method: POST
      payload: |
        {
          "message": "Data processing complete",
          "summary": "{state.read_summary.content}",
          "bytes_processed": {state.read_summary.bytes_read}
        }
```


# Write

Write hooks allow you to write content to files in any file-based storage connection. This is particularly useful for creating reports, saving processed data, generating configuration files, or writing logs.

## Configuration

```yaml
- type: write
  to: "connection/path/to/file.txt"     # Required: Destination Location
  content: "text content to write"      # Required: Content to write
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id      # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.
```

## Properties

| Property    | Required | Description                                                                                                                             |
| ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| to          | Yes      | The destination [location](/sling-cli/environment#location-string) string. Contains connection name and file path.                      |
| content     | Yes      | The content to write to the file. Supports variable substitution. Can also use `file://path/to/file` to read content from a local file. |
| on\_failure | No       | What to do if the write fails (abort/warn/quiet/skip)                                                                                   |

## Output

When the write hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

```yaml
status: success  # Status of the hook execution
target_url: "s3://bucket/path/to/file.txt"  # The normalized URI of the target file
bytes_written: 1024  # Number of bytes written
```

You can access these values in subsequent hooks using the following syntax (`jmespath`):

* `{state.hook_id.status}` - Status of the hook execution
* `{state.hook_id.target_url}` - The normalized URI of the target file
* `{state.hook_id.bytes_written}` - Number of bytes written

## Examples

### Generate Data Processing Report

Create a summary report after data processing:

```yaml
hooks:
  post:
    - type: write
      to: "s3/reports/processing_summary_{timestamp.YYYY-MM-DD}.txt"
      content: |
        Data Processing Summary
        ======================
        
        Stream: {run.stream.name}
        Start Time: {run.start_time}
        End Time: {run.end_time}
        Total Rows: {run.total_rows}
        Status: {run.status}
        
        Target: {target.connection}/{target.object}
        Total Bytes Written: {run.total_bytes}
        
        Generated on: {timestamp.YYYY-MM-DD HH:mm:ss}
```

### Create JSON Configuration File

Generate a configuration file with runtime data:

```yaml
hooks:
  pre:
    - type: write
      to: "local/config/runtime_config.json"
      content: |
        {
          "processing_date": "{timestamp.YYYY-MM-DD}",
          "source_connection": "{source.connection}",
          "target_connection": "{target.connection}",
          "stream_name": "{run.stream.name}",
          "environment": "{env.ENV_NAME}",
          "user": "{env.USER}"
        }
```

### Write Query Results to File

Save query results as a formatted report:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      query: |
        SELECT 
          COUNT(*) as total_records,
          MAX(created_at) as latest_record,
          MIN(created_at) as oldest_record
        FROM {run.object.full_name}
      id: stats_query
      
    - type: write
      to: "gcs/reports/table_stats_{run.stream.name}_{timestamp.YYYY-MM-DD}.txt"
      content: |
        Table Statistics Report
        =====================
        
        Table: {run.object.full_name}
        Total Records: {state.stats_query.result[0].total_records}
        Latest Record: {state.stats_query.result[0].latest_record}
        Oldest Record: {state.stats_query.result[0].oldest_record}
        
        Report generated: {timestamp.YYYY-MM-DD HH:mm:ss}
```

### Create Error Log

Write error information to a log file when processing fails:

```yaml
hooks:
  post:
    - type: write
      if: run.status == "error"
      to: "local/logs/error_log_{timestamp.YYYY-MM-DD}.txt"
      content: |
        ERROR LOG ENTRY
        ===============
        
        Timestamp: {timestamp.YYYY-MM-DD HH:mm:ss}
        Stream: {run.stream.name}
        Source: {source.connection}/{source.object}
        Target: {target.connection}/{target.object}
        Error: {run.error}
        
        Environment: {env.ENV_NAME}
        User: {env.USER}
        
        ---
      on_failure: warn
```

### Generate CSV Report

Create a CSV file with processed data statistics:

```yaml
hooks:
  post:
    - type: write
      to: "s3/reports/daily_stats_{timestamp.YYYY-MM-DD}.csv"
      content: |
        date,stream_name,source_connection,target_connection,rows_processed,bytes_processed,status,duration_seconds
        {timestamp.YYYY-MM-DD},{run.stream.name},{source.connection},{target.connection},{run.total_rows},{run.total_bytes},{run.status},{run.duration}
```

### Write Content from Local File

Write content from a local file to a remote location:

```yaml
hooks:
  post:
    - type: write
      to: "s3/reports/daily_report_{timestamp.YYYY-MM-DD}.html"
      content: "file://templates/report_template.html"
```

### Write Processed Content

Process stored content and write it to a new file:

```yaml
hooks:
  pre:
    - type: read
      from: "s3/templates/email_template.html"
      into: "template"
      
  post:
    - type: write
      to: "local/output/personalized_email_{timestamp.YYYY-MM-DD-HH-mm}.html"
      content: |
        {store.template | replace('{{USER_NAME}}', '{env.USER_NAME}') | replace('{{DATE}}', '{timestamp.YYYY-MM-DD}') | replace('{{ROWS_PROCESSED}}', '{run.total_rows}')}
```

### Create Backup Metadata

Write metadata about the backup operation:

```yaml
hooks:
  end:
    - type: write
      to: "s3/backups/metadata/backup_{timestamp.YYYY-MM-DD-HH-mm}.json"
      content: |
        {
          "backup_timestamp": "{timestamp.YYYY-MM-DD HH:mm:ss}",
          "source": {
            "connection": "{source.connection}",
            "object": "{source.object}",
            "total_rows": {run.total_rows}
          },
          "target": {
            "connection": "{target.connection}",
            "object": "{target.object}",
            "bytes_written": {run.total_bytes}
          },
          "status": "{run.status}",
          "duration_seconds": {run.duration},
          "environment": "{env.ENV_NAME}"
        }
```

### Write Multi-line SQL Script

Generate a SQL script based on runtime data:

```yaml
hooks:
  post:
    - type: write
      to: "local/sql/cleanup_{run.stream.name}_{timestamp.YYYY-MM-DD}.sql"
      content: |
        -- Cleanup script for {run.stream.name}
        -- Generated on {timestamp.YYYY-MM-DD HH:mm:ss}
        
        BEGIN;
        
        -- Archive old data
        CREATE TABLE {run.object.full_name}_archive_{timestamp.YYYY_MM_DD} AS
        SELECT * FROM {run.object.full_name}
        WHERE created_at < CURRENT_DATE - INTERVAL '90 days';
        
        -- Delete old data
        DELETE FROM {run.object.full_name}
        WHERE created_at < CURRENT_DATE - INTERVAL '90 days';
        
        -- Update statistics
        ANALYZE {run.object.full_name};
        
        COMMIT;
        
        -- Summary: Processed {run.total_rows} rows
```

### Conditional Writing

Write different content based on conditions:

```yaml
hooks:
  post:
    - type: write
      if: run.total_rows > 1000
      to: "s3/alerts/high_volume_{timestamp.YYYY-MM-DD}.txt"
      content: |
        HIGH VOLUME ALERT
        =================
        
        Stream: {run.stream.name}
        Rows Processed: {run.total_rows}
        Threshold: 1000
        Time: {timestamp.YYYY-MM-DD HH:mm:ss}
        
        This requires immediate attention.
      
    - type: write
      if: run.total_rows <= 1000
      to: "s3/logs/normal_volume_{timestamp.YYYY-MM-DD}.txt"
      content: |
        Normal processing completed for {run.stream.name}: {run.total_rows} rows processed.
```


# Pipelines

Use Pipelines to orchestrate multiple steps in sequence

A Pipeline in Sling allows you to execute multiple steps in sequence. Each step can be a different type of operation, enabling you to create complex workflows by chaining together various actions like running replications, executing queries, making HTTP requests, and more.

{% hint style="success" %}
Sling Pipelines integrate seamlessly with the [Sling VSCode Extension](/sling-cli/vscode). The extension provides schema validation, auto-completion, hover documentation, and diagnostics for your pipeline configurations, making it easier to author and debug complex workflows.
{% endhint %}

## Pipeline Configuration

A pipeline is defined in YAML format with a `steps` key at the root level containing an array of steps. Each step supports the same types and configurations as [Hooks](/concepts/hooks).

```yaml
steps:
  - type: log
    message: "Starting pipeline execution"

  - type: replication
    path: path/to/replication.yaml
    id: my_replication

  - type: query
    if: state.my_replication.status == "success"
    connection: my_database
    query: "UPDATE status SET completed = true"

env:
  MY_KEY: VALUE
```

## Available Step Types

Pipelines support all the same types as Hooks:

| Step Type   | Description                                                | Documentation                                   |
| ----------- | ---------------------------------------------------------- | ----------------------------------------------- |
| Check       | Validate conditions and control flow                       | [Check Step](/concepts/hooks/check)             |
| Command     | Run any command/process                                    | [Command Step](/concepts/hooks/command)         |
| Copy        | Transfer files between local or remote storage connections | [Copy Step](/concepts/hooks/copy)               |
| Delete      | Remove files from local or remote storage connections      | [Delete Step](/concepts/hooks/delete)           |
| Group       | Run sequences of steps or loop over values                 | [Group Step](/concepts/hooks/group)             |
| HTTP        | Make HTTP requests to external services                    | [HTTP Step](/concepts/hooks/http)               |
| Inspect     | Inspect a file or folder                                   | [Inspect Step](/concepts/hooks/inspect)         |
| List        | List files in folder                                       | [List Step](/concepts/hooks/list)               |
| Log         | Output custom messages and create audit trails             | [Log Step](/concepts/hooks/log)                 |
| Query       | Execute SQL queries against any defined connection         | [Query Step](/concepts/hooks/query)             |
| Read        | Read contents of files from storage connections            | [Read Step](/concepts/hooks/read)               |
| Replication | Run a Replication                                          | [Replication Step](/concepts/hooks/replication) |
| Routine     | Execute reusable step sequences from external files        | [Routine Step](/concepts/hooks/routine)         |
| Store       | Store values for later in-process access                   | [Store Step](/concepts/hooks/store)             |
| Write       | Write content to files in storage connections              | [Write Step](/concepts/hooks/write)             |

## Common Step Properties

Each step shares the same common properties as hooks:

| Property     | Description                                                                       | Required                 |
| ------------ | --------------------------------------------------------------------------------- | ------------------------ |
| `type`       | The type of step (`query`/ `http`/ `check`/ `copy` / `delete`/ `log` / `inspect`) | Yes                      |
| `if`         | Optional condition to determine if the step should execute                        | No                       |
| `id`         | A specific identifier to refer to the step output data                            | No                       |
| `on_failure` | What to do if the step fails (see [On Failure Behaviors](#on-failure-behaviors))  | No (defaults to `abort`) |

### On Failure Behaviors

The `on_failure` property controls what happens **when a step fails**. It only takes effect if the step errors — on success it has no impact. The following values are supported:

| Value             | Behavior                                                                                                                                                                      |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `abort` (default) | Stops the pipeline immediately and fails the run with the error. This is the behavior when `on_failure` is not set.                                                           |
| `warn`            | Logs a warning with the error message and **continues to the next step**. The run is not failed.                                                                              |
| `quiet`           | Silently swallows the error (no log output) and **continues to the next step**. The run is not failed.                                                                        |
| `break`           | Stops the current sequence of steps gracefully (without failing the run) and moves on. Useful to stop early without marking the pipeline as failed.                           |
| `retry`           | Retries the failed step once. If it fails again, the error is raised.                                                                                                         |
| `defer`           | Only meaningful inside a [group](/concepts/hooks/group): records the error but lets the remaining steps in the group finish, then surfaces the error at the end of the group. |

## Variables Available

Pipeline steps have access to the runtime state which includes various variables that can be referenced using curly braces `{variable}`. The available variables include:

* `runtime_state` - Contains all state variables available
* `env.*` - All variables defined in the `env`
* `timestamp.*` - Various timestamp parts information
* `steps.*` - Output data from previous steps (referenced by their `id`)
* `execution.*` - Execution-level context, including:
  * `execution.cli_args.*` - The flags passed on the command line when running the pipeline (see [Reading CLI Flags](#reading-cli-flags) below).

You can view all available variables by using a log step:

```yaml
steps:
  - type: log
    message: '{runtime_state}'
```

### Reading CLI Flags

Starting in v1.5.22, the `execution.cli_args` map exposes the flags passed to `sling run`, so a pipeline can adapt its behavior based on command-line input. The key is the flag's long name with hyphens replaced by underscores (e.g. `--src-conn` becomes `execution.cli_args.src_conn`).

```yaml
# sling run -p pipeline.yaml --streams tag:transactions --mode full-refresh --limit 5
steps:
  - type: log
    message: 'requested streams => {execution.cli_args.streams}'

  - type: check
    check: execution.cli_args.streams[0] == "tag:transactions"
    message: 'expected the transactions stream selector'
```

#### Available Keys

Only flags that are actually passed appear in `cli_args` (plus the boolean flags, which always appear defaulted to `0`). The valid keys are:

`replication`, `pipeline`, `directory`, `streams`, `select`, `primary_key`, `update_key`, `mode`, `limit`, `offset`, `range`, `where`, `src_conn`, `src_stream`, `src_options`, `tgt_conn`, `tgt_object`, `tgt_options`, `columns`, `transforms`, `env`, `cdc_options`, `debug`, `trace`, `stdout`, `examples`

The comma-separated flags `streams`, `primary_key` and `select` are always **arrays**, even when a single value is passed — index them to read individual entries (e.g. `execution.cli_args.streams[0]`).

#### Missing Keys

If a flag was **not** passed, its key is absent from `cli_args`. Referencing a missing key directly in a `check`/`if` expression raises an error (`object has no member "..."`), so use `jmespath` to read a possibly-missing key safely — it returns empty instead of erroring:

```yaml
  # safe: is_empty() is true when --where was not passed
  - check: is_empty(jmespath(execution.cli_args, "where"))

  # or test presence with keys()
  - check: contains(keys(execution.cli_args), "streams")
```

In a `log` message (or any rendered template), a missing key simply renders as the un-substituted `{execution.cli_args.where}` literal rather than erroring.

## Example Pipeline

Here's a complete example that demonstrates various pipeline capabilities:

```yaml
env:
  DATABASE: production
  NOTIFY_URL: https://api.example.com/webhook

steps:
  # Log the start of execution
  - type: log
    message: "Starting pipeline execution"

  # Run a replication
  - type: replication
    path: replications/daily_sync.yaml
    id: daily_sync
    on_failure: warn

  # Validate the results
  - type: check
    check: state.daily_sync.status == "success"
    message: "Daily sync failed"
    on_failure: abort

  # Update status in database
  - type: query
    connection: "{env.DATABASE}"
    query: |
      UPDATE pipeline_status 
      SET last_run = current_timestamp
      WHERE name = 'daily_sync'
    on_failure: warn

  # Send notification
  - type: http
    url: "{env.NOTIFY_URL}"
    method: POST
    payload: |
      {
        "pipeline": "daily_sync",
        "status": "success",
      }

  # Log completion
  - type: log
    message: "Pipeline completed successfully"
```

## Best Practices

1. **Error Handling**: Use appropriate `on_failure` behaviors for each step
2. **Validation**: Include check steps to validate critical conditions
3. **Logging**: Add log steps for better observability
4. **Modularity**: Break down complex operations into multiple steps
5. **Conditions**: Use `if` conditions to control step execution
6. **Variables**: Leverage environment variables and runtime state for dynamic configuration
7. **Identifiers**: Use meaningful `id`s for steps when you need to reference their output later

## Running a Pipeline

You can run a pipeline using the Sling CLI:

```bash
sling run --pipeline path/to/pipeline.yaml
```


# Examples

This page provides practical examples of Sling Pipelines, demonstrating how to chain multiple steps for complex data workflows. These examples build upon the concepts from the [Pipeline](/concepts/pipeline), [Hooks](/concepts/hooks) and [Functions](/concepts/functions) documentation.

Each example includes:

* A brief description
* The YAML configuration
* Key concepts demonstrated

## Basic Pipeline with Logging and Replication

This simple pipeline logs the start, runs a replication, and logs the completion with runtime state.

```yaml
steps:
  - type: log
    message: 'Starting pipeline execution on {date_format(now(), "%Y-%m-%d %H:%M:%S")}. Runtime state: {runtime_state}'

  - type: replication
    path: path/to/your/replication.yaml
    id: main_replication

  - type: log
    message: 'Pipeline completed. Final state: {runtime_state}'
    level: info

  - type: command
    command: 'echo "Replication status: {upper(state.main_replication.status)}"'
    print: true
```

**Key Concepts:**

* Basic sequencing of steps
* Using [`log`](/concepts/hooks/log) for monitoring with `date_format` and `now` functions
* Running a [`replication`](/concepts/hooks/replication) as a step
* Accessing state from previous steps
* Executing system commands with [`command`](/concepts/hooks/command) using `upper` function

## File Processing Pipeline with Looping

This pipeline lists files from S3, copies them to Azure, and logs the process using a group for looping.

```yaml
steps:
  - type: list
    id: s3_files
    location: aws_s3/your-bucket/files/
    recursive: true
    only: files

  - type: group
    loop: state.s3_files.result
    steps:
      - type: log
        message: 'Processing file {loop.index + 1}: {loop.value.name} ({loop.value.size} bytes)'

      - type: copy
        from: '{loop.value.location}'
        to: azure_storage/processed/{coalesce(loop.value.name, "unnamed_file")}.processed
        id: file_copy

      - type: log
        if: state.file_copy.bytes_written > 0
        message: 'Successfully copied {state.file_copy.bytes_written} bytes'
        level: info

  - type: log
    message: 'Processed {length(state.s3_files.result)} files'
```

**Key Concepts:**

* Listing files with [`list`](/concepts/hooks/list)
* Looping with [`group`](/concepts/hooks/group) and `loop`
* Conditional logging with [`log`](/concepts/hooks/log)
* File transfer using [`copy`](/concepts/hooks/copy)
* Accessing loop variables (`loop.index`, `loop.value`)
* Using functions like `upper`, `split_part`, `coalesce`, and `length` in expressions

## Data Quality Pipeline

This pipeline runs a replication, performs quality checks via queries, and notifies if issues are found.

```yaml
steps:
  - type: replication
    path: replications/data_sync.yaml
    id: data_sync

  - type: query
    connection: target_db
    query: |
      SELECT COUNT(*) as invalid_count
      FROM target_schema.my_table
      WHERE some_column IS NULL
    id: quality_check
    into: qc_results

  - type: check
    check: store.qc_results[0].invalid_count == 0
    failure_message: 'Found {store.qc_results[0].invalid_count} invalid records'
    on_failure: warn

  - type: http
    if: store.qc_results[0].invalid_count > 0
    url: https://alerts.example.com/notify
    method: POST
    payload: |
      {
        "issue": "Data quality failure",
        "details": "Invalid records: {store.qc_results[0].invalid_count}",
        "table": "target_schema.my_table",
        "checked_at": "{date_format(now(), "%Y-%m-%d")}"
      }
```

**Key Concepts:**

* Running replications with [`replication`](/concepts/hooks/replication)
* Executing database queries with [`query`](/concepts/hooks/query)
* Validation with [`check`](/concepts/hooks/check)
* Sending notifications via [`http`](/concepts/hooks/http) using `date_format` function
* Storing query results with `into`
* Conditional execution
* Accessing stored values with `store.`

## Cleanup and Archiving Pipeline

This pipeline archives files after processing and cleans up temporary data.

```yaml
steps:
  - type: list
    id: temp_files
    location: local//tmp/processing/
    only: files

  - type: group
    loop: state.temp_files.result
    steps:
      - type: copy
        from: '{loop.value.location}'
        to: aws_s3/archive/{timestamp.YYYY}/{timestamp.MM}/{loop.value.name}'

      - type: delete
        location: '{loop.value.location}'
        on_failure: warn

  - type: log
    message: 'Archived and deleted {length(state.temp_files.result)} files'
```

**Key Concepts:**

* File discovery using [`list`](/concepts/hooks/list)
* Iterative processing with [`group`](/concepts/hooks/group)
* Archiving files with [`copy`](/concepts/hooks/copy)
* Cleanup using [`delete`](/concepts/hooks/delete)
* Logging results with [`log`](/concepts/hooks/log)
* Using timestamps in file paths
* Error handling with `on_failure`

## Advanced Pipeline with Groups and Conditions

This pipeline uses nested groups, conditions, and multiple step types for a complex workflow.

```yaml
steps:
  - type: group
    id: preparation
    steps:
      - type: log
        message: 'Preparing environment'

      - type: command
        command: mkdir -p /tmp/processing
        print: true

  - type: replication
    path: replications/main.yaml
    id: main_rep

  - type: group
    if: state.main_rep.status == "success"
    steps:
      - type: query
        connection: target_db
        query: VACUUM ANALYZE {state.main_rep.object.full_name}

      - type: log
        message: 'Optimization complete'

  - type: group
    if: state.main_rep.status == "error"
    steps:
      - type: log
        message: 'Error occurred: {state.main_rep.error}'
        level: error

      - type: http
        url: https://errors.example.com/report
        method: POST
        payload: '{state.main_rep}'
```

**Key Concepts:**

* Organized workflows using [`group`](/concepts/hooks/group)
* Logging with [`log`](/concepts/hooks/log)
* System commands via [`command`](/concepts/hooks/command)
* Data replication with [`replication`](/concepts/hooks/replication)
* Database optimization using [`query`](/concepts/hooks/query)
* Nested groups for organization
* Conditional execution based on previous step status
* Error handling branch
* JSON serialization with `tojson`

These examples demonstrate the flexibility of Sling Pipelines and how to use built-in functions in expressions. You can combine and extend them based on your specific needs. For more details on individual step types, refer to the [Hooks](/concepts/hooks) documentation. For a full list of available functions, see the [functions documentation](/concepts/functions).


# Data Quality

Learn how to use sling for data quality

Sling provides several powerful features to ensure and maintain data quality throughout your data pipeline:

### Constraints

[Constraints](https://github.com/slingdata-io/sling-docs/blob/master/concepts/constraints.md) allow you to validate data at ingestion time using SQL-like syntax. They're specified at the column level and can prevent invalid data from entering your system.

```yaml
streams:
  my_stream:
    columns:
      # Ensure email is valid
      email: string | value ~ '^[^@]+@[^@]+\.[^@]+$'
      
      # Status can only be specific values
      status: string | value in ('active', 'pending', 'inactive')
      
      # Amount must be positive
      amount: decimal | value > 0
```

### Check Hooks

[Check](https://github.com/slingdata-io/sling-docs/blob/master/hooks/check.md) hooks enable you to implement custom validation logic at any point in your pipeline. They're particularly useful for:

* Validating row counts
* Ensuring data freshness
* Implementing complex business rules

```yaml
hooks:
  post:
    # Ensure minimum row count
    - type: check
      check: "run.total_rows >= 1000"
      on_failure: abort
    
    # Verify recent data
    - type: check
      check: "state.freshness_check.result.last_update >= timestamp.unix - 3600"
      on_failure: warn
```

### 3. Query Hooks

[Query](https://github.com/slingdata-io/sling-docs/blob/master/hooks/query.md) hooks allow you to run SQL-based quality checks and store results:

```yaml
hooks:
  post:
    - type: query
      connection: target_db
      query: |
        INSERT INTO quality_metrics (
          stream_name,
          check_time,
          null_rate,
          duplicate_rate
        )
        SELECT 
          '{run.stream.name}',
          CURRENT_TIMESTAMP,
          SUM(CASE WHEN email IS NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
          COUNT(*) - COUNT(DISTINCT id) * 100.0 / COUNT(*)
        FROM {run.object.full_name}
```

### Failure Handling

All data quality features support flexible failure handling:

* `abort`: Stop processing immediately
* `warn`: Continue but emit a warning
* `skip`: Skip the problematic record (for constraints)
* `quiet`: Continue silently

This allows you to implement the appropriate level of strictness for your use case.

By combining these features, you can build robust data quality checks throughout your pipeline, from ingestion to final delivery.


# Constraints

Learn how to use constraints

Constraints are a powerful feature that allow you to evaluate each value of a certain column and handle any failures. They can be specified using SQL-like syntax, separated by a `|` symbol. The advantage of using constraints is that data quality can be ensured *while* ingesting the data (at runtime), not way later in the pipeline.

```yaml
source: source_name
target: target_name

streams:
  my_stream:
    columns:
      # id values cannot be null
      id: bigint | value is not null

      # status values can only be active or inactive
      status: string | value in ('active', 'inactive')

      # col_1 value length can only be 6, 7 or 8
      col_1: string(8) | value_len > 5 and value_len <= 8
```

#### Handling failures

Sling looks for the environment variable `SLING_ON_CONSTRAINT_FAILURE` to know what to do.

Here are the values allowed in `SLING_ON_CONSTRAINT_FAILURE`:

* `warn`: This is the default. When using the Sling Platform, this will emit a warning status, which can notify you (Email, Slack, etc.)
* `skip`: Skip the record (do not ingest into target)
* `abort`: Will immediately abort the run, and fail/error. When using the Sling Platform, this can notify you (Email, Slack, etc.)

#### Supported Operators

* `is null` - Check if value is null
* `is not null` - Check if value is not null
* `==` - Equal to
* `!=` or `<>` - Not equal to
* `>` - Greater than
* `>=` - Greater than or equal to
* `<` - Less than
* `<=` - Less than or equal to
* `~` - Matches regex pattern
* `!~` - Does not match regex pattern
* `in` - Value matches any in list
* `not in` - Value does not match any in list
* `and` - Combine multiple conditions (all must be true)
* `or` - Combine multiple conditions (at least one must be true)

#### Special Variables

* `value` - Record value for respective column
* `value_len` - Length of record value for respective column

#### Using CLI Flags

```bash
# template. data type must be specified when using constraints
sling run --columns '{ "<column_name>": "<data_type> | <constraint expression>" }'

# values cannot null
sling run --columns '{ "my_column": "bigint | value is not null" }'

# cannot be zero
sling run --columns '{ "my_column": "bigint | value != 0" }'

```

#### Using YAML

Using the `defaults` and `streams` keys, you can specify different columns/constraints for each stream.

```yaml
source: source_name
target: target_name

defaults:
  # apply to all streams by default
  columns:
    id: bigint | value is not null  # value cannot be null

streams:
  # inherit defaults
  my_stream:

  my_other_stream:
    columns:
      # can evaluate value_len (number of digits)
      id: bigint | value_len > 0 or value = -10  

      # enum values
      my_column: string | value in ('first', 'second', 'third')

      # pattern matching or is null
      my_other_column: string | value ~ 'abc(d)' or value is null
```


# Functions

Unlock the power of your data workflows with these versatile built-in functions. They enable sophisticated data transformations, validations, and manipulations, and can be seamlessly integrated into your [pipelines](/concepts/pipeline), [hooks](/concepts/hooks), and [transforms](/concepts/replication/transforms).

{% hint style="success" %}
**CLI Pro Required**: Functions require a [CLI Pro token](/sling-cli/cli-pro) or [Platform Plan](https://github.com/slingdata-io/sling-docs/blob/master/concepts/sling-platform/platform.md).
{% endhint %}

## String Functions

| Function                            | Description                                                                                   | Parameters                                                                                                          | Returns            | Example                                                    |
| ----------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------ | ---------------------------------------------------------- |
| `contains(string, substring)`       | Checks if string contains substring                                                           | `string`, `substring`                                                                                               | Boolean            | `contains("hello world", "world")` → `true`                |
| `join(array, separator)`            | Joins array elements into string                                                              | `array`, `separator`                                                                                                | String             | `join(["a", "b", "c"], ", ")` → `"a, b, c"`                |
| `length(string\|array\|map)`        | Gets length of string/array/map                                                               | `value`: String, array or map                                                                                       | Number             | `length("hello")` → `5`, `length([1,2])` → `2`             |
| `lower(string)`                     | Converts string to lowercase                                                                  | `string`: Input string                                                                                              | Lowercase string   | `lower("HELLO")` → `"hello"`                               |
| `snake(string)`                     | Converts string to snake\_case (spaces become underscores)                                    | `string`: Input string                                                                                              | snake\_case string | `snake("Hello World")` → `"hello_world"`                   |
| `slugify(string)`                   | Converts string to identifier / URL friendly slug (lowercase with hyphens, alphanumeric only) | `string`: Input string                                                                                              | URL slug string    | `slugify("Hello, World!")` → `"hello-world"`               |
| `replace(string, pattern, replace)` | Replaces occurrences of pattern                                                               | `string`, `pattern`, `replacement`                                                                                  | Modified string    | `replace("hello", "l", "x")` → `"hexxo"`                   |
| `split_part(string, sep, index)`    | Gets part of split string by index                                                            | `string`, `separator`, `index` (0-based)                                                                            | String part        | `split_part("a,b,c", ",", 1)` → `"b"`                      |
| `split(string, separator)`          | Splits string into array                                                                      | `string`, `separator`                                                                                               | Array of strings   | `split("a,b,c", ",")` → `["a", "b", "c"]`                  |
| `substring(string, start[, end])`   | Extracts part of a string (0-indexed)                                                         | <p><code>string</code>: Input string<br><code>start</code>: Start index<br><code>end</code>: Optional end index</p> | Substring          | `substring("hello world", 0, 5)` → `"hello"`               |
| `trim(string)`                      | Removes whitespace from start/end                                                             | `string`: Input string                                                                                              | Trimmed string     | `trim(" hello ")` → `"hello"`                              |
| `upper(string)`                     | Converts string to uppercase                                                                  | `string`: Input string                                                                                              | Uppercase string   | `upper("hello")` → `"HELLO"`                               |
| `parse_ms_uuid(string)`             | Parses Microsoft UUID format                                                                  | `string`: 16-byte UUID string                                                                                       | Parsed UUID        | `parse_ms_uuid(binary_uuid)` → `"12345678-..."`            |
| `replace_0x00(string)`              | Removes null bytes from string                                                                | `string`: Input string                                                                                              | Cleaned string     | `replace_0x00("hello\x00world")` → `"helloworld"`          |
| `remove_diacritics(string)`         | Removes accents/diacritics                                                                    | `string`: Input string                                                                                              | ASCII string       | `remove_diacritics("café")` → `"cafe"`                     |
| `replace_non_printable(string)`     | Removes non-printable characters                                                              | `string`: Input string                                                                                              | Cleaned string     | `replace_non_printable("hello\x01world")` → `"helloworld"` |

## Numeric Functions

| Function                         | Description                 | Parameters                                                              | Returns           | Example                                     |
| -------------------------------- | --------------------------- | ----------------------------------------------------------------------- | ----------------- | ------------------------------------------- |
| `bool_parse(value)`              | Converts value to boolean   | `value`: Value to convert                                               | Boolean or error  | `bool_parse("true")` → `true`               |
| `float_format(value, format)`    | Formats float (Go format)   | <p><code>value</code>: Number<br><code>format</code>: Format string</p> | Formatted string  | `float_format(42.5678, "%.2f")` → `"42.57"` |
| `float_parse(value)`             | Converts value to float     | `value`: Value to convert                                               | Float or error    | `float_parse("42.5")` → `42.5`              |
| `greatest(array\|val1, val2...)` | Finds maximum value         | `array` or multiple values                                              | Maximum value     | `greatest(1, 5, 3)` → `5`                   |
| `int_format(value, format)`      | Formats integer (Go format) | <p><code>value</code>: Number<br><code>format</code>: Format string</p> | Formatted string  | `int_format(42, "%05d")` → `"00042"`        |
| `int_parse(value)`               | Converts value to integer   | `value`: Value to convert                                               | Integer or error  | `int_parse("42")` → `42`                    |
| `int_range(start, end[, step])`  | Generates integer range     | `start`, `end`: Integers, `step`: Optional integer (default 1)          | Array of integers | `int_range(1, 5)` → `[1,2,3,4,5]`           |
| `is_greater(val1, val2)`         | Checks if `val1 > val2`     | `val1`, `val2`: Values to compare                                       | Boolean           | `is_greater(5, 3)` → `true`                 |
| `is_less(val1, val2)`            | Checks if `val1 < val2`     | `val1`, `val2`: Values to compare                                       | Boolean           | `is_less(3, 5)` → `true`                    |
| `least(array\|val1, val2...)`    | Finds minimum value         | `array` or multiple values                                              | Minimum value     | `least(1, 5, 3)` → `1`                      |

## Date Functions

Uses Go's `time` package and `strftime` conventions via [timefmt-go](https://github.com/itchyny/timefmt-go). Please refer to [`man 3 strftime`](https://linux.die.net/man/3/strftime) and [`man 3 strptime`](https://linux.die.net/man/3/strptime) for formatter syntax.

| Function                           | Description                                                                                              | Parameters                                                                                                         | Returns              | Example                                                                                       |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------- | --------------------------------------------------------------------------------------------- |
| `now()`                            | Gets current date and time                                                                               | None                                                                                                               | Time object          | `now()`                                                                                       |
| `date_parse(string[, format])`     | Parses string to time object                                                                             | <p><code>string</code>: Date string<br><code>format</code>: "auto" or <code>strftime</code></p>                    | Time object or error | `date_parse("2022-01-01T10:00:00Z", "auto")`                                                  |
| `date_format(date, format)`        | Formats time object to string                                                                            | <p><code>date</code>: Time object<br><code>format</code>: <code>strftime</code> format</p>                         | Formatted string     | `date_format(now(), "%Y-%m-%d")` → `"2023-10-27"` (example date)                              |
| `date_add(date, duration[, unit])` | Adds duration to time object                                                                             | `date`, `duration` (int), `unit` (string, default "s")                                                             | Time object          | `date_add(now(), -7, "day")`                                                                  |
| `date_diff(date1, date2[, unit])`  | Time between dates                                                                                       | `date1`, `date2`, `unit` (string, default "s")                                                                     | Number (float)       | `date_diff(date_add(now(), 1, "day"), now(), "hour")` → `24.0`                                |
| `date_trunc(date, unit)`           | Truncates date to unit start                                                                             | `date`, `unit` ("year", "month", "day", "hour", etc.)                                                              | Time object          | `date_trunc(now(), "month")` → First day of current month at 00:00:00                         |
| `date_extract(date, part)`         | Extracts part from date                                                                                  | `date`, `part` ("year", "month", "day", "hour", etc.)                                                              | Number               | `date_extract(now(), "year")` → `2023` (example year)                                         |
| `date_last(date[, period])`        | Gets last day of period                                                                                  | `date`, `period` ("month", "year", default "month")                                                                | Time object          | `date_last(now())` → Last day of current month                                                |
| `date_first(date[, period])`       | Gets first day of period                                                                                 | `date`, `period` ("month", "year", default "month")                                                                | Time object          | `date_first(now())` → First day of current month                                              |
| `date_range(start, end[, step])`   | Creates array of dates                                                                                   | `start`, `end`: Dates, `step`: Duration string or int+unit                                                         | Array of dates       | `date_range("2023-01-01", "2023-01-03", "1d")` → `["2023-01-01", "2023-01-02", "2023-01-03"]` |
| `date_timezone(date, timezone)`    | Converts time to specified [IANA Timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) | <p><code>date</code>: Time object<br><code>timezone</code>: Timezone string (e.g., "UTC", "America/New\_York")</p> | Time object          | `date_timezone(now(), "America/New_York")` → Time in NY timezone                              |

*Date function `unit`/`part`/`period` parameters often accept: "year", "month", "week", "day", "hour", "minute", "second".* *`range` function with dates requires time objects as start/end.*

## Value Handling Functions

| Function                        | Description                                                 | Parameters                                         | Returns                  | Example                                                              |
| ------------------------------- | ----------------------------------------------------------- | -------------------------------------------------- | ------------------------ | -------------------------------------------------------------------- |
| `cast(value, type)`             | Converts value to type                                      | `value`, `type` ("int", "float", "string", "bool") | Converted value or error | `cast("42", "int")` → `42`                                           |
| `coalesce(val1, val2, ...)`     | Returns first non-null value                                | Multiple values                                    | First non-null value     | `coalesce(null, sync.val, state.val, "default")`                     |
| `element(array, index)`         | Gets element at 0-based index                               | `array`, `index` (integer)                         | Element or error         | `element(["a", "b"], 1)` → `"b"`                                     |
| `equals(val1, val2)`            | Checks deep equality                                        | `val1`, `val2`                                     | Boolean                  | `equals(response.status, 200)` → `true`                              |
| `first_valid(val1, val2, ...)`  | Returns first non-null and non-empty value                  | Multiple values                                    | First valid value        | `first_valid("", state.val, "default")` → `state.val` (if not empty) |
| `if(condition, then, else)`     | Conditional expression                                      | `condition` (bool), `then_val`, `else_val`         | Selected value           | `if(state.count > 0, "has items", "empty")`                          |
| `is_empty(value)`               | Checks if value is empty                                    | `value` (string, array, map)                       | Boolean                  | `is_empty("")` → `true`, `is_empty([])` → `true`                     |
| `is_null(value)`                | Checks if value is null                                     | `value`                                            | Boolean                  | `is_null(state.optional_param)` → `true` or `false`                  |
| `null_if(value, sentinel, ...)` | Returns null when value equals any sentinel, else the value | `value`, one or more `sentinel` values             | Value or null            | `null_if(record.amount, false)` → `null` (if value is `false`)       |
| `require(val[, error_msg])`     | Ensures value is not null or error                          | `val`, `error_msg` (optional)                      | Value or error           | `require(secrets.api_key, "API Key is required")`                    |
| `try_cast(value, type)`         | Tries conversion, returns null                              | `value`, `type`                                    | Converted value or null  | `try_cast("abc", "int")` → `null`                                    |

## Collection Functions

| Function                            | Description                                          | Parameters                                                                         | Returns           | Example                                                                                                         |
| ----------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------- |
| `array(val1, val2, ...)`            | Creates array from values                            | Multiple values                                                                    | Array             | `array(1, 2, 3)` → `[1, 2, 3]`                                                                                  |
| `chunk(array\|queue, size)`         | Splits array/queue into chunks                       | `array` or `queue.name`, `size` (int)                                              | Channel of arrays | Used in `iterate`: `over: chunk(queue.ids, 50)`                                                                 |
| `filter(array, expression)`         | Filters array using expression                       | `array`, `expression` (string condition)                                           | Filtered array    | `filter([1, 2, 3], "element > 1")` → `[2, 3]`                                                                   |
| `get_path(object, path)`            | Gets value using dot notation                        | `object`, `path` (string, e.g., "a.b\[0]")                                         | Value at path     | `get_path(response.json, "user.profile.email")`                                                                 |
| `jmespath(object, expression)`      | Evaluates JMESPath expression                        | `object`, `expression` (string)                                                    | Query result      | `jmespath(response.json, "data.items[?age > 30]")`                                                              |
| `jq(object, expression)`            | Evaluates jq expression                              | `object`, `expression` (jq filter string)                                          | Array of results  | `jq(response.json, ".data.items[] \| select(.age > 30)")`                                                       |
| `keys(map)`                         | Gets keys from map                                   | `map` object                                                                       | Array of keys     | `keys({"a": 1, "b": 2})` → `["a", "b"]`                                                                         |
| `exists(collection, item)`          | Checks if key exists in map or value exists in array | `collection`: Map or array, `item`: Key or value to find                           | Boolean           | `exists({"a": 1}, "a")` → `true`, `exists([1, 2], 2)` → `true`                                                  |
| `object(k1, v1, k2, v2, ...)`       | Creates object from key/value pairs                  | Even number of arguments (key, value pairs)                                        | Map/object        | `object("name", "John", "age", 30)` → `{"name": "John", "age": 30}`                                             |
| `pluck(data, column_name)`          | Extracts values from a column                        | `data`: Array of maps, `column_name`: String                                       | Array of values   | `pluck([{"name": "John", "age": 30}, {"name": "Jane", "age": 25}], "name")` → `["John", "Jane"]`                |
| `explode(record, field_name)`       | Expands a record by unnesting an array field         | `record`: Map with a nested array field, `field_name`: Key of the array to explode | Array of maps     | `explode({"id": 1, "items": [{"k": "a"}, {"k": "b"}]}, "items")` → `[{"id": 1, "k": "a"}, {"id": 1, "k": "b"}]` |
| `range(start, end[, step])`         | Generates range (auto-detects type)                  | `start`, `end`: Numbers or dates, `step`: Optional                                 | Array or channel  | `range(1, 5)` → `[1,2,3,4,5]`, delegates to `int_range` or `date_range`                                         |
| `sort(array[, descending])`         | Sorts array elements                                 | `array`, `descending` (optional bool)                                              | Sorted array      | `sort([3, 1, 2])` → `[1, 2, 3]`                                                                                 |
| `values(map)`                       | Gets values from map                                 | `map` object                                                                       | Array of values   | `values({"a": 1, "b": 2})` → `[1, 2]`                                                                           |
| `object_rename(map, old, new, ...)` | Renames keys in map                                  | `map`: Object, followed by pairs of `old_key`, `new_key`                           | Modified map      | `object_rename({"a": 1, "b": 2}, "a", "x")` → `{"x": 1, "b": 2}`                                                |
| `object_delete(map, key1, ...)`     | Deletes keys from map                                | `map`: Object, followed by keys to delete                                          | Modified map      | `object_delete({"a": 1, "b": 2}, "a")` → `{"b": 2}`                                                             |
| `object_casing(map, casing)`        | Transforms keys in map to specified casing           | `map`: Object, `casing`: "snake", "camel", "upper", "lower"                        | Modified map      | `object_casing({"firstName": "John"}, "snake")` → `{"first_name": "John"}`                                      |
| `object_merge(map1, map2, ...)`     | Merges multiple maps together                        | Two or more maps to merge (later maps override earlier)                            | Merged map        | `object_merge({"a": 1}, {"b": 2}, {"c": 3})` → `{"a": 1, "b": 2, "c": 3}`                                       |
| `object_zip(keys, values)`          | Creates object from key/value arrays                 | `keys`: Array of strings, `values`: Array of values (matched by position)          | Map/object        | `object_zip(["a", "b"], [1, 2])` → `{"a": 1, "b": 2}`                                                           |

> **`jq()` vs `jmespath()`:** Both query JSON data, but use different syntax. `jmespath()` (alias: `jp()`) uses JMESPath syntax and returns a single value. `jq()` uses [jq filter syntax](https://jqlang.github.io/jq/manual/) (dot-prefix paths, pipe operators, `select()` for filtering) and always returns an array of results. In processor expressions, append `[0]` to get a single value from `jq()`: `jq(record, ".user.name")[0]`. In `response.records.jq`, the array is handled natively.

## Encoding/Decoding Functions

| Function                    | Description            | Parameters                                                     | Returns            | Example                                               |
| --------------------------- | ---------------------- | -------------------------------------------------------------- | ------------------ | ----------------------------------------------------- |
| `encode_url(string)`        | URL encodes a string   | `string`                                                       | Encoded string     | `encode_url("a b")` → `"a%20b"`                       |
| `decode_url(string)`        | URL decodes a string   | `string`                                                       | Decoded string     | `decode_url("a%20b")` → `"a b"`                       |
| `encode_base64(string)`     | Base64 encodes string  | `string`                                                       | Encoded string     | `encode_base64("user:pass")` → `"dXNlcjpwYXNz"`       |
| `decode_base64(string)`     | Base64 decodes string  | `string`                                                       | Decoded string     | `decode_base64("dXNlcjpwYXNz")` → `"user:pass"`       |
| `hash(string[, algorithm])` | Creates hash of string | `string`, `algorithm` ("md5", "sha1", "sha256", default "md5") | Hash string (hex)  | `hash("hello", "md5")` → `"5d4..."`                   |
| `json_parse(string)`        | Parses JSON string     | `string`: JSON string                                          | Object/Array/Value | `json_parse('{"name": "John"}')` → `{"name": "John"}` |

## Utility Functions

| Function                                      | Description                         | Parameters                                                                                           | Returns                | Example                                                             |
| --------------------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------------- |
| `uuid()`                                      | Generates random UUID v4            | None                                                                                                 | UUID string            | `uuid()` → `"..."`                                                  |
| `log(message)`                                | Logs a message during evaluation    | `message`                                                                                            | The message (passthru) | `log("Processing record: " + record.id)`                            |
| `regex_match(string, pattern)`                | Checks if string matches pattern    | `string`, `pattern` (Go regex)                                                                       | Boolean                | `regex_match("img_123.jpg", "^img_.*\\.jpg$")` → `true`             |
| `regex_extract(string, pattern[, idx])`       | Extracts matches using regex        | `string`, `pattern`, `idx` (optional)                                                                | Matches/group or null  | `regex_extract("id=123", "id=(\\d+)", 1)` → `"123"`                 |
| `regex_replace(string, pattern, replacement)` | Replaces pattern matches            | `string`, `pattern` (Go regex), `replacement`                                                        | Modified string        | `regex_replace("hello123world", "\\d+", "-")` → `"hello-world"`     |
| `pretty_table(data)`                          | Renders array of maps as table      | `data`: Array of maps                                                                                | Formatted table string | `pretty_table([{"name": "John", "age": 30}])` → formatted table     |
| `type_of(value)`                              | Gets type of value                  | `value`                                                                                              | Type string            | `type_of(42)` → `"integer"`                                         |
| `conn_property(connection, key)`              | Gets property value from connection | <p><code>connection</code>: Connection name (string)<br><code>key</code>: Property name (string)</p> | Property value         | `conn_property("my_db", "host")` → `"localhost"`                    |
| `machine_stats()`                             | Gets machine resource statistics    | None                                                                                                 | Object with stats      | `machine_stats()` → `{"memory_percent": 45.2, "cpu_percent": 23.5}` |

```
```


# File to Database

Examples of using Sling to load data from storage systems to databases

We first need to make sure our connections are available in our environment. See [Environment](https://github.com/slingdata-io/sling-docs/blob/master/environment.md), [Storage Connections](/connections/file-connections) and [Database Connections](/connections/database-connections) for more details.

{% tabs %}
{% tab title="Linux / Mac" %}

```bash
export MY_TARGET_DB='...'
export SLING_SAMPLE_SIZE=2000 # increase the sample size to infer types. Default is 900.
export SLING_THREADS=3 # run streams concurrently

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_TARGET_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

{% endtab %}

{% tab title="Windows" %}

```powershell
# using windows Powershell
$env:MY_TARGET_DB = '...'
$env:SLING_SAMPLE_SIZE = 2000 # increase the sample size to infer types. Default is 900.
$env:SLING_THREADS = 3 # run streams concurrently

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_TARGET_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

{% endtab %}
{% endtabs %}

<details>

<summary>Local Storage (CSV) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ cat /tmp/my_file.csv | sling run --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_file.csv' \
  --columns '{ "*": string }' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_csv_folder/' \
  --columns '{col2: string, col3: string}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_csv_folder/' \
  --transforms '[remove_accents]' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file://C:/Temp/my_csv_folder/' \
  --transforms '[remove_accents]' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'

streams:
  # a stream with many parts, all sub files will be merged into one table
  "file:///tmp/my_csv_folder/":
    columns:
      col2: string # cast `col2` as string
    transforms: [remove_accents] # Apply transforms. Here we are removing diacritics (accents) from string values.
    source_options:
      format: csv

  # expand all files into individual streams, each file will load into its own table
  "file:///tmp/my_csv_folder/*.csv":
    object: 'target_schema.{stream_file_name}'

  # consider as a single stream (don't expand into individual streams)
  "file:///tmp/my_csv_folder/prefix_*.csv":
    object: 'target_schema.my_new_table'
    single: true

  "file:///tmp/my_file.csv":
    columns:
      "*": string # cast all columns to string

  # Windows path format
  "file://C:/Temp/my_file.csv":
    columns:
      "*": string # cast all columns to string

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'
os.environ['SLING_SAMPLE_SIZE'] = '2000'  # increase the sample size to infer types
os.environ['SLING_THREADS'] = '3'  # run streams concurrently

# Single CSV file import
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={'mode': Mode.FULL_REFRESH},
    streams={
        'file:///tmp/my_file.csv': ReplicationStream(
            object='target_schema.target_table',
            columns={'*': 'string'}  # cast all columns to string
        )
    }
)

replication.run()

# Multiple CSV files with various options
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}'
    },
    streams={
        # A stream with many parts, all sub files will be merged into one table
        'file:///tmp/my_csv_folder/': ReplicationStream(
            columns={'col2': 'string'},  # cast col2 as string
            transforms=['remove_accents'],  # Apply transforms
            source_options=SourceOptions(format=Format.CSV)
        ),
        # Expand all files into individual streams
        'file:///tmp/my_csv_folder/*.csv': ReplicationStream(
            object='target_schema.{stream_file_name}'
        ),
        # Consider as a single stream (don't expand)
        'file:///tmp/my_csv_folder/prefix_*.csv': ReplicationStream(
            object='target_schema.my_new_table',
            single=True
        ),
        # Windows path format
        'file://C:/Temp/my_file.csv': ReplicationStream(
            columns={'*': 'string'}
        )
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',  # adds a _sling_stream_url column
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Local Storage (Excel) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-stream 'file:///path/to/test.excel.xlsx' --src-options '{ sheet: "Sheet2!A:F" }' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'

streams:
  # expand all files into a stream, each file will load into its own table
  "file:///tmp/my_excel_folder/*.xlsx":
    object: 'target_schema.{stream_file_name}'
    source_options:
      sheet: "Sheet1!A:F"
  
   # consider as a single stream (don't expand into individual streams)
  "file:///tmp/my_excel_folder/prefix_*.xlsx":
    object: 'target_schema.my_new_table'
    single: true
    source_options:
      sheet: "Sheet1!A:F"

  "file:///path/to/test.excel.xlsx":
    columns:
      "*": string # cast all columns to string
    source_options:
      sheet: "Sheet2!A:F"

  # Windows path format
  "file://C:/Temp/my_file.xlsx":
    columns:
      "col2": integer # cast col2 to integer
    source_options:
      sheet: "Sheet2!A:F"

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_ROW_NUM_COLUMN: true # adds a _sling_row_num column with the row number
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'

# Excel file import with sheet specification
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}'
    },
    streams={
        # Expand all files into streams
        'file:///tmp/my_excel_folder/*.xlsx': ReplicationStream(
            object='target_schema.{stream_file_name}',
            source_options=SourceOptions(sheet='Sheet1!A:F')
        ),
        # Consider as a single stream (don't expand)
        'file:///tmp/my_excel_folder/prefix_*.xlsx': ReplicationStream(
            object='target_schema.my_new_table',
            single=True,
            source_options=SourceOptions(sheet='Sheet1!A:F')
        ),
        # Single Excel file
        'file:///path/to/test.excel.xlsx': ReplicationStream(
            columns={'*': 'string'},  # cast all columns to string
            source_options=SourceOptions(sheet='Sheet2!A:F')
        ),
        # Windows path format
        'file://C:/Temp/my_file.xlsx': ReplicationStream(
            columns={'col2': 'integer'},  # cast col2 to integer
            source_options=SourceOptions(sheet='Sheet2!A:F')
        )
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',  # adds a _sling_stream_url column
        'SLING_ROW_NUM_COLUMN': 'true'  # adds a _sling_row_num column
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Local Storage (JSON) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ cat /tmp/my_file.json | sling run --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_file.json' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_json_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

# Windows path format
$ sling run --src-stream 'file://C:/Temp/my_json_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.target_table'
  source_options:
    format: json

streams:
  "file:///tmp/my_json_folder/":

  # expand all files into a stream, each file will load into its own table
  "file:///tmp/my_json_folder/*.json":
    object: 'target_schema.{stream_file_name}'

  # consider as a single stream (don't expand into individual streams)
  "file:///tmp/my_json_folder/prefix_*.json":
    object: 'target_schema.my_new_table'
    single: true

  "file:///tmp/my_file.json":

  # Windows path format
  "file://C:/Temp/my_file.json":
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'

# JSON file import
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.target_table',
        'source_options': SourceOptions(format=Format.JSON)
    },
    streams={
        # JSON folder
        'file:///tmp/my_json_folder/': {},
        # Expand all files into streams
        'file:///tmp/my_json_folder/*.json': ReplicationStream(
            object='target_schema.{stream_file_name}'
        ),
        # Consider as a single stream (don't expand)
        'file:///tmp/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_new_table',
            single=True
        ),
        # Single JSON file
        'file:///tmp/my_file.json': {},
        # Windows path format
        'file://C:/Temp/my_file.json': {}
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Local Storage (JSON Flattened) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ cat /tmp/my_file.json | sling run --src-options '{flatten: true}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_file.json' \
  --src-options '{flatten: true}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_json_folder/' \
  --src-options '{flatten: true}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \

# Windows path format
$ sling run --src-stream 'file://C:/Temp/my_json_folder/' \
  --src-options '{flatten: true}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.target_table'
  source_options:
    format: json
    flatten: true

streams:
  "file:///tmp/my_json_folder/":

  # expand all files into a stream, each file will load into its own table
  "file:///tmp/my_json_folder/*.json":
    object: 'target_schema.{stream_file_name}'

  # consider as a single stream (don't expand into individual streams)
  "file:///tmp/my_json_folder/prefix_*.json":
    object: 'target_schema.my_new_table'
    single: true

  "file:///tmp/my_file.json":

  # Windows path format
  "file://C:/Temp/my_file.json":

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'

# JSON file import with flattening
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.target_table',
        'source_options': SourceOptions(
            format=Format.JSON,
            flatten=True
        )
    },
    streams={
        # JSON folder
        'file:///tmp/my_json_folder/': {},
        # Expand all files into streams
        'file:///tmp/my_json_folder/*.json': ReplicationStream(
            object='target_schema.{stream_file_name}'
        ),
        # Consider as a single stream (don't expand)
        'file:///tmp/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_new_table',
            single=True
        ),
        # Single JSON file
        'file:///tmp/my_file.json': {},
        # Windows path format
        'file://C:/Temp/my_file.json': {}
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Local Storage (Parquet) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-stream 'file:///tmp/my_file.parquet' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_parquet_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

# Windows path format
$ sling run --src-stream 'file://C:/Temp/my_parquet_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.target_table'
  source_options:
    format: parquet

streams:
  "file:///tmp/my_parquet_folder/":

  # expand all files into a stream, each file will load into its own table
  "file:///tmp/my_parquet_folder/*.parquet":
    object: 'target_schema.{stream_file_name}'

  # consider as a single stream (don't expand into individual streams)
  "file:///tmp/my_parquet_folder/prefix_*.parquet":
    object: 'target_schema.my_new_table'
    single: true

  "file:///tmp/my_file.parquet":

  # Windows path format
  "file://C:/Temp/my_file.parquet":
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'

# Parquet file import
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.target_table',
        'source_options': SourceOptions(format=Format.PARQUET)
    },
    streams={
        # Parquet folder
        'file:///tmp/my_parquet_folder/': {},
        # Expand all files into streams
        'file:///tmp/my_parquet_folder/*.parquet': ReplicationStream(
            object='target_schema.{stream_file_name}'
        ),
        # Consider as a single stream (don't expand)
        'file:///tmp/my_parquet_folder/prefix_*.parquet': ReplicationStream(
            object='target_schema.my_new_table',
            single=True
        ),
        # Single Parquet file
        'file:///tmp/my_file.parquet': {},
        # Windows path format
        'file://C:/Temp/my_file.parquet': {}
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Local Storage (SAS7BDAT) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-stream 'file:///tmp/my_file.sas7bdat' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file:///tmp/my_sas7bdat_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

# Windows path format
$ sling run --src-stream 'file://C:/tmp/my_file.sas7bdat' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-stream 'file://C:/Temp/my_sas7bdat_folder/' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: LOCAL
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.target_table'
  source_options:
    format: sas7bdat

streams:
  "file:///tmp/my_sas7bdat_folder/":
  "file:///tmp/my_file.sas7bdat":

  # Windows path format
  "file://C:/Temp/my_file.sas7bdat":
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_TARGET_DB'] = '...'

# SAS7BDAT file import
replication = Replication(
    source='LOCAL',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.target_table',
        'source_options': SourceOptions(format=Format.SAS)
    },
    streams={
        # SAS7BDAT folder
        'file:///tmp/my_sas7bdat_folder/': {},
        # Single SAS7BDAT file
        'file:///tmp/my_file.sas7bdat': {},
        # Windows path format
        'file://C:/Temp/my_file.sas7bdat': {}
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>SFTP Storage (CSV) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SFTP --src-stream '/path/to/my_file.csv' \
  --columns '{ "*": string }' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-conn MY_SFTP --src-stream '/path/to/my_csv_folder/' \
  --columns '{col2: string, col3: string}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

$ sling run --src-conn MY_SFTP --src-stream '/path/to/my_csv_folder/' \
  --transforms '[remove_accents]' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SFTP
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'

streams:
  "/path/to/my_csv_folder/":
    columns:
      col2: string # cast `col2` as string
    transforms: [remove_accents] # Apply transforms. Here we are removing diacritics (accents) from string values.
    source_options:
      format: csv

  # expand all files into a stream, each file will load into its own table
  "/path/to/my_csv_folder/*.csv":

  # consider as a single stream (don't expand into individual streams)
  "/path/to/my_csv_folder/prefix_*.csv":
    object: my_scheam.my_new_table
    single: true

  "/path/to/my_file.csv":
    columns:
      "*": string # cast all columns to string

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_SFTP'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# SFTP CSV file import
replication = Replication(
    source='MY_SFTP',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}'
    },
    streams={
        # SFTP CSV folder
        '/path/to/my_csv_folder/': ReplicationStream(
            columns={'col2': 'string'},  # cast col2 as string
            transforms=['remove_accents'],  # Apply transforms
            source_options=SourceOptions(format=Format.CSV)
        ),
        # Expand all files into streams
        '/path/to/my_csv_folder/*.csv': {},
        # Consider as a single stream (don't expand)
        '/path/to/my_csv_folder/prefix_*.csv': ReplicationStream(
            object='my_scheam.my_new_table',
            single=True
        ),
        # Single SFTP CSV file
        '/path/to/my_file.csv': ReplicationStream(
            columns={'*': 'string'}  # cast all columns to string
        )
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (CSV) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_csv_folder/' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_csv_folder/' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_csv_folder/' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GOOGLE_DRIVE --src-stream 'gdrive://folder_id/my_csv_folder/' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_file.csv' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_file.csv' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.csv' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GOOGLE_DRIVE --src-stream 'gdrive://folder_id/my_file.csv' --columns '{col2: string}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_CONN
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  columns:
    '*': string # cast all columns as string
  source_options:
    format: csv

streams:
  # no need to specify scheme://bucket
  "my_file.csv":
  "my_csv_folder/":               # single stream for whole folder
  "my_csv_folder/*.csv":          # individual streams for each file
  "my_csv_folder/prefix_*.csv":   # single stream for all files
    object: 'target_schema.my_csv_data'
    single: true

  "s3://my-bucket/my_csv_folder/":
  "s3://my-bucket/my_csv_folder/*.csv":
  "s3://my-bucket/my_csv_folder/prefix_*.csv":
    object: 'target_schema.my_csv_data'
    single: true
  "s3://my-bucket/my_file.csv":

  "gs://my-bucket/my_csv_folder/":
  "gs://my-bucket/my_csv_folder/*.csv":
  "gs://my-bucket/my_csv_folder/prefix_*.csv":
    object: 'target_schema.my_csv_data'
    single: true
  "gs://my-bucket/my_file.csv":

  "gdrive://folder_id/my_csv_folder/":
  "gdrive://folder_id/my_csv_folder/*.csv":
  "gdrive://folder_id/my_csv_folder/prefix_*.csv":
    object: 'target_schema.my_csv_data'
    single: true
  "gdrive://folder_id/my_file.csv":

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_CONN'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage CSV file import
replication = Replication(
    source='MY_FILE_CONN',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'columns': {'*': 'string'},  # cast all columns as string
        'source_options': SourceOptions(format=Format.CSV)
    },
    streams={
        # No need to specify scheme://bucket
        'my_file.csv': {},
        'my_csv_folder/': {},  # single stream for whole folder
        'my_csv_folder/*.csv': {},  # individual streams for each file
        'my_csv_folder/prefix_*.csv': ReplicationStream(  # single stream for all files
            object='target_schema.my_csv_data',
            single=True
        ),
        
        # With full S3 path
        's3://my-bucket/my_csv_folder/': {},
        's3://my-bucket/my_csv_folder/*.csv': {},
        's3://my-bucket/my_csv_folder/prefix_*.csv': ReplicationStream(
            object='target_schema.my_csv_data',
            single=True
        ),
        's3://my-bucket/my_file.csv': {},
        # Google Cloud Storage
        'gs://my-bucket/my_csv_folder/': {},
        'gs://my-bucket/my_csv_folder/*.csv': {},
        'gs://my-bucket/my_csv_folder/prefix_*.csv': ReplicationStream(
            object='target_schema.my_csv_data',
            single=True
        ),
        'gs://my-bucket/my_file.csv': {},
        # Google Drive
        'gdrive://folder_id/my_csv_folder/': {},
        'gdrive://folder_id/my_csv_folder/*.csv': {},
        'gdrive://folder_id/my_csv_folder/prefix_*.csv': ReplicationStream(
            object='target_schema.my_csv_data',
            single=True
        ),
        'gdrive://folder_id/my_file.csv': {}
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (JSON) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_STORAGE
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  columns:
    '*': string # cast all columns as string
  source_options:
    format: json

streams:
  # no need to specify scheme://bucket
  "my_file.json":
  "my_json_folder/":                # single stream for whole folder
  "my_json_folder/*.json":          # individual streams for each file
  "my_json_folder/prefix_*.json":   # single stream for all files
    object: 'target_schema.my_json_data'
    single: true

  "s3://my-bucket/my_json_folder/": 
  "s3://my-bucket/my_json_folder/*.json":
  "s3://my-bucket/my_json_folder/prefix_*.json":
    object: 'target_schema.my_json_data'
    single: true
  "s3://my-bucket/my_file.json":

  "gs://my-bucket/my_json_folder/":
  "gs://my-bucket/my_json_folder/*.json":
  "gs://my-bucket/my_json_folder/prefix_*.json":
    object: 'target_schema.my_json_data'
    single: true
  "gs://my-bucket/my_file.json":
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_STORAGE'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage JSON file import
replication = Replication(
    source='MY_FILE_STORAGE',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'columns': {'*': 'string'},  # cast all columns as string
        'source_options': SourceOptions(format=Format.JSON)
    },
    streams={
        # No need to specify scheme://bucket
        'my_file.json': {},
        'my_json_folder/': {},  # single stream for whole folder
        'my_json_folder/*.json': {},  # individual streams for each file
        'my_json_folder/prefix_*.json': ReplicationStream(  # single stream for all files
            object='target_schema.my_json_data',
            single=True
        ),
        # With full S3 path
        's3://my-bucket/my_json_folder/': {},
        's3://my-bucket/my_json_folder/*.json': {},
        's3://my-bucket/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_json_data',
            single=True
        ),
        's3://my-bucket/my_file.json': {},
        # Google Cloud Storage
        'gs://my-bucket/my_json_folder/': {},
        'gs://my-bucket/my_json_folder/*.json': {},
        'gs://my-bucket/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_json_data',
            single=True
        ),
        'gs://my-bucket/my_file.json': {}
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (JSON Flattened) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-options '{flatten: true}' --src-stream 's3://my-bucket/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-options '{flatten: true}' --src-stream 'gs://my-bucket/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-options '{flatten: true}' --src-stream 'https://my_account.blob.core.windows.net/my-container/my_json_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-options '{flatten: true}' --src-stream 's3://my-bucket/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-options '{flatten: true}' --src-stream 'gs://my-bucket/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-options '{flatten: true}' --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.json' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_STORAGE
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  columns:
    '*': string # cast all columns as string
  source_options:
    format: json
    flatten: true

streams:
  # no need to specify scheme://bucket
  "my_json_folder/":                # single stream for whole folder
  "my_json_folder/*.json":          # individual streams for each file
  "my_json_folder/prefix_*.json":   # single stream for all files
    object: 'target_schema.my_json_data'
    single: true

  "s3://my-bucket/my_json_folder/":
  "s3://my-bucket/my_json_folder/*.json":
  "s3://my-bucket/my_json_folder/prefix_*.json":
    object: 'target_schema.my_json_data'
    single: true
  "s3://my-bucket/my_file.json":

  "gs://my-bucket/my_json_folder/":
  "gs://my-bucket/my_json_folder/*.json":
  "gs://my-bucket/my_json_folder/prefix_*.json":
    object: 'target_schema.my_json_data'
    single: true
  "gs://my-bucket/my_file.json":

env:
  SLING_SAMPLE_SIZE: 2000 # increase the sample size to infer types (default=900).
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_STORAGE'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage JSON file import with flattening
replication = Replication(
    source='MY_FILE_STORAGE',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'columns': {'*': 'string'},  # cast all columns as string
        'source_options': SourceOptions(
            format=Format.JSON,
            flatten=True
        )
    },
    streams={
        # No need to specify scheme://bucket
        'my_json_folder/': {},  # single stream for whole folder
        'my_json_folder/*.json': {},  # individual streams for each file
        'my_json_folder/prefix_*.json': ReplicationStream(  # single stream for all files
            object='target_schema.my_json_data',
            single=True
        ),
        # With full S3 path
        's3://my-bucket/my_json_folder/': {},
        's3://my-bucket/my_json_folder/*.json': {},
        's3://my-bucket/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_json_data',
            single=True
        ),
        's3://my-bucket/my_file.json': {},
        # Google Cloud Storage
        'gs://my-bucket/my_json_folder/': {},
        'gs://my-bucket/my_json_folder/*.json': {},
        'gs://my-bucket/my_json_folder/prefix_*.json': ReplicationStream(
            object='target_schema.my_json_data',
            single=True
        ),
        'gs://my-bucket/my_file.json': {}
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (Parquet) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_parquet_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_parquet_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_parquet_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_file.parquet' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_file.parquet' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.parquet' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_STORAGE
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  source_options:
    format: parquet

streams:
  # no need to specify scheme://bucket
  "my_file.parquet":
  "my_parquet_folder/":                   # single stream for whole folder
  "my_parquet_folder/*.parquet":          # one stream for each file
  "my_parquet_folder/prefix_*.parquet":   # single stream for all files
    object: 'target_schema.my_parquet_data'
    single: true

  "s3://my-bucket/my_parquet_folder/":
  "s3://my-bucket/my_parquet_folder/*.parquet":
  "s3://my-bucket/my_parquet_folder/prefix_*.parquet":
    object: 'target_schema.my_parquet_data'
    single: true
  "s3://my-bucket/my_file.parquet":

  "gs://my-bucket/my_parquet_folder/":
  "gs://my-bucket/my_parquet_folder/*.parquet":
  "gs://my-bucket/my_parquet_folder/prefix_*.parquet":
    object: 'target_schema.my_parquet_data'
    single: true
  "gs://my-bucket/my_file.parquet":

env:
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_STORAGE'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage Parquet file import
replication = Replication(
    source='MY_FILE_STORAGE',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'source_options': SourceOptions(format=Format.PARQUET)
    },
    streams={
        # No need to specify scheme://bucket
        'my_file.parquet': {},
        'my_parquet_folder/': {},  # single stream for whole folder
        'my_parquet_folder/*.parquet': {},  # one stream for each file
        'my_parquet_folder/prefix_*.parquet': ReplicationStream(  # single stream for all files
            object='target_schema.my_parquet_data',
            single=True
        ),
        # With full S3 path
        's3://my-bucket/my_parquet_folder/': {},
        's3://my-bucket/my_parquet_folder/*.parquet': {},
        's3://my-bucket/my_parquet_folder/prefix_*.parquet': ReplicationStream(
            object='target_schema.my_parquet_data',
            single=True
        ),
        's3://my-bucket/my_file.parquet': {},
        # Google Cloud Storage
        'gs://my-bucket/my_parquet_folder/': {},
        'gs://my-bucket/my_parquet_folder/*.parquet': {},
        'gs://my-bucket/my_parquet_folder/prefix_*.parquet': ReplicationStream(
            object='target_schema.my_parquet_data',
            single=True
        ),
        'gs://my-bucket/my_file.parquet': {}
    },
    env={
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (Avro) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_avro_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_avro_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_avro_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_file.avro' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_file.avro' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.avro' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_STORAGE
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  source_options:
    format: avro

streams:
  # no need to specify scheme://bucket
  "my_file.avro":
  "my_avro_folder/":                # single stream for whole folder
  "my_avro_folder/*.avro":          # one stream for each file
  "my_avro_folder/prefix_*.avro":   # single stream for all files
    object: 'target_schema.my_avro_data'
    single: true

  "s3://my-bucket/my_avro_folder/":
  "s3://my-bucket/my_avro_folder/*.avro":
  "s3://my-bucket/my_avro_folder/prefix_*.avro":
    object: 'target_schema.my_avro_data'
    single: true
  "s3://my-bucket/my_file.avro":

  "gs://my-bucket/my_avro_folder/":
  "gs://my-bucket/my_avro_folder/*.avro":
  "gs://my-bucket/my_avro_folder/prefix_*.avro":
    object: 'target_schema.my_avro_data'
    single: true
  "gs://my-bucket/my_file.avro":
  
env:
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_STORAGE'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage Avro file import
replication = Replication(
    source='MY_FILE_STORAGE',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'source_options': SourceOptions(format=Format.AVRO)
    },
    streams={
        # No need to specify scheme://bucket
        'my_file.avro': {},
        'my_avro_folder/': {},  # single stream for whole folder
        'my_avro_folder/*.avro': {},  # one stream for each file
        'my_avro_folder/prefix_*.avro': ReplicationStream(  # single stream for all files
            object='target_schema.my_avro_data',
            single=True
        ),
        # With full S3 path
        's3://my-bucket/my_avro_folder/': {},
        's3://my-bucket/my_avro_folder/*.avro': {},
        's3://my-bucket/my_avro_folder/prefix_*.avro': ReplicationStream(
            object='target_schema.my_avro_data',
            single=True
        ),
        's3://my-bucket/my_file.avro': {},
        # Google Cloud Storage
        'gs://my-bucket/my_avro_folder/': {},
        'gs://my-bucket/my_avro_folder/*.avro': {},
        'gs://my-bucket/my_avro_folder/prefix_*.avro': ReplicationStream(
            object='target_schema.my_avro_data',
            single=True
        ),
        'gs://my-bucket/my_file.avro': {}
    },
    env={
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cloud Storage (XML) ⇨ Database</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_xml_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_xml_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_xml_folder/' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_S3_BUCKET --src-stream 's3://my-bucket/my_file.xml' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_GS_BUCKET --src-stream 'gs://my-bucket/my_file.xml' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

$ sling run --src-conn MY_AZURE_CONT --src-stream 'https://my_account.blob.core.windows.net/my-container/my_file.xml' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_FILE_STORAGE
target: MY_TARGET_DB

defaults:
  mode: full-refresh
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  source_options:
    format: xml

streams:
  # no need to specify scheme://bucket
  "my_xml_folder/":
  "my_file.xml":

  "s3://my-bucket/my_xml_folder/":
  "s3://my-bucket/my_file.xml":

  "gs://my-bucket/my_xml_folder/":
  "gs://my-bucket/my_file.xml":
  
env:
  SLING_STREAM_URL_COLUMN: true # adds a _sling_stream_url column with file path
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode, Format
import os

# Set environment variables
os.environ['MY_FILE_STORAGE'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Cloud Storage XML file import
replication = Replication(
    source='MY_FILE_STORAGE',
    target='MY_TARGET_DB',
    defaults={
        'mode': Mode.FULL_REFRESH,
        'object': 'target_schema.{stream_file_folder}_{stream_file_name}',
        'source_options': SourceOptions(format=Format.XML)
    },
    streams={
        # No need to specify scheme://bucket
        'my_xml_folder/': {},
        'my_file.xml': {},
        # With full S3 path
        's3://my-bucket/my_xml_folder/': {},
        's3://my-bucket/my_file.xml': {},
        # Google Cloud Storage
        'gs://my-bucket/my_xml_folder/': {},
        'gs://my-bucket/my_file.xml': {}
    },
    env={
        'SLING_STREAM_URL_COLUMN': 'true',
        'SLING_THREADS': '3'
    }
)

replication.run()
```

{% endcode %}

</details>


# Custom SQL

Sling allows you to use custom DuckDB SQL statements to read from files, giving you more control over the data ingestion process. This is particularly useful when you need to perform transformations or filtering during the read operation.

## CLI Flags Examples

### Full Refresh Mode

In the example below, when we specify the source connection `aws_s3`, sling will auto-inject the necessary secrets for proper auth.

```bash
# Read CSV files with custom SQL
sling run \
  --src-conn aws_s3 \
  --src-stream "select * from read_csv('s3://my-bucket/data/*.csv') where amount > 1000" \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.large_transactions' \
  --mode full-refresh

# Read Parquet files with custom SQL and aggregation
sling run \
  --src-conn aws_s3 \
  --src-stream "select date_trunc('month', date) as month, sum(amount) as total 
                from read_parquet('gs://my-bucket/data/*.parquet')
                group by 1" \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.monthly_totals' \
  --mode full-refresh
```

### Incremental Mode

```bash
# Incremental load using timestamp column
sling run \
  --src-stream "select * from read_csv('s3://my-bucket/data/*.csv') 
                where {incremental_where_cond}" \
  --src-options '{"sql": true}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.transactions' \
  --mode incremental \
  --primary-key id \
  --update-key created_at
```

## Replication Configuration

You can also use DuckDB SQL in your replication configuration:

```yaml
source: AWS_S3
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  # Using SQL to read and transform CSV data
  daily_sales_summary:
    object: analytics.daily_sales_summary
    sql: |
      select date, 
              sum(case when type='sale' then amount else 0 end) as sales,
              sum(case when type='refund' then amount else 0 end) as refunds
      from read_csv('s3://my-bucket/transactions/*.csv')
      group by date

  # Incremental load with custom SQL
  events:
    object: analytics.events
    sql: |
      select * from read_parquet('s3://my-bucket/events/*.parquet')
      where event_timestamp > coalesce({incremental_value}, '2001-01-01')
    mode: incremental
    update_key: event_timestamp

  # Join multiple files
  enriched_transactions:
    object: analytics.enriched_transactions
    sql: |
      select t.*, c.category 
      from read_csv('s3://my-bucket/transactions.csv') t
      left join read_parquet('s3://my-bucket/categories.parquet') c
        on t.category_id = c.id
```

***

## Using Python

Using custom SQL to read from files with the Python API:

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode
import os

# Set environment variables
os.environ['AWS_S3'] = '...'
os.environ['MY_TARGET_DB'] = '...'

# Full Refresh Mode with custom SQL
replication = Replication(
    source='AWS_S3',
    target='MY_TARGET_DB',
    defaults={'mode': Mode.FULL_REFRESH},
    streams={
        # Using SQL to read and transform CSV data
        'daily_sales_summary': ReplicationStream(
            object='analytics.daily_sales_summary',
            sql="""
                select date, 
                       sum(case when type='sale' then amount else 0 end) as sales,
                       sum(case when type='refund' then amount else 0 end) as refunds
                from read_csv('s3://my-bucket/transactions/*.csv')
                group by date
            """
        ),

        # Incremental load with custom SQL
        'events': ReplicationStream(
            object='analytics.events',
            sql="""
                select * from read_parquet('s3://my-bucket/events/*.parquet')
                where event_timestamp > coalesce({incremental_value}, '2001-01-01')
            """,
            mode=Mode.INCREMENTAL,
            update_key='event_timestamp'
        ),

        # Join multiple files
        'enriched_transactions': ReplicationStream(
            object='analytics.enriched_transactions',
            sql="""
                select t.*, c.category 
                from read_csv('s3://my-bucket/transactions.csv') t
                left join read_parquet('s3://my-bucket/categories.parquet') c
                  on t.category_id = c.id
            """
        )
    }
)

# Run the replication
replication.run()

# Simple custom SQL example
replication = Replication(
    source='AWS_S3',
    target='MY_TARGET_DB',
    streams={
        "my_stream": ReplicationStream(
            object='target_schema.large_transactions',
            sql="select * from read_csv('s3://my-bucket/data/*.csv') where amount > 1000",
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()

# Incremental load with source options
replication = Replication(
    source='LOCAL',  # When source is LOCAL, need to use source_options
    target='MY_TARGET_DB',
    streams={
        'my_stream': ReplicationStream(
            sql="""select * from read_csv('s3://my-bucket/data/*.csv') 
                     where {incremental_where_cond}""",
            source_options=SourceOptions(sql=True),
            object='target_schema.transactions',
            mode=Mode.INCREMENTAL,
            primary_key='id',
            update_key='created_at'
        )
    }
)

replication.run()
```

{% endcode %}

## Features

* **SQL Functions**: Access to DuckDB's rich SQL function library
* **File Format Support**: Works with CSV, Parquet, JSON, and other formats supported by DuckDB
* **Aggregations**: Perform aggregations and transformations during read
* **Joins**: Join data from multiple files
* **Filtering**: Apply filters to reduce data transfer
* **Type Casting**: Use SQL CAST functions for type conversions

## Notes

1. Sling with auto-download the duckdb binary into the Sling [home directory](/sling-cli/environment#credentials-location). You can specify the desired duckDB version with env var `DUCKDB_VERSION`.
2. Use DuckDB's `read_*` functions to specify input files
3. For incremental loads, use the placeholder variables such as `{incremental_where_cond}` and `{incremental_value}`. See [here](/concepts/replication/modes) for more details.
4. File paths support wildcards (`*`) for matching multiple files. See [Reading Multiple Files](https://duckdb.org/docs/data/multiple_files/overview.html).
5. Cloud storage paths (`s3://`, `gs://`, etc.) are supported with proper credentials. Make sure to specify the respective source connection, sling will auto-inject the needed [secrets](https://duckdb.org/docs/configuration/secrets_manager) before running the query. If you are facing issues with auth not working, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# Incremental

Examples of using Sling to load data from storage systems to databases

## Using the File timestamp

The current approach for incrementally loading files into a database is using the file timestamp.

Here is an example replication, incrementally loading files from an S3 bucket into a Postgres database:

```yaml
source: aws_s3
target: postgres

defaults:
  object: 'target_schema.{stream_file_folder}_{stream_file_name}'
  mode: incremental
  update_key: _sling_loaded_at # <-- uses the _sling_loaded_at column in the target table
  columns:
    '*': string # cast all columns as string
  source_options:
    format: csv

streams:
  # no need to specify scheme://bucket
  "my_csv_folder/*.csv":          # individual streams for each file
  "my_csv_folder/":                     # single stream for whole folder
    object: 'target_schema.my_csv_data' # overwrite default object
  
  "my_csv_folder/prefix_*.csv":   
    object: 'target_schema.my_csv_data' # overwrite default object

env:
  SLING_LOADED_AT_COLUMN: timestamp
```

## Using SLING\_STATE

We can also provide a environment variable called `SLING_STATE`, which is a location where sling will store the respective incremental values. See [Global Variables](/sling-cli/variables#global-environment-variables) for more details. This allows you to modify the value directly if you want to change the incremental marker value manually.

Let us assume we have parquet files in the following format, at a daily level:

```
|- dumps/orders/2024/09/21/*.parquet
|- dumps/orders/2024/09/22/*.parquet
|- dumps/orders/2024/09/23/*.parquet
...
```

We can specify the stream string as `dumps/orders/{YYYY}/{MM}/{DD}/*.parquet`

```yaml
source: aws_s3
target: postgres

defaults:
  mode: incremental

streams:
  # no need to specify scheme://bucket
  "dumps/orders/{YYYY}/{MM}/{DD}/":
    id: orders   # id to run only this stream
    object: 'target_schema.orders'
    source_options:
      format: parquet

env:
  # uses the `path/to/folder` in the same AWS_S3 connection
  SLING_STATE: AWS_S3/path/to/folder
```

### Backfilling

Best Practice for first load is to backfill the period range of interest. Sling will ingest all data found according to the stream path provided. Upon completion, the incremental date value will be set to the last date in the backfill range. Therefore it is necessary to run backfills in ascending order (e.g `2021-01-01,2021-12-31`, `2022-01-01,2022-12-31`, etc).

```bash
# backfill from 2021-01-01 to 2022-01-01
# incremental state value will be set to 2022-01-01
sling run -d -r replication.yaml --mode backfill \
  --range 2021-01-01,2022-01-01 \
  --streams "orders" # only run "orders" stream
```

### Incremental

Run normally. Sling will process the next incremental data value

```shell
sling run -r replication.yaml -d
```


# Multiple Files & Cross-Bucket

Examples of loading multiple files and cross-bucket paths in Sling

This guide demonstrates advanced techniques for loading files from multiple paths, including files spanning multiple cloud storage buckets or containers using a single connection.

## Loading Multiple Paths in a Single Stream

Sling allows you to specify multiple file paths for a single stream using the `files` key. This is useful when you want to combine data from several files into one target table.

### Basic Multiple Files

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: aws_s3
target: postgres

defaults:
  mode: full-refresh

streams:
  combined_data:
    files:
      - data/customers_2024.csv       # Single file
      - data/customers_2023.csv       # Single file
      - data/archive/                 # All files in folder
      - data/legacy/*.csv             # Wildcard pattern
    object: public.all_customers
    source_options:
      format: csv
      header: true
```

{% endcode %}

### Cross-Bucket File Loading

A powerful feature of Sling is the ability to access files from multiple buckets using a single S3 connection. By specifying full `s3://` URIs in the `files` array, you can pull data from different buckets (as long as your credentials have access to all of them).

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: aws_s3
target: postgres

defaults:
  mode: full-refresh
  target_options:
    adjust_column_type: true

streams:
  # Method 1: Using full S3 URIs as stream names
  's3://bucket-west/data/sales.csv':
    object: public.sales_west

  's3://bucket-east/data/sales.csv':
    object: public.sales_east

  # Method 2: Combining files from multiple buckets into one table
  combined_sales:
    files:
      - s3://bucket-west/data/sales.csv           # Single file
      - s3://bucket-east/data/                    # All files in folder
      - s3://bucket-archive/historical/           # All files in folder
      - s3://bucket-legacy/sales/*.csv            # Wildcard pattern
    object: public.all_sales

env:
  SLING_STREAM_URL_COLUMN: true  # Track which file each row came from
```

{% endcode %}

{% hint style="info" %}
**Cross-Bucket Access**: When using full URIs like `s3://bucket-name/path`, make sure your AWS credentials (configured in the connection) have read access to all the buckets referenced. The same principle applies to GCS (`gs://`) and Azure (`https://`) storage.
{% endhint %}

### Cross-Container Loading for Other Cloud Providers

The same technique works with Google Cloud Storage and Azure Blob Storage:

{% tabs %}
{% tab title="Google Cloud Storage" %}

```yaml
source: gcs_conn
target: bigquery

streams:
  # From multiple GCS buckets
  combined_logs:
    files:
      - gs://prod-bucket/logs/2024/           # All files in folder
      - gs://staging-bucket/logs/             # All files in folder
      - gs://dev-bucket/logs/app.json         # Single file
      - gs://archive-bucket/logs/**/*.json    # Recursive wildcard
    object: dataset.all_logs
    source_options:
      format: json
      flatten: true
```

{% endtab %}

{% tab title="Azure Blob Storage" %}

```yaml
source: azure_conn
target: snowflake

streams:
  # From multiple Azure containers
  combined_data:
    files:
      - https://account.blob.core.windows.net/container1/data/        # All files in folder
      - https://account.blob.core.windows.net/container2/exports/     # All files in folder
      - https://account.blob.core.windows.net/archive/2024/*.parquet  # Wildcard pattern
    object: schema.all_data
```

{% endtab %}
{% endtabs %}

## Combining Folders, Wildcards, and Files

You can mix and match folder paths, wildcard patterns, and individual files for maximum flexibility:

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: aws_s3
target: snowflake

defaults:
  mode: full-refresh

streams:
  # Load from multiple sources into one table
  all_regions:
    files:
      - s3://bucket/us-west/                  # All files in folder
      - s3://bucket/us-east/daily/            # All files in subfolder
      - s3://bucket/eu-central/*.csv          # Wildcard pattern
      - s3://bucket/apac/**/*.csv             # Recursive wildcard
      - s3://bucket/legacy/important.csv      # Single specific file
    object: warehouse.all_regions
    single: true  # Treat all matching files as one stream
    source_options:
      format: csv

env:
  SLING_STREAM_URL_COLUMN: true
  SLING_THREADS: 5
```

{% endcode %}


# Reading Excel

Examples of reading Excel files with various range options

Sling supports reading Excel files (`.xlsx` and macro-enabled `.xlsm`) with flexible range selection options. You can extract data from specific sheets, cell ranges, row ranges, or column ranges.

{% hint style="info" %}
Macro-enabled workbooks (`.xlsm`) are read the same way as `.xlsx` files — Sling extracts the sheet data and ignores any embedded VBA macros. All range, sheet, and source options below apply equally to `.xlsm` files.
{% endhint %}

## Range Syntax

The `sheet` source option supports several range formats:

| Format        | Example          | Description                              |
| ------------- | ---------------- | ---------------------------------------- |
| No sheet      | *(empty)*        | Uses first sheet in workbook             |
| Sheet only    | `Sheet1`         | Read entire sheet                        |
| Range only    | `!A:F`           | First sheet, columns A through F         |
| Cell range    | `Sheet1!A1:F100` | Standard cell range                      |
| Column range  | `Sheet1!A:F`     | All rows, columns A through F            |
| Row range     | `Sheet1!5:20`    | Rows 5-20, auto-detect columns with data |
| Row start     | `Sheet1!5:`      | From row 5 to end, auto-detect columns   |
| Partial range | `Sheet1!A5:F`    | From A5 to column F, last row            |

{% hint style="info" %}
When no sheet name is provided (or when using `!` prefix without a sheet name), Sling automatically uses the first sheet in the workbook.
{% endhint %}

## Excel to Database

### Using CLI Flags

{% code title="sling.sh" overflow="wrap" %}

```bash
# Read first sheet (no sheet option = uses first sheet)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.excel_data' \
  --mode full-refresh

# Read first sheet with column range (! prefix without sheet name)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "!A:F" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.excel_data' \
  --mode full-refresh

# Read specific sheet
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_data' \
  --mode full-refresh

# Read specific cell range (A1 to F100)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales!A1:F100" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_data' \
  --mode full-refresh

# Read column range (all rows, columns A through F)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales!A:F" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_data' \
  --mode full-refresh

# Read row range (rows 5-20, auto-detect columns)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales!5:20" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_subset' \
  --mode full-refresh

# Read from row 5 to end (auto-detect columns and last row)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales!5:" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_from_row5' \
  --mode full-refresh

# Read partial range (from A5 to column F, extends to last row)
$ sling run --src-stream 'file:///path/to/data.xlsx' \
  --src-options '{ sheet: "Sales!A5:F" }' \
  --tgt-conn MY_POSTGRES \
  --tgt-object 'public.sales_data' \
  --mode full-refresh
```

{% endcode %}

### Using Replication

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: LOCAL
target: MY_POSTGRES

defaults:
  mode: full-refresh

streams:
  # Read entire first sheet
  "file:///data/reports/quarterly.xlsx":
    object: public.quarterly_report

  # Read specific sheet by name
  "file:///data/reports/annual.xlsx":
    object: public.annual_summary
    source_options:
      sheet: "Summary"

  # Read specific cell range (useful for reports with headers/footers to skip)
  "file:///data/reports/financial.xlsx":
    object: public.financial_data
    source_options:
      sheet: "Q4 Results!B3:H50"

  # Read column range - all rows but only columns A through E
  "file:///data/reports/inventory.xlsx":
    object: public.inventory
    source_options:
      sheet: "Stock!A:E"

  # Read row range - rows 10-500, auto-detects columns with data
  # Useful when data starts after header rows or intro text
  "file:///data/reports/transactions.xlsx":
    object: public.transactions
    source_options:
      sheet: "Data!10:500"

  # Partial range - from row 5 to end, columns A through G
  # Great for skipping title rows while reading to the end
  "file:///data/reports/employees.xlsx":
    object: public.employees
    source_options:
      sheet: "Directory!A5:G"

  # Cast specific columns to ensure correct types
  "file:///data/reports/sales.xlsx":
    object: public.sales
    source_options:
      sheet: "Monthly!A:J"
    columns:
      sale_date: date
      amount: decimal
      quantity: integer

  # Multiple sheets from same file using wildcards
  "file:///data/reports/*.xlsx":
    object: public.{stream_file_name}
    source_options:
      sheet: "Sheet1!A:Z"

env:
  SLING_SAMPLE_SIZE: 2000  # Increase sample size for better type inference
  SLING_ROW_NUM_COLUMN: true  # Add row number column for tracking
```

{% endcode %}

### Using Python

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, Mode
import os

os.environ['MY_POSTGRES'] = '...'

# Example 1: Basic Excel to database with different range types
replication = Replication(
    source='LOCAL',
    target='MY_POSTGRES',
    defaults={'mode': Mode.FULL_REFRESH},
    streams={
        # Entire sheet
        'file:///data/reports/data.xlsx': ReplicationStream(
            object='public.full_data',
            source_options=SourceOptions(sheet='Sheet1')
        ),

        # Specific cell range
        'file:///data/reports/financial.xlsx': ReplicationStream(
            object='public.financial',
            source_options=SourceOptions(sheet='Q4!B2:K100')
        ),

        # Column range (all rows)
        'file:///data/reports/inventory.xlsx': ReplicationStream(
            object='public.inventory',
            source_options=SourceOptions(sheet='Stock!A:F')
        ),

        # Row range (auto-detect columns)
        'file:///data/reports/logs.xlsx': ReplicationStream(
            object='public.logs',
            source_options=SourceOptions(sheet='Activity!15:1000')
        ),

        # Partial range (skip header rows)
        'file:///data/reports/contacts.xlsx': ReplicationStream(
            object='public.contacts',
            source_options=SourceOptions(sheet='People!A3:H')
        )
    },
    env={
        'SLING_SAMPLE_SIZE': '2000',
        'SLING_ROW_NUM_COLUMN': 'true'
    }
)

replication.run()
```

{% endcode %}

## Excel to File (Parquet, CSV, JSON)

Convert Excel files to other formats like Parquet for efficient storage and querying.

### Using CLI Flags

{% code title="sling.sh" overflow="wrap" %}

```bash
# Excel to Parquet
$ sling run --src-stream 'file:///data/report.xlsx' \
  --src-options '{ sheet: "Data!A:M" }' \
  --tgt-conn LOCAL \
  --tgt-object 'file:///output/report.parquet'

# Excel to CSV
$ sling run --src-stream 'file:///data/report.xlsx' \
  --src-options '{ sheet: "Summary!5:100" }' \
  --tgt-conn LOCAL \
  --tgt-object 'file:///output/summary.csv'

# Excel to JSON Lines
$ sling run --src-stream 'file:///data/report.xlsx' \
  --src-options '{ sheet: "Records" }' \
  --tgt-conn LOCAL \
  --tgt-object 'file:///output/records.jsonl'

# Excel to cloud storage (S3)
$ sling run --src-stream 'file:///data/report.xlsx' \
  --src-options '{ sheet: "Data!A2:Z" }' \
  --tgt-conn MY_S3 \
  --tgt-object 's3://my-bucket/exports/report.parquet'
```

{% endcode %}

### Using Replication

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: LOCAL
target: MY_S3

defaults:
  mode: full-refresh

streams:
  # Convert Excel sheets to Parquet files
  "file:///data/reports/sales_2024.xlsx":
    object: "s3://data-lake/sales/2024/data.parquet"
    source_options:
      sheet: "Transactions!A:P"

  # Extract specific rows to CSV
  "file:///data/reports/quarterly.xlsx":
    object: "s3://exports/quarterly_summary.csv"
    source_options:
      sheet: "Summary!2:50"
    target_options:
      format: csv

  # Multiple Excel files to Parquet
  "file:///data/reports/monthly_*.xlsx":
    object: "s3://data-lake/monthly/{stream_file_name}.parquet"
    source_options:
      sheet: "Data!A:K"
```

{% endcode %}

### Using Python

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, TargetOptions, Mode

# Convert Excel to Parquet on S3
replication = Replication(
    source='LOCAL',
    target='MY_S3',
    defaults={'mode': Mode.FULL_REFRESH},
    streams={
        # Excel to Parquet
        'file:///data/large_dataset.xlsx': ReplicationStream(
            object='s3://analytics/datasets/data.parquet',
            source_options=SourceOptions(sheet='Raw!A:AA')
        ),

        # Excel to partitioned Parquet
        'file:///data/events.xlsx': ReplicationStream(
            object='s3://analytics/events/',
            source_options=SourceOptions(sheet='Events!5:10000'),
            target_options=TargetOptions(
                format='parquet',
                file_max_rows=100000
            )
        )
    }
)

replication.run()
```

{% endcode %}

## Advanced Options

### Password-Protected Excel Files

{% code title="replication.yaml" %}

```yaml
source: LOCAL
target: MY_POSTGRES

streams:
  "file:///data/secure_report.xlsx":
    object: public.secure_data
    source_options:
      sheet: "Confidential!A:F"
      password: "${EXCEL_PASSWORD}"  # Use environment variable
```

{% endcode %}

### Custom Date Patterns

{% code title="replication.yaml" %}

```yaml
source: LOCAL
target: MY_POSTGRES

streams:
  "file:///data/international_dates.xlsx":
    object: public.date_data
    source_options:
      sheet: "Dates!A:D"
      short_date_pattern: "DD/MM/YYYY"
      long_date_pattern: "DD MMMM YYYY"
      long_time_pattern: "HH:mm:ss"
```

{% endcode %}

### Type Casting

{% code title="replication.yaml" %}

```yaml
source: LOCAL
target: MY_POSTGRES

streams:
  "file:///data/mixed_types.xlsx":
    object: public.typed_data
    source_options:
      sheet: "Data!A:J"
    columns:
      id: integer
      price: decimal
      created_at: timestamp
      is_active: bool
      metadata: json
      "*": string  # Cast remaining columns to string
```

{% endcode %}

## Environment Variables

| Variable                  | Description                                       | Default |
| ------------------------- | ------------------------------------------------- | ------- |
| `SLING_SAMPLE_SIZE`       | Number of rows to sample for type inference       | 900     |
| `SLING_ROW_NUM_COLUMN`    | Add `_sling_row_num` column with Excel row number | false   |
| `SLING_STREAM_URL_COLUMN` | Add `_sling_stream_url` column with file path     | false   |

## Tips

1. **Skip Header Rows**: Use row ranges like `5:1000` or partial ranges like `A5:F` to skip title rows or headers that aren't your column names.
2. **Auto-Detect Columns**: Row-only ranges (`10:500`) automatically detect the rightmost column with data, useful for sheets where column count varies.
3. **Memory Efficiency**: Sling streams data row-by-row, so large Excel files are processed efficiently without loading the entire file into memory.
4. **Type Inference**: Increase `SLING_SAMPLE_SIZE` for better type detection on large files with varied data.
5. **Multiple Sheets**: Process multiple sheets by creating separate stream entries for each sheet in your replication config.


# Database to Database

Examples of using Sling to load data from one database to another

We first need to make sure our connections are available in our environment. See [Environment](https://github.com/slingdata-io/sling-docs/blob/master/environment.md) and [Database Connections](/connections/database-connections) for more details.

{% tabs %}
{% tab title="Linux / Mac" %}

```bash
export MY_SOURCE_DB='...'
export MY_TARGET_DB='...'

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_TARGET_DB  | DB - Snowflake   | env variable    |
+---------------+------------------+-----------------+
```

{% endtab %}

{% tab title="Windows" %}

```powershell
# using windows Powershell
$env:MY_SOURCE_DB = '...'
$env:MY_TARGET_DB = '...'

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_TARGET_DB  | DB - Snowflake   | env variable    |
+---------------+------------------+-----------------+
```

{% endtab %}
{% endtabs %}

<details>

<summary>Database ⇨ Database (Full Refresh)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  object: '{target_schema}.{stream_schema}_{stream_table}'
  mode: full-refresh

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern

  source_schema.another_table:

  # chunking into 6 equal sized streams
  source_schema.large_table:
    primary_key: id
    update_key: updated_at
    source_options:
      chunk_count: 6

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="database\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode

# Single stream example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()

# Multiple streams example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        object='{target_schema}.{stream_schema}_{stream_table}',
        mode=Mode.FULL_REFRESH
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table'  # override default object pattern
        ),
        'source_schema.another_table': {},
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Custom SQL)</summary>

See also [Custom SQL Examples](/examples/database-to-database/custom-sql).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'select * from my_schema.my_table where col1 is not null' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

# we can also read from a SQL file (/path/to/query.sql)
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream file:///path/to/query.sql \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  source_schema.source_table.1:
    sql: |
      select *
      from my_schema.my_table
      where col1 is not null
    object: target_schema.target_table

  source_schema.source_table.2:
    sql: file:///path/to/query.sql
    object: target_schema.target_table
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode

# Single stream example with inline SQL
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table.1': ReplicationStream(
            sql="""
                select *
                from my_schema.my_table
                where col1 is not null
            """,
            object='target_schema.target_table',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()

# Multiple streams example with SQL file reference
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.FULL_REFRESH
    ),
    streams={
        'source_schema.source_table.1': ReplicationStream(
            sql="""
                select *
                from my_schema.my_table
                where col1 is not null
            """,
            object='target_schema.target_table'
        ),
        'source_schema.source_table.2': ReplicationStream(
            sql='file:///path/to/query.sql',
            object='target_schema.target_table'
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Incremental / Backfill)</summary>

See also [Incremental](/examples/database-to-database/incremental) and [Backfill](/examples/database-to-database/backfill) examples.

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
# limit to 10M records at a time. Will be sorted by update_key ASC.
# just loop command until all data is transferred / caught up.
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --primary-key 'id' \
  --update-key 'last_modified_dt' \
  --mode incremental \
  --limit 1000000 -d

# Backfill specific date range
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --primary-key 'id' \
  --update-key 'last_modified_dt' \
  --mode backfill \
  --range '2021-01-01,2021-02-01'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  object: '{target_schema}.{stream_schema}_{stream_table}'
  primary_key: [id]
  update_key: last_modified_dt
  source_options:
    limit: 10000000 # limit to 10M records at a time

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern
    update_key: updated_at # override default update key

  # backfill
  source_schema.backfill_table:
    mode: backfill
    object: target_schema.backfill_table
    primary_key: [some_id]
    update_key: updated_at # override default update key
    source_options:
      range: 2021-01-01,2021-02-01 # specific date range
      chunk_size: 7d               # 7-day stream chunking/splitting

  source_schema.another_table:
    target_options:
      delete_missing: soft  # track deletes from source table

  # use delete_insert strategy for MySQL (no native MERGE)
  source_schema.mysql_table:
    object: target_schema.mysql_table
    target_options:
      merge_strategy: delete_insert

  # chunking with count-based approach
  source_schema.large_incremental_table:
    object: target_schema.large_incremental_table
    primary_key: [id]
    update_key: updated_at
    source_options:
      chunk_count: 8  # split into 8 equal chunks for parallel processing

  # chunking with time-range approach
  source_schema.events_table:
    object: target_schema.events_table
    primary_key: [event_id]
    update_key: event_timestamp
    source_options:
      chunk_size: 1d  # process 1 day at a time

env:
  SLING_THREADS: 4 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, TargetOptions, Mode, MergeStrategy

# Incremental load with limit
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.INCREMENTAL,
        object='{target_schema}.{stream_schema}_{stream_table}',
        primary_key=['id'],
        update_key='last_modified_dt',
        source_options=SourceOptions(
            limit=10000000  # limit to 10M records at a time
        )
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',  # override default object pattern
            update_key='updated_at'  # override default update key
        ),
        # backfill example
        'source_schema.backfill_table': ReplicationStream(
            mode=Mode.BACKFILL,
            object='target_schema.backfill_table',
            primary_key=['some_id'],
            update_key='updated_at',
            source_options=SourceOptions(
                range='2021-01-01,2021-02-01',  # specific date range
                chunk_size='7d'  # 7-day stream chunking/splitting
            )
        ),
        'source_schema.another_table': ReplicationStream(
            target_options=TargetOptions(
                delete_missing='soft'  # track deletes from source table
            )
        ),
        # use delete_insert strategy for MySQL (no native MERGE)
        'source_schema.mysql_table': ReplicationStream(
            object='target_schema.mysql_table',
            target_options=TargetOptions(
                merge_strategy=MergeStrategy.DELETE_INSERT
            )
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Incremental - New Data Upsert)</summary>

See also [Incremental Examples](/examples/database-to-database/incremental).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode incremental \
  --primary-key 'id' \
  --update-key 'last_modified_dt'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  object: '{target_schema}.{stream_schema}_{stream_table}'
  primary_key: [id]
  update_key: last_modified_dt

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern
    update_key: updated_at # override default update key

  source_schema.another_table:
  
  source_schema.some_table:
    target_options:
      delete_missing: soft  # track deletes from source table

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true
env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, TargetOptions, Mode

# Single stream example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.INCREMENTAL,
            primary_key=['id'],
            update_key='last_modified_dt'
        )
    }
)

replication.run()

# Multiple streams example with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.INCREMENTAL,
        object='{target_schema}.{stream_schema}_{stream_table}',
        primary_key=['id'],
        update_key='last_modified_dt'
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',  # override default object pattern
            update_key='updated_at'  # override default update key
        ),
        'source_schema.another_table': {},
        'source_schema.some_table': ReplicationStream(
            target_options=TargetOptions(
                delete_missing='soft'  # track deletes from source table
            )
        ),
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Incremental - Full Data Upsert)</summary>

See also [Incremental Examples](/examples/database-to-database/incremental).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode incremental \
  --primary-key 'id'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  object: '{target_schema}.{stream_schema}_{stream_table}'
  mode: incremental
  primary_key: [id]

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern
    primary_key: [ col1, col2 ] # override default primary_key

  source_schema.another_table:
  
  source_schema.some_table:
    target_options:
      delete_missing: soft  # track deletes from source table

  # append-only audit table (insert strategy - never update existing records)
  source_schema.audit_log:
    object: target_schema.audit_log
    target_options:
      merge_strategy: insert

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, SourceOptions, TargetOptions, Mode, MergeStrategy

# Single stream example (full data upsert - no update_key)
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.INCREMENTAL,
            primary_key=['id']
        )
    }
)

replication.run()

# Multiple streams example with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        object='{target_schema}.{stream_schema}_{stream_table}',
        mode=Mode.INCREMENTAL,
        primary_key=['id']
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',  # override default object pattern
            primary_key=['col1', 'col2']  # override default primary_key
        ),
        'source_schema.another_table': {},
        'source_schema.some_table': ReplicationStream(
            target_options=TargetOptions(
                delete_missing='soft'  # track deletes from source table
            )
        ),
        # append-only audit table (insert strategy - never update existing records)
        'source_schema.audit_log': ReplicationStream(
            object='target_schema.audit_log',
            target_options=TargetOptions(
                merge_strategy=MergeStrategy.INSERT
            )
        ),
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Incremental - Append Only)</summary>

See also [Incremental Examples](/examples/database-to-database/incremental).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode incremental \
  --update-key 'created_dt'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  object: '{target_schema}.{stream_schema}_{stream_table}'
  update_key: created_dt

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern
    update_key: created_at  # override default update_key

  source_schema.another_table:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true
env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode

# Single stream example (append only)
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.INCREMENTAL,
            update_key='created_dt'
        )
    }
)

replication.run()

# Multiple streams example with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.INCREMENTAL,
        object='{target_schema}.{stream_schema}_{stream_table}',
        update_key='created_dt'
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',  # override default object pattern
            update_key='created_at'  # override default update_key
        ),
        'source_schema.another_table': {},
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Truncate)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode truncate
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: truncate
  object: '{target_schema}.{stream_schema}_{stream_table}'

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern

  source_schema.another_table:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true
    
env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

# Single stream example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode='truncate'
        )
    }
)

replication.run()

# Multiple streams example with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.TRUNCATE,
        object='{target_schema}.{stream_schema}_{stream_table}'
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table'  # override default object pattern
        ),
        'source_schema.another_table': {},
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Database (Snapshot)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode snapshot
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: snapshot
  object: '{target_schema}.{stream_schema}_{stream_table}'

streams:
  source_schema.source_table:
    object: target_schema.target_table # override default object pattern

  source_schema.another_table:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
  my_schema.forbidden_table:
    disabled: true
    
env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

# Single stream example
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table',
            mode=Mode.SNAPSHOT
        )
    }
)

replication.run()

# Multiple streams example with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_TARGET_DB',
    defaults=ReplicationStream(
        mode=Mode.SNAPSHOT,
        object='{target_schema}.{stream_schema}_{stream_table}'
    ),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='target_schema.target_table'  # override default object pattern
        ),
        'source_schema.another_table': {},
        # all tables in schema, except "forbidden_table"
        'my_schema.*': {},
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>


# Parallel Chunking ⚡

Chunking is a feature in Sling that breaks down large data transfers into smaller, manageable parts. This is particularly useful for optimizing performance, managing resources, and enabling parallel processing during incremental and backfill operations. Chunking is available with Sling [CLI Pro](/sling-cli/cli-pro) or on a [Platform](/sling-platform/platform) plan.

## Supported Chunk Types

Sling supports several chunking strategies via the `source_options.chunk_size` or `source_options.chunk_count` parameters:

**Time-based chunks**:

* Hours: e.g., `6h`
* Days: e.g., `7d`
* Weeks: e.g., `1w`
* Months: e.g., `1m`
* Years: e.g., `1y`

**Numeric chunks**:

* Integer ranges: e.g., `1000` for chunks of 1000 records

**Count-based chunks** (v1.4.14+):

* Specific number of chunks: e.g., `5` to split data into 5 equal parts

**Expression-based chunks** (v1.4.14+):

* Custom expressions: e.g., `mod(abs(hashtext(column_name)), {chunk_count})`

Each chunk is processed independently, allowing for parallel execution when combined with [`SLING_THREADS`](/sling-cli/cli-pro#parallel-processing-and-retries).

## Chunking in Different Modes

Chunking works across all replication modes (full-refresh, truncate, incremental, and backfill), helping process large datasets by breaking them into smaller batches. This is useful for memory management, progress tracking, and reducing source database load.

### Time-Range Chunking

Time-range chunking splits data based on date/time columns across different modes:

```yaml
source: postgres
target: snowflake

defaults:
  primary_key: id
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.orders:
    mode: full-refresh
    update_key: created_at
    source_options:
      chunk_size: 1m  # Process 1 month at a time
  
  public.events:
    mode: truncate
    update_key: event_timestamp
    source_options:
      chunk_size: 7d  # Process 1 week at a time
  
  public.logs:
    mode: incremental
    update_key: updated_at
    source_options:
      chunk_size: 1d  # Process 1 day at a time

env:
  SLING_THREADS: 4  # Enable parallel processing
```

### Numeric-Range Chunking

Numeric-range chunking splits data based on numeric columns like IDs:

```yaml
source: postgres
target: snowflake

defaults:
  primary_key: id
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.products:
    mode: full-refresh
    update_key: id
    source_options:
      chunk_size: 1000  # Process 1000 records per chunk
  
  public.customers:
    mode: truncate
    update_key: customer_id
    source_options:
      chunk_size: 500   # Process 500 customers per chunk

env:
  SLING_THREADS: 4  # Enable parallel processing
```

### Count-based Chunking

Count-based chunking splits data into a specific number of equal chunks:

```yaml
source: postgres
target: snowflake

defaults:
  primary_key: id
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.large_table:
    mode: full-refresh
    update_key: updated_at
    source_options:
      chunk_count: 10  # Split the dataset into 10 equal chunks
  
  public.historical_data:
    mode: incremental
    update_key: created_at
    source_options:
      chunk_count: 5   # Split into 5 chunks for processing

env:
  SLING_THREADS: 4  # Enable parallel processing
```

### Chunking by Expression

Expression-based chunking allows you to define custom SQL expressions to distribute data across chunks using the `chunk_expr` parameter. This works across all modes and is particularly useful for:

* Hash-based distribution for even data splitting
* Custom partitioning logic based on specific columns
* Complex expressions that don't rely on sequential values
* No update key needed

```yaml
source: postgres
target: snowflake

defaults:
  primary_key: id
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.customers:
    mode: full-refresh
    source_options:
      chunk_expr: mod(abs(hashtext(coalesce(first_name, ''))), {chunk_count})
      chunk_count: 4  # Creates 4 chunks with hash-based distribution
  
  public.orders:
    mode: truncate
    source_options:
      chunk_expr: mod(customer_id, {chunk_count})
      chunk_count: 6  # Creates 6 chunks based on customer_id modulo

  public.events:
    mode: incremental
    update_key: created_at
    source_options:
      chunk_expr: case when event_type = 'premium' then 0 else mod(user_id, {chunk_count} - 1) + 1 end
      chunk_count: 5  # Premium events in chunk 0, others distributed in chunks 1-4
  
  public.products:
    mode: full-refresh
    source_options:
      chunk_expr: mod(abs(hashtext(category || product_name)), {chunk_count})
      chunk_count: 3  # Hash-based on category + product name

env:
  SLING_THREADS: 4  # Enable parallel processing
```

### Mixed Chunking Strategies

```yaml
source: postgres
target: oracle

defaults:
  mode: incremental
  primary_key: id
  object: oracle.{stream_table_lower}
  target_options:
    use_bulk: false

streams:
  public.sales_data:
    update_key: sale_date
    source_options:
      chunk_size: 1m  # Monthly chunks
  
  public.user_activities:
    update_key: activity_id
    source_options:
      chunk_size: 50000  # 50k records per chunk
  
  public.dynamic_data:
    update_key: last_modified
    source_options:
      chunk_count: 8  # Split into 8 equal parts

env:
  SLING_THREADS: 4  # Enable parallel processing
```

For more on incremental mode basics, see [incremental.md](/examples/database-to-database/incremental).

## Chunking in Backfill Mode

Backfill mode with chunking allows loading historical data in smaller ranges, optimizing for large datasets.

```yaml
source: postgres
target: oracle

defaults:
  mode: backfill
  object: oracle.{stream_table_lower}
  primary_key: [id]

streams:
  public.orders_mariadb_pg:
    update_key: update_dt
    source_options:
      range: '2018-11-01,2018-12-01'
      chunk_size: 10d  # Process in 10-day chunks

  public.orders_sqlserver_pg:
    update_key: date
    source_options:
      range: '2019-01-01,2019-06-01'
      chunk_size: 2m   # Process in 2-month chunks

  public.orders_snowflake_pg:
    update_key: id  # same as primary key
    source_options:
      range: '1,800'
      chunk_size: 200  # Process in chunks of 200 IDs

  public.large_table_pg:
    update_key: created_at
    source_options:
      range: '2023-01-01,2023-12-31'
      chunk_count: 6  # Split into 6 equal time-based chunks

  public.user_data_pg:
    update_key: user_id
    source_options:
      range: '1000,50000'
      chunk_count: 10  # Split into 10 equal numeric chunks

env:
  SLING_THREADS: 3  # Process 3 streams concurrently
```

Sling splits the specified range into smaller sub-ranges based on `chunk_size` or `chunk_count`. Each sub-range is processed as a separate stream.

For more on backfill mode, see [backfill.md](/examples/database-to-database/backfill).

## Chunking with Custom SQL

Combine custom SQL queries with chunking using the `{incremental_where_cond}` variable in your SQL.

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  primary_key: [id]
  update_key: updated_at

streams:
  my_schema.large_orders:
    sql: |
      select 
        o.id,
        o.order_date,
        o.customer_id,
        o.status,
        o.updated_at,
        c.name as customer_name
      from my_schema.orders o
      join my_schema.customers c on o.customer_id = c.id
      where o.finalized and ({incremental_where_cond})
      order by o.updated_at asc
    object: target_schema.enriched_orders_chunked
    source_options:
      chunk_size: 1m  # Process in monthly chunks

  my_schema.historical_events:
    sql: |
      select *
      from my_schema.events
      where type = 'init' and ({incremental_where_cond})
    object: target_schema.events_chunked
    update_key: event_timestamp
    source_options:
      chunk_count: 4  # Split into 4 equal chunks

  my_schema.user_activities:
    sql: |
      select *
      from my_schema.user_activities
      where {incremental_where_cond}
    object: target_schema.activities_chunked
    update_key: activity_id
    source_options:
      chunk_size: 10000  # Process 10,000 records per chunk
      range: '1,100000'  # Optional range for numeric chunking

  public.complex_query:
    sql: |
      select 
        u.id,
        u.username,
        u.created_at,
        p.purchase_count
      from users u
      join (
        select user_id, count(*) as purchase_count
        from purchases
        group by user_id
      ) p on u.id = p.user_id
      where u.plan = 'Pro'
        and ( {incremental_where_cond} )
      order by u.created_at asc
    object: target_schema.user_purchase_summary
    update_key: created_at
    source_options:
      chunk_expr: mod(abs(hashtext(u.username)), {chunk_count})  # Process by modulo of hash
      chunk_count: 8  # Creates 8 chunks based on username hash

env:
  SLING_THREADS: 4  # Enable parallel processing of chunks
```

For more on custom SQL, see [custom-sql.md](/examples/database-to-database/custom-sql).

## Key Benefits and Notes

* **Parallelism**: Use `SLING_THREADS` for concurrent chunk processing
* **Error Recovery**: Chunks allow resuming from failures without restarting everything
* **Progress Tracking**: Better visibility for large operations
* **Requirements**: Works with `incremental` and `backfill` modes; needs appropriate `update_key` type
* **Best Practices**: Test chunk sizes for optimal performance; monitor resource usage

For complete replication mode details, see [Replication Modes](/concepts/replication/modes).


# Custom SQL

## Full Refresh with Custom SQL

**Using CLI Flags**

{% code overflow="wrap" %}

```bash
# Using inline SQL
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'select * from my_schema.my_table where status = "active"' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh

# Using SQL from a file
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream file:///path/to/query.sql \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode full-refresh
```

{% endcode %}

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  my_schema.my_table.1:
    sql: |
      select 
        id,
        first_name,
        last_name,
        email,
        status
      from my_schema.my_table 
      where status = 'active'
    object: target_schema.active_users

  my_schema.my_table.2:
    sql: file:///path/to/query.sql
    object: target_schema.custom_table
```

{% endcode %}

## Incremental with Custom SQL

**Using CLI Flags**

{% code overflow="wrap" %}

```bash
# Using inline SQL with incremental variables
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'select * from my_schema.my_table where updated_at > {incremental_where_cond}' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode incremental \
  --primary-key 'id' \
  --update-key 'updated_at'

# Using SQL file with incremental loading
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream file:///path/to/incremental_query.sql \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.target_table' \
  --mode incremental \
  --primary-key 'id' \
  --update-key 'updated_at'
```

{% endcode %}

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  primary_key: [id]
  update_key: updated_at

streams:
  my_schema.orders:
    sql: |
      select 
        o.id,
        o.order_date,
        o.customer_id,
        o.status,
        o.updated_at,
        c.name as customer_name
      from my_schema.orders o
      join my_schema.customers c on o.customer_id = c.id
      where {incremental_where_cond}
      order by o.updated_at asc
    object: target_schema.enriched_orders

  my_schema.transactions:
    sql: |
      with ranked_transactions as (
        select 
          *,
          row_number() over (partition by transaction_id order by modified_at desc) as rn
        from my_schema.transactions
        where modified_at > coalesce({incremental_value}, '2001-01-01')
      )
      select * from ranked_transactions 
      where rn = 1
    object: target_schema.latest_transactions
    update_key: modified_at  # override default update_key

  my_schema.daily_metrics:
    sql: file:///path/to/daily_metrics.sql
    object: target_schema.daily_metrics
    primary_key: [date, metric_id]
```

{% endcode %}

The examples above demonstrate:

* Using both inline SQL and SQL files
* Joining multiple tables in custom SQL
* Using incremental variables (`{incremental_value}` and `{incremental_where_cond}`). See [here](/concepts/replication/modes#incremental-mode-strategies) for details.
* Handling duplicates with window functions
* Overriding default primary keys and update keys

## Custom-SQL Chunking

Combine custom SQL queries with chunking using the `{incremental_where_cond}` variable in your SQL. See the [chunking documentation](/examples/database-to-database/chunking#chunking-with-custom-sql) for details.

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: incremental
  primary_key: [id]
  update_key: updated_at

streams:
  my_schema.large_orders:
    sql: |
      select 
        o.id,
        o.order_date,
        o.customer_id,
        o.status,
        o.updated_at,
        c.name as customer_name
      from my_schema.orders o
      join my_schema.customers c on o.customer_id = c.id
      where o.finalized and ({incremental_where_cond})
      order by o.updated_at asc
    object: target_schema.enriched_orders_chunked
    source_options:
      chunk_size: 1m  # Process in monthly chunks

  my_schema.historical_events:
    sql: |
      select *
      from my_schema.events
      where type = 'init' and ({incremental_where_cond})
    object: target_schema.events_chunked
    update_key: event_timestamp
    source_options:
      chunk_count: 4  # Split into 4 equal chunks

  my_schema.user_activities:
    sql: |
      select *
      from my_schema.user_activities
      where {incremental_where_cond}
    object: target_schema.activities_chunked
    update_key: activity_id
    source_options:
      chunk_size: 10000  # Process 10,000 records per chunk
      range: '1,100000'  # Optional range for numeric chunking

  public.complex_query:
    sql: |
      select 
        u.id,
        u.username,
        u.created_at,
        p.purchase_count
      from users u
      join (
        select user_id, count(*) as purchase_count
        from purchases
        group by user_id
      ) p on u.id = p.user_id
      where u.plan = 'Pro'
        and ( {incremental_where_cond} )
      order by u.created_at asc
    object: target_schema.user_purchase_summary
    update_key: created_at
    source_options:
      chunk_expr: mod(abs(hashtext(u.username)), {chunk_count})  # Process by modulo of hash
      chunk_count: 8  # Creates 8 chunks based on username hash

env:
  SLING_THREADS: 4  # Enable parallel processing of chunks
```


# Incremental

Examples of using Sling to incrementally load data from databases to databases

## New Data Upsert

This mode performs incremental loading by only processing new/updated records based on an update key. It requires both a primary key and update key.

```yaml
source: postgres
target: snowflake

defaults:
  mode: incremental
  primary_key: id  
  update_key: updated_at
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.orders:
    # Will only load records where updated_at is greater than the max value in target
  
  public.customers:
    primary_key: [customer_id]  # Override default primary key
    update_key: last_modified  # Override default update key
```

## Full Data Upsert

This mode performs incremental loading by processing the full source dataset and upserting records based on the primary key. No update key is required.

```yaml
source: postgres 
target: snowflake

defaults:
  mode: incremental
  primary_key: id
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.products:
    # Will load all records and upsert based on id
  
  public.categories:
    primary_key: [category_id, region]  # Composite primary key
```

## Append Only

This mode performs incremental loading by only appending new records based on an update key, without updating existing records. No primary key is required.

```yaml
source: postgres
target: snowflake

defaults:
  mode: incremental
  update_key: created_at
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.events:
    # Will only append records where created_at is greater than max value in target
  
  public.logs:
    update_key: timestamp  # Override default update key
```

## Custom SQL

This mode allows using custom SQL queries with incremental loading by using special variables that Sling will replace at runtime. See [here](/concepts/replication/modes#incremental-or-backfill-mode-with-custom-sql) from more details.

```yaml
source: postgres
target: snowflake

defaults:
  mode: incremental
  primary_key: id
  update_key: modified_at
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.orders:
    sql: |
      select *, coalesce(created_at, updated_at) as modified_at
      from public.orders 
      where {incremental_where_cond}
      order by modified_at asc
  
  public.customers:
    sql: |
      with ranked_customers as (
        select *
        from public.customers
        where modified_at > coalesce({incremental_value}, '2001-01-01')
      )
      select * from ranked_customers where rn = 1
```

## Incremental Chunking

In incremental mode, chunking helps process large datasets by breaking them into smaller batches based on the update key. This is useful for memory management, progress tracking, and reducing source database load. See the [chunking documentation](/examples/database-to-database/chunking) for details.

```yaml
source: postgres
target: snowflake

defaults:
  mode: incremental
  primary_key: id
  update_key: updated_at
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.orders:
    source_options:
      chunk_size: 1m  # Process 1 month at a time
  
  public.events:
    source_options:
      chunk_size: 7d  # Process 1 week at a time
  
  public.logs:
    source_options:
      chunk_size: 1d  # Process 1 day at a time

env:
  SLING_THREADS: 4  # Enable parallel processing
```

## Using SLING\_STATE

If we wish to store the incremental state externally (and avoid using the max value of the target table), we can use the state feature. We need to provide an environment variable called `SLING_STATE`, which is a location where sling will store the respective incremental values. See [Global Variables](/sling-cli/variables#global-environment-variables) for more details.

Here is an example, where sling will store the incremental values in the `my/state` path, in the `AWS_S3` connection:

```yaml
source: postgres
target: snowflake

defaults:
  object: new_schema.{stream_schema}_{stream_table}

streams:
  public.*:
    mode: incremental
    primary_key: id
    update_key: update_dt
  
  public.accounts:
    mode: full-refresh

env:
  SLING_STATE: AWS_S3/my/state
```

## Delete Missing Records

When loading data incrementally, you may want to handle records that exist in the target but are missing from the source. Sling supports both hard deletes (physically removing records) and soft deletes (marking with timestamps).

See the [Capture Deletes](/examples/database-to-database/capture_deletes) page for detailed examples and configuration options.


# Capture Deletes

Examples of using Sling to capture deleted records during incremental loads

When loading data incrementally, you may want to handle records that exist in the target but are missing from the source. The `delete_missing` option supports two modes:

* `hard`: Physically deletes records from the target table that no longer exist in the source
* `soft`: Marks records as deleted in the target table by setting a deletion timestamp

{% hint style="warning" %}
Be careful when enabling this feature on massive tables. The primary key column(s) is fully selected from the source stream each run in order to determine which records don't exist anymore. For large tables, consider using `source_where` and `target_where` to scope the delete detection to a subset of data (see [Scoped Delete with WHERE Clauses](#scoped-delete-with-where-clauses) below).
{% endhint %}

{% hint style="success" %}
For true incremental delete capture without scanning the source table, consider using [Change Capture (CDC)](/concepts/change-capture) mode. CDC reads deletes directly from the database transaction log, making it efficient even on massive tables.
{% endhint %}

## Hard Delete

Physically removes records from the target table that no longer exist in the source.

```yaml
source: MY_POSTGRES
target: MY_SNOWFLAKE

defaults:
  mode: incremental
  update_key: updated_at
  primary_key: id  # primary key is required for delete_missing

streams:
  finance.accounts:
    object: finance.accounts_target
    target_options:
      delete_missing: hard  # will remove records that don't exist in source
```

## Soft Delete

Marks records as deleted in the target table by setting a `_sling_deleted_at` timestamp column.

```yaml
source: MY_POSTGRES
target: MY_SNOWFLAKE

defaults:
  mode: incremental
  update_key: updated_at
  primary_key: id  # primary key is required for delete_missing
  target_options:
    delete_missing: soft  # will mark records as deleted with timestamp

streams:
  finance.accounts:
    object: finance.accounts_target
```

When using `soft` delete mode, Sling will add a `_sling_deleted_at` timestamp column to track when records were marked as deleted.

## Scoped Delete with WHERE Clauses

For large tables with years of historical data, comparing all primary keys can be expensive. You can scope the delete detection to a subset of data using `source_where` and `target_where` clauses. This is useful when you only need to detect deletes in recent data (e.g., records from the last 30 days). This feature is added in *v1.5.6*.

```yaml
source: MY_SQLSERVER
target: MY_POSTGRES

defaults:
  mode: incremental
  update_key: updated_at
  primary_key: id

streams:
  sales.orders:
    object: sales.orders_target
    target_options:
      delete_missing:
        type: soft
        # Only check for deletes in records from the last 30 days
        source_where: created_at >= DATEADD(day, -30, GETDATE())  # SQL Server syntax
        target_where: created_at >= NOW() - INTERVAL '30 days'   # PostgreSQL syntax
```

The extended `delete_missing` configuration accepts:

* `type` **(required)**: Either `soft` or `hard`
* `where`: WHERE clause applied to both source and target (use when both databases share the same SQL syntax)
* `source_where`: WHERE clause for the source query (uses source database SQL syntax)
* `target_where`: WHERE clause for the target query (uses target database SQL syntax)

{% hint style="info" %}
When using different database types (e.g., SQL Server to PostgreSQL), use `source_where` and `target_where` with the appropriate SQL syntax for each database. When source and target use the same SQL dialect, you can use the simpler `where` option.
{% endhint %}

**Example with single `where` clause (same database types):**

```yaml
target_options:
  delete_missing:
    type: hard
    where: created_at >= '2024-01-01'  # works when source and target use same syntax
```

## Important Notes

* The `delete_missing` option requires that you specify a `primary_key` to uniquely identify records
* For incremental loads, an `update_key` is also required to determine which records to process
* The comparison is done using a temporary table to efficiently identify missing records
* When using `source_where`/`target_where`, only records matching the WHERE clause are considered for delete detection
* **Schema configuration**: Ensure your target database connection has the correct default schema configured. For databases that support schema search paths:

  * **PostgreSQL**: Set `search_path` in your connection string (e.g., `?search_path=my_schema`) or configure it at the database/role level
  * **SQL Server**: The default schema is determined by the user's default schema setting
  * **Oracle**: Set the `current_schema` or ensure the user has appropriate schema permissions

  This is particularly important when running multiple streams concurrently, as the delete operation uses temporary tables that need to resolve correctly within the target schema.


# Backfill

Backfill mode allows you to load historical data within a specific range based on an update key. This is useful when you need to reload data for a particular time period or range of values.

## Basic Backfill

Basic backfill requires:

* A primary key to uniquely identify records
* An update key to determine the range
* A range specification defining the start and end values

**Using CLI Flags**

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'my_schema.orders' \
  --tgt-conn MY_TARGET_DB \
  --tgt-object 'target_schema.orders' \
  --mode backfill \
  --primary-key order_id \
  --update-key order_date \
  --range '2023-01-01,2023-12-31'
```

{% endcode %}

**Using Replication Config**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_SOURCE_DB
target: MY_TARGET_DB

defaults:
  mode: backfill
  primary_key: [id]
  object: target_schema.{stream_table}

streams:
  my_schema.orders:
    update_key: order_date
    source_options:
      range: '2023-01-01,2023-12-31'

  my_schema.transactions:
    update_key: id # same as primary key
    source_options:
      range: '100000,200000'
```

{% endcode %}

## Backfill Chunking

Backfill mode with chunking allows loading historical data in smaller ranges, optimizing for large datasets. See the [chunking documentation](/examples/database-to-database/chunking#chunking-in-backfill-mode) for details.

```yaml
source: postgres
target: oracle

defaults:
  mode: backfill
  object: oracle.{stream_table_lower}
  primary_key: [id]

streams:
  public.orders_mariadb_pg:
    update_key: update_dt
    source_options:
      range: '2018-11-01,2018-12-01'
      chunk_size: 10d  # Process in 10-day chunks

  public.orders_sqlserver_pg:
    update_key: date
    source_options:
      range: '2019-01-01,2019-06-01'
      chunk_size: 2m   # Process in 2-month chunks

  public.orders_snowflake_pg:
    update_key: id  # same as primary key
    source_options:
      range: '1,800'
      chunk_size: 200  # Process in chunks of 200 IDs

  public.large_table_pg:
    update_key: created_at
    source_options:
      range: '2023-01-01,2023-12-31'
      chunk_count: 6  # Split into 6 equal time-based chunks

  public.user_data_pg:
    update_key: user_id
    source_options:
      range: '1000,50000'
      chunk_count: 10  # Split into 10 equal numeric chunks

env:
  SLING_THREADS: 3  # Process 3 streams concurrently
```


# Schema Migration

Examples of using Sling to migrate database schema attributes along with data

Schema migration (*v1.5.7*) is a powerful feature that allows Sling to replicate not just data, but also important schema attributes like primary keys, foreign keys, indexes, default values, nullable constraints, auto-increment columns, and column/table descriptions from source to target databases.

{% hint style="success" %}
**CLI Pro / Platform Feature**: Schema migration requires a [CLI Pro Max token](/sling-cli/cli-pro) or [Advanced Platform Plan](/sling-platform/platform).
{% endhint %}

{% hint style="info" %}
Schema migration is controlled via the `SLING_SCHEMA_MIGRATION` environment variable and is disabled by default.
{% endhint %}

## Overview

When migrating databases, preserving schema attributes is often critical for maintaining data integrity and application compatibility. Sling's schema migration feature extracts extended metadata from source tables and applies the corresponding constraints and properties to target tables.

### Supported Attributes

| Attribute        | Description                     | Example                               |
| ---------------- | ------------------------------- | ------------------------------------- |
| `primary_key`    | Primary key constraints         | `id INT PRIMARY KEY`                  |
| `foreign_key`    | Foreign key relationships       | `REFERENCES customers(id)`            |
| `indexes`        | Non-primary key indexes         | `CREATE INDEX idx_name ON table(col)` |
| `auto_increment` | Identity/auto-increment columns | `IDENTITY(1,1)` or `SERIAL`           |
| `nullable`       | NOT NULL constraints            | `email VARCHAR(255) NOT NULL`         |
| `default_value`  | Column default values           | `DEFAULT CURRENT_TIMESTAMP`           |
| `description`    | Column and table comments       | `COMMENT ON COLUMN...`                |

### Supported Databases

Schema migration works between any combination of these databases. The level of support varies by database—see the compatibility matrix below for details.

#### Full Support

* PostgreSQL
* MySQL / MariaDB
* SQL Server / Azure SQL
* Oracle
* Redshift
* Databricks

#### Partial Support

* Snowflake (no indexes)
* BigQuery (no auto-increment, no default values, no indexes)
* SQLite (no descriptions)
* DuckDB (no auto-increment, no default values)
* ClickHouse (primary key, nullable, description only)

### Database Compatibility Matrix

The following matrix shows which schema attributes are supported for each database:

| Database                   | Auto-Increment | Primary Key | Foreign Key | Default Value | Nullable | Indexes | Description |
| -------------------------- | :------------: | :---------: | :---------: | :-----------: | :------: | :-----: | :---------: |
| **PostgreSQL**             |        ✓       |      ✓      |      ✓      |       ✓       |     ✓    |    ✓    |      ✓      |
| **MySQL / MariaDB**        |        ✓       |      ✓      |      ✓¹     |       ✓       |     ✓    |    ✓    |      ✓      |
| **SQL Server / Azure SQL** |        ✓       |      ✓      |      ✓      |       ✓       |     ✓    |    ✓    |      ✓      |
| **Oracle**                 |        ✓       |      ✓      |      ✓²     |       ✓       |     ✓    |    ✓    |      ✓      |
| **Redshift**               |        ✓       |      ✓³     |      ✓³     |       ✓       |     ✓    |    —⁴   |      ✓      |
| **Databricks**             |       ✓⁵       |      ✓⁶     |      ✓⁶     |       ✓       |     ✓    |    —⁷   |      ✓      |
| **Snowflake**              |        ✓       |      ✓⁸     |      ✓⁸     |       ✓       |     ✓    |    —    |      ✓      |
| **BigQuery**               |        —       |      ✓⁹     |      ✓⁹     |       —       |     ✓    |    —    |      ✓      |
| **SQLite**                 |       ✓¹⁰      |      ✓      |     ✓¹¹     |       ✓       |     ✓    |    ✓    |      —      |
| **DuckDB**                 |        —       |      ✓      |      ✓      |       —       |     ✓    |    ✓    |      ✓      |
| **ClickHouse**             |        —       |     ✓¹²     |      —      |       ✓       |     ✓    |    —    |      ✓      |

**Legend:** ✓ = Supported | — = Not supported

**Notes:**

1. MySQL/MariaDB: Foreign keys require InnoDB engine; MyISAM ignores FK constraints
2. Oracle: Supports `ON DELETE` actions but not `ON UPDATE` actions for foreign keys
3. Redshift: Primary keys and foreign keys are **informational only** (not enforced)
4. Redshift: Does not support traditional indexes; use SORTKEY/DISTKEY instead
5. Databricks: Auto-increment only supports `BIGINT` column type
6. Databricks: Primary keys and foreign keys are **informational only** (not enforced); only NOT NULL is enforced
7. Databricks: Does not support traditional indexes; use Z-ordering instead
8. Snowflake: Primary keys and foreign keys are **not enforced** in standard tables (metadata only); only NOT NULL is enforced
9. BigQuery: Primary keys and foreign keys use `NOT ENFORCED` syntax (metadata only for query optimization)
10. SQLite: Auto-increment only works with `INTEGER PRIMARY KEY` columns
11. SQLite: Foreign keys require `PRAGMA foreign_keys = ON` per connection (disabled by default)
12. ClickHouse: Primary key defines sort order and sparse index; does not enforce uniqueness

### Constraint Enforcement Matrix

{% hint style="warning" %}
**Important:** Not all databases enforce constraints at runtime. Some databases accept constraint DDL for metadata/documentation purposes but do not validate data against those constraints. This matrix shows which constraints are actually enforced.
{% endhint %}

| Database            | PK Enforced |   FK Enforced  | NOT NULL Enforced | Default Enforced |
| ------------------- | :---------: | :------------: | :---------------: | :--------------: |
| **PostgreSQL**      |    ✓ Yes    |      ✓ Yes     |       ✓ Yes       |       ✓ Yes      |
| **MySQL / MariaDB** |    ✓ Yes    | ✓ Yes (InnoDB) |       ✓ Yes       |       ✓ Yes      |
| **SQL Server**      |    ✓ Yes    |      ✓ Yes     |       ✓ Yes       |       ✓ Yes      |
| **Oracle**          |    ✓ Yes    |      ✓ Yes     |       ✓ Yes       |       ✓ Yes      |
| **Redshift**        |     ✗ No    |      ✗ No      |       ✓ Yes       |       ✓ Yes      |
| **Databricks**      |     ✗ No    |      ✗ No      |       ✓ Yes       |       ✓ Yes      |
| **Snowflake**       |     ✗ No    |      ✗ No      |       ✓ Yes       |       ✓ Yes      |
| **BigQuery**        |     ✗ No    |      ✗ No      |       ✓ Yes       |      Partial     |
| **SQLite**          |    ✓ Yes    |     ✓ Yes\*    |       ✓ Yes       |       ✓ Yes      |
| **DuckDB**          |    ✓ Yes    |      ✓ Yes     |       ✓ Yes       |       ✓ Yes      |
| **ClickHouse**      |     ✗ No    |       N/A      |        ✗ No       |       ✓ Yes      |

**Legend:** ✓ Yes = Enforced at runtime | ✗ No = Metadata only (not enforced) | \* = Requires configuration

{% hint style="info" %}
**SQLite Note:** Foreign key enforcement requires `PRAGMA foreign_keys = ON` to be set on each database connection. This is disabled by default for backwards compatibility.
{% endhint %}

{% hint style="info" %}
**Analytical Databases:** Redshift, Snowflake, BigQuery, and Databricks are optimized for analytical workloads. They accept constraint definitions for documentation and query optimization hints, but do not enforce them at runtime. Data integrity must be validated upstream in ETL pipelines.
{% endhint %}

### What You Can Use for Each Database

{% tabs %}
{% tab title="PostgreSQL" %}
**All features supported and enforced.** PostgreSQL is fully compatible with schema migration.

```yaml
env:
  SLING_SCHEMA_MIGRATION: all
```

| Attribute      | Supported |  Enforced  |
| -------------- | :-------: | :--------: |
| Primary Key    |     ✓     |    ✓ Yes   |
| Foreign Key    |     ✓     |    ✓ Yes   |
| Auto-increment |     ✓     |    ✓ Yes   |
| Default Value  |     ✓     |    ✓ Yes   |
| NOT NULL       |     ✓     |    ✓ Yes   |
| Indexes        |     ✓     | Functional |
| Description    |     ✓     |  Metadata  |

* Auto-increment uses `GENERATED BY DEFAULT AS IDENTITY`
* Full foreign key support with `ON DELETE` and `ON UPDATE` actions
* Column/table comments via `COMMENT ON` syntax
  {% endtab %}

{% tab title="MySQL / MariaDB" %}
**All features supported and enforced** (with InnoDB engine).

```yaml
env:
  SLING_SCHEMA_MIGRATION: all
```

| Attribute      | Supported |       Enforced      |
| -------------- | :-------: | :-----------------: |
| Primary Key    |     ✓     |        ✓ Yes        |
| Foreign Key    |     ✓     | ✓ Yes (InnoDB only) |
| Auto-increment |     ✓     |        ✓ Yes        |
| Default Value  |     ✓     |        ✓ Yes        |
| NOT NULL       |     ✓     |        ✓ Yes        |
| Indexes        |     ✓     |      Functional     |
| Description    |     ✓     |       Metadata      |

* Auto-increment uses `AUTO_INCREMENT` keyword
* Foreign keys require **InnoDB** engine (MyISAM ignores FK constraints)
* Comments via `COMMENT` clause
  {% endtab %}

{% tab title="SQL Server" %}
**All features supported and enforced.** SQL Server and Azure SQL are fully compatible.

```yaml
env:
  SLING_SCHEMA_MIGRATION: all
```

| Attribute      | Supported |  Enforced  |
| -------------- | :-------: | :--------: |
| Primary Key    |     ✓     |    ✓ Yes   |
| Foreign Key    |     ✓     |    ✓ Yes   |
| Auto-increment |     ✓     |    ✓ Yes   |
| Default Value  |     ✓     |    ✓ Yes   |
| NOT NULL       |     ✓     |    ✓ Yes   |
| Indexes        |     ✓     | Functional |
| Description    |     ✓     |  Metadata  |

* Auto-increment uses `IDENTITY(seed, increment)` syntax
* Foreign keys can be disabled with `NOCHECK` if needed
* Descriptions via Extended Properties (`MS_Description`)
  {% endtab %}

{% tab title="Oracle" %}
**All features supported and enforced** with one limitation.

```yaml
env:
  SLING_SCHEMA_MIGRATION: all
```

| Attribute      | Supported |        Enforced        |
| -------------- | :-------: | :--------------------: |
| Primary Key    |     ✓     |          ✓ Yes         |
| Foreign Key    |     ✓     | ✓ Yes (ON DELETE only) |
| Auto-increment |     ✓     |          ✓ Yes         |
| Default Value  |     ✓     |          ✓ Yes         |
| NOT NULL       |     ✓     |          ✓ Yes         |
| Indexes        |     ✓     |       Functional       |
| Description    |     ✓     |        Metadata        |

* Auto-increment uses `GENERATED BY DEFAULT AS IDENTITY`
* Foreign keys support `ON DELETE` actions but **not `ON UPDATE`** (Oracle limitation)
* Comments via `COMMENT ON COLUMN` / `COMMENT ON TABLE`
  {% endtab %}

{% tab title="Snowflake" %}
**DDL supported but constraints not enforced.** Snowflake is an analytical database.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,auto_increment,nullable,default_value,description
```

| Attribute      | Supported |       Enforced       |
| -------------- | :-------: | :------------------: |
| Primary Key    |     ✓     | ✗ No (metadata only) |
| Foreign Key    |     ✓     | ✗ No (metadata only) |
| Auto-increment |     ✓     |         ✓ Yes        |
| Default Value  |     ✓     |         ✓ Yes        |
| NOT NULL       |     ✓     |         ✓ Yes        |
| Indexes        |     —     |          N/A         |
| Description    |     ✓     |       Metadata       |

* Auto-increment uses `AUTOINCREMENT START ... INCREMENT ...`
* **Primary keys and foreign keys are not enforced** — they exist for documentation and query optimization only
* Only `NOT NULL` constraints are enforced in standard tables
* Indexes are not applicable (Snowflake uses clustering keys instead)
  {% endtab %}

{% tab title="BigQuery" %}
**Limited support.** BigQuery constraints are metadata only.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,nullable,description
```

| Attribute      | Supported |       Enforced      |
| -------------- | :-------: | :-----------------: |
| Primary Key    |     ✓     | ✗ No (NOT ENFORCED) |
| Foreign Key    |     ✓     | ✗ No (NOT ENFORCED) |
| Auto-increment |     —     |         N/A         |
| Default Value  |     —     |         N/A         |
| NOT NULL       |     ✓     |        ✓ Yes        |
| Indexes        |     —     |         N/A         |
| Description    |     ✓     |       Metadata      |

* Primary keys and foreign keys use `NOT ENFORCED` syntax (query optimization hints only)
* Column descriptions via `ALTER TABLE ... SET OPTIONS`
* No identity columns—use application-generated IDs or `GENERATE_UUID()`
  {% endtab %}

{% tab title="Redshift" %}
**DDL supported but PK/FK not enforced.** Redshift is an analytical data warehouse.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,auto_increment,nullable,default_value,description
```

| Attribute      | Supported |       Enforced       |
| -------------- | :-------: | :------------------: |
| Primary Key    |     ✓     | ✗ No (informational) |
| Foreign Key    |     ✓     | ✗ No (informational) |
| Auto-increment |     ✓     |         ✓ Yes        |
| Default Value  |     ✓     |         ✓ Yes        |
| NOT NULL       |     ✓     |         ✓ Yes        |
| Indexes        |     —     |          N/A         |
| Description    |     ✓     |       Metadata       |

* Auto-increment uses `IDENTITY(seed, increment)`
* **Primary keys and foreign keys are informational only** — used as query optimizer hints, not enforced
* Does not support traditional indexes; use `SORTKEY` and `DISTKEY` for performance
* Comments via `COMMENT ON` syntax
  {% endtab %}

{% tab title="SQLite" %}
**Most features supported.** Foreign keys require explicit enabling.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,auto_increment,nullable,default_value,indexes
```

| Attribute      | Supported |       Enforced       |
| -------------- | :-------: | :------------------: |
| Primary Key    |     ✓     |         ✓ Yes        |
| Foreign Key    |     ✓     | ✓ Yes (when enabled) |
| Auto-increment |     ✓     |         ✓ Yes        |
| Default Value  |     ✓     |         ✓ Yes        |
| NOT NULL       |     ✓     |         ✓ Yes        |
| Indexes        |     ✓     |      Functional      |
| Description    |     —     |          N/A         |

* Auto-increment only works with `INTEGER PRIMARY KEY` columns
* **Foreign keys require `PRAGMA foreign_keys = ON`** per connection (disabled by default)
* No native column/table comment support
  {% endtab %}

{% tab title="DuckDB" %}
**Partial support but constraints are enforced.** DuckDB enforces integrity unlike most analytical DBs.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,nullable,indexes,description
```

| Attribute      | Supported |  Enforced  |
| -------------- | :-------: | :--------: |
| Primary Key    |     ✓     |    ✓ Yes   |
| Foreign Key    |     ✓     |    ✓ Yes   |
| Auto-increment |     —     |     N/A    |
| Default Value  |     —     |     N/A    |
| NOT NULL       |     ✓     |    ✓ Yes   |
| Indexes        |     ✓     | Functional |
| Description    |     ✓     |  Metadata  |

* No native identity columns—use sequences with `DEFAULT nextval('seq')`
* **Foreign keys are enforced** (unlike Snowflake, BigQuery, Redshift)
* Indexes are fully supported via `duckdb_indexes()` metadata
* Column descriptions via `COMMENT ON` syntax
  {% endtab %}

{% tab title="Databricks" %}
**DDL supported but PK/FK not enforced.** Databricks uses Delta Lake.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,auto_increment,nullable,default_value,description
```

| Attribute      | Supported |       Enforced       |
| -------------- | :-------: | :------------------: |
| Primary Key    |     ✓     | ✗ No (informational) |
| Foreign Key    |     ✓     | ✗ No (informational) |
| Auto-increment |     ✓     |         ✓ Yes        |
| Default Value  |     ✓     |         ✓ Yes        |
| NOT NULL       |     ✓     |         ✓ Yes        |
| Indexes        |     —     |          N/A         |
| Description    |     ✓     |       Metadata       |

* Auto-increment uses `GENERATED BY DEFAULT AS IDENTITY` (**BIGINT only**)
* **Note:** When Databricks is used as a *source*, identity column detection is limited (Databricks doesn't expose identity metadata via `information_schema`)
* **Primary keys and foreign keys are informational only** — used for query optimization
* Only `NOT NULL` and `CHECK` constraints are enforced
* Does not support traditional indexes; use Z-ordering for performance
  {% endtab %}

{% tab title="ClickHouse" %}
**Limited support.** ClickHouse is optimized for OLAP, not constraint enforcement.

```yaml
env:
  SLING_SCHEMA_MIGRATION: primary_key,nullable,default_value,description
```

| Attribute      | Supported |        Enforced        |
| -------------- | :-------: | :--------------------: |
| Primary Key    |     ✓     | ✗ No (sort order only) |
| Foreign Key    |     —     |           N/A          |
| Auto-increment |     —     |           N/A          |
| Default Value  |     ✓     |          ✓ Yes         |
| NOT NULL       |     ✓     |          ✗ No          |
| Indexes        |     —     |           N/A          |
| Description    |     ✓     |        Metadata        |

* **Primary key defines sort order and sparse index** — does not enforce uniqueness
* No foreign key support
* No auto-increment columns (use `generateSerialID()` in v25.1+)
* NOT NULL is not enforced; use `Nullable(T)` wrapper to allow NULLs
* Uses data-skipping indexes (MinMax, Bloom) instead of traditional indexes
  {% endtab %}
  {% endtabs %}

## Enabling Schema Migration

Set the `SLING_SCHEMA_MIGRATION` environment variable to enable specific attributes:

```yaml
env:
  # Enable all schema attributes
  SLING_SCHEMA_MIGRATION: all

  # Or enable specific attributes (comma-separated)
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,indexes
```

### Available Options

| Value            | Description                             |
| ---------------- | --------------------------------------- |
| `all`            | Enable all schema migration attributes  |
| `primary_key`    | Migrate primary key constraints         |
| `foreign_key`    | Migrate foreign key relationships       |
| `indexes`        | Migrate indexes (non-PK)                |
| `auto_increment` | Migrate identity/auto-increment columns |
| `nullable`       | Migrate NOT NULL constraints            |
| `default_value`  | Migrate column default values           |
| `description`    | Migrate column and table comments       |

## Basic Example

Migrate all schema attributes from SQL Server to PostgreSQL:

```yaml
source: mssql
target: postgres

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  dbo.customers:
  dbo.orders:
  dbo.products:

env:
  SLING_SCHEMA_MIGRATION: all
```

## Foreign Key Migration

When migrating foreign keys, Sling automatically handles table ordering to ensure parent tables are created before child tables that reference them.

{% hint style="warning" %}
All referenced tables must be included in the replication. If table `orders` has a foreign key to `customers`, both tables must be in the streams list.
{% endhint %}

### Automatic Table Ordering

Sling performs topological sorting based on foreign key dependencies. Even if you list tables in the wrong order, they will be processed correctly:

```yaml
source: mssql
target: postgres

defaults:
  mode: full-refresh

# Tables listed in wrong order - Sling will reorder them automatically
# order_items -> orders -> customers (FK dependencies)
streams:
  dbo.order_items:
    object: public.order_items

  dbo.orders:
    object: public.orders

  dbo.customers:  # Will be processed first due to FK dependencies
    object: public.customers

env:
  SLING_SCHEMA_MIGRATION: foreign_key,primary_key
  SLING_THREADS: 3  # works with threads!
```

Sling will automatically reorder streams to: `customers` → `orders` → `order_items`

### Handling Circular Dependencies

If circular foreign key dependencies are detected, Sling will report an error. Options to resolve:

1. Remove one table from the replication
2. Disable foreign key migration
3. Handle FK creation manually via hooks

## Auto-Increment / Identity Columns

Sling migrates auto-increment columns with their seed and increment values:

```yaml
source: mssql     # IDENTITY(1000, 10)
target: postgres  # GENERATED BY DEFAULT AS IDENTITY (START WITH 1000 INCREMENT BY 10)

streams:
  dbo.invoices:
    object: public.invoices

env:
  SLING_SCHEMA_MIGRATION: auto_increment,primary_key
```

### Database-Specific Behavior

| Source                         | Target     | Result                             |
| ------------------------------ | ---------- | ---------------------------------- |
| SQL Server `IDENTITY(1,1)`     | PostgreSQL | `GENERATED BY DEFAULT AS IDENTITY` |
| PostgreSQL `SERIAL`            | MySQL      | `AUTO_INCREMENT`                   |
| Oracle `GENERATED AS IDENTITY` | SQL Server | `IDENTITY(seed, incr)`             |

## Default Values

Sling translates default value expressions between databases:

```yaml
source: mssql
target: postgres

streams:
  dbo.audit_logs:

env:
  SLING_SCHEMA_MIGRATION: default_value
```

### Default Value Translation

| Expression        | SQL Server     | PostgreSQL                             | MySQL               | Oracle                          |
| ----------------- | -------------- | -------------------------------------- | ------------------- | ------------------------------- |
| Current timestamp | `GETDATE()`    | `CURRENT_TIMESTAMP`                    | `CURRENT_TIMESTAMP` | `SYSDATE`                       |
| UTC timestamp     | `GETUTCDATE()` | `CURRENT_TIMESTAMP AT TIME ZONE 'UTC'` | `UTC_TIMESTAMP()`   | `SYS_EXTRACT_UTC(SYSTIMESTAMP)` |
| UUID              | `NEWID()`      | `gen_random_uuid()`                    | `UUID()`            | `SYS_GUID()`                    |
| Boolean true      | `1`            | `true`                                 | `1`                 | `1`                             |
| Boolean false     | `0`            | `false`                                | `0`                 | `0`                             |

## Column & Table Descriptions

Migrate comments and descriptions to preserve documentation:

```yaml
source: mssql
target: postgres

streams:
  dbo.products:

env:
  SLING_SCHEMA_MIGRATION: description
```

This will migrate:

* Column comments (e.g., `COMMENT ON COLUMN products.price IS 'Product retail price'`)
* Table comments (e.g., `COMMENT ON TABLE products IS 'Product catalog'`)

## Indexes

Migrate non-primary key indexes to preserve query performance:

```yaml
source: mssql
target: postgres

streams:
  dbo.customers:

env:
  SLING_SCHEMA_MIGRATION: indexes
```

This migrates regular indexes, unique indexes, and composite indexes.

{% hint style="info" %}
Index names are automatically generated in the target database to avoid conflicts. The format is `idx_{table}_{columns}`.
{% endhint %}

## Complete E-Commerce Migration Example

A comprehensive example migrating an e-commerce schema with all relationships:

```yaml
source: mssql
target: postgres

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  # Parent tables (no FK dependencies)
  dbo.categories:
  dbo.customers:

  # Child tables (have FK to parent tables)
  dbo.products:     # FK to categories
  dbo.orders:       # FK to customers

  # Grandchild tables (have FK to child tables)
  dbo.order_items:  # FK to orders and products

env:
  SLING_SCHEMA_MIGRATION: all
```

## Selective Attribute Migration

Enable only specific attributes based on your needs:

### Data Integrity Focus

```yaml
env:
  # Focus on constraints that ensure data integrity
  SLING_SCHEMA_MIGRATION: primary_key,foreign_key,nullable
```

### Performance Focus

```yaml
env:
  # Focus on attributes that affect query performance
  SLING_SCHEMA_MIGRATION: primary_key,indexes
```

### Documentation Preservation

```yaml
env:
  # Preserve schema documentation only
  SLING_SCHEMA_MIGRATION: description
```

## Using with Pipelines

Schema migration can also be used within pipelines:

```yaml
steps:
  # Setup source schema
  - connection: mssql
    query: |
      CREATE TABLE dbo.users (
        id INT IDENTITY(1,1) PRIMARY KEY,
        email NVARCHAR(255) NOT NULL,
        created_at DATETIME DEFAULT GETDATE()
      );

  # Run replication with schema migration
  - replication:
      source: mssql
      target: postgres

      streams:
        dbo.users:
          object: public.users

      env:
        SLING_SCHEMA_MIGRATION: all

  # Verify constraints were created
  - connection: postgres
    query: |
      SELECT constraint_type, constraint_name
      FROM information_schema.table_constraints
      WHERE table_name = 'users'
```

## Incremental Mode Considerations

Schema migration attributes are applied during table creation. When using `incremental` mode:

* **First run**: Table is created with all schema attributes
* **Subsequent runs**: Only data is upserted; schema is not modified

```yaml
source: mssql
target: postgres

defaults:
  mode: incremental
  primary_key: id
  update_key: updated_at

streams:
  dbo.orders:
    object: public.orders

env:
  SLING_SCHEMA_MIGRATION: all
```

## Troubleshooting

### Missing Foreign Key Dependencies

**Error**: `missing foreign key dependencies: 'orders' depends on 'customers' (not in stream list)`

**Solution**: Add the missing referenced table to your streams:

```yaml
streams:
  dbo.customers:  # Add the referenced table
  dbo.orders:
```

### Circular Dependency Detected

**Error**: `circular foreign key dependency detected involving: table_a, table_b`

**Solution**: Either disable FK migration or handle FKs manually with hooks:

```yaml
hooks:
  end:
    - connection: postgres
      query: |
        ALTER TABLE table_a ADD CONSTRAINT fk_a_b
        FOREIGN KEY (b_id) REFERENCES table_b(id);
```

### Unsupported Default Expression

If a default expression cannot be translated, Sling will pass it through as-is with a debug warning. You may need to manually adjust the default in the target database.

### Identity Column Conflicts

When using `auto_increment` migration, Sling uses `GENERATED BY DEFAULT AS IDENTITY` (PostgreSQL/Oracle) to allow explicit value inserts during data migration while still supporting auto-generation for new inserts.

## Best Practices

1. **Test First**: Run schema migration on a test environment before production
2. **Start Selective**: Begin with `primary_key` only, then add more attributes
3. **Include All Dependencies**: Ensure all FK-referenced tables are in the replication
4. **Use Full-Refresh**: Schema attributes are best applied with `full-refresh` mode initially
5. **Review Generated Schema**: After migration, review the target schema for any needed adjustments
6. **Consider Indexes Separately**: Large tables may benefit from creating indexes after data load via hooks

## See Also

* [Replication Modes](/concepts/replication/modes)
* [Hooks](https://github.com/slingdata-io/sling-docs/blob/master/concepts/hooks/hooks.md)
* [Target Options](/concepts/replication/target-options)


# Database to File

Examples of using Sling to load data from databases to storage systems

We first need to make sure our connections are available in our environment. See [Environment](https://github.com/slingdata-io/sling-docs/blob/master/environment.md), [Storage Connections](/connections/file-connections) and [Database Connections](/connections/database-connections) for more details.

{% tabs %}
{% tab title="Linux / Mac" %}

```bash
export MY_SOURCE_DB='...'

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

{% endtab %}

{% tab title="Windows" %}

```powershell
# using windows Powershell
$env:MY_SOURCE_DB = '...'

$ sling conns list
+---------------+------------------+-----------------+
| CONN NAME     | CONN TYPE        | SOURCE          |
+---------------+------------------+-----------------+
| MY_S3_BUCKET  | FileSys - S3     | sling env yaml  |
| MY_SOURCE_DB  | DB - PostgreSQL  | env variable    |
| MY_GS_BUCKET  | FileSys - Google | sling env yaml  |
| MY_AZURE_CONT | FileSys - Azure  | sling env yaml  |
+---------------+------------------+-----------------+
```

{% endtab %}
{% endtabs %}

<details>

<summary>Database ⇨ Local Storage (CSV)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_file.csv'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_csv_folder/*.csv'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_csv_folder/' \
  --tgt-options '{file_max_rows: 100000, format: csv}'

# Windows path format
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file://C:/Temp/my_csv_folder/' \
  --tgt-options '{file_max_rows: 100000, format: csv}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: LOCAL

defaults:
  target_options:
    format: csv

streams:
  source_schema.source_table:
    object: file:///tmp/my_file.csv

  source_schema.source_table1:
    object: file:///tmp/my_csv_folder/*.csv

  source_schema.source_table2:
    object: file:///tmp/my_csv_folder/
    target_options:
      file_max_rows: 100000

  source_schema.source_table3:
    object: file://C:/Temp/my_csv_folder/ # Windows Path format
    target_options:
      file_max_rows: 100000

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'

# Single file export
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.csv'
        )
    }
)

# Run the replication
replication.run()

# Multiple streams with target options
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.CSV),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.csv'
        ),
        'source_schema.source_table1': ReplicationStream(
            object='file:///tmp/my_csv_folder/*.csv'
        ),
        'source_schema.source_table2': ReplicationStream(
            object='file:///tmp/my_csv_folder/',
            target_options=TargetOptions(file_max_rows=100000)
        ),
        'source_schema.source_table3': ReplicationStream(
            object='file://C:/Temp/my_csv_folder/',  # Windows Path format
            target_options=TargetOptions(file_max_rows=100000)
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()

# Schema wildcard with disabled stream
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.CSV),
    streams={
        'my_schema.*': ReplicationStream(
            object='file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ STDOUT</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --stdout
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Local Storage (JSON)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_file.json'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_json_folder/*.json'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_json_folder/' \
  --tgt-options '{file_max_bytes: 4000000, format: json}'

# Windows Path format
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file://C:/Temp/my_json_folder/' \
  --tgt-options '{file_max_bytes: 4000000, format: json}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: LOCAL

defaults:
  target_options:
    format: json

streams:
  source_schema.source_table:
    object: file:///tmp/my_file.json

  source_schema.source_table1:
    object: file:///tmp/my_json_folder/*.json

  source_schema.source_table2:
    object: file:///tmp/my_json_folder/
    target_options:
      file_max_bytes: 4000000

  source_schema.source_table3:
    object: file://C:/Temp/my_json_folder/
    target_options:
      file_max_bytes: 4000000

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'

# Single JSON file export
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.json'
        )
    }
)

# Run the replication
replication.run()

# Multiple streams with target options
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.JSON),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.json'
        ),
        'source_schema.source_table1': ReplicationStream(
            object='file:///tmp/my_json_folder/*.json'
        ),
        'source_schema.source_table2': ReplicationStream(
            object='file:///tmp/my_json_folder/',
            target_options=TargetOptions(file_max_bytes=4000000)
        ),
        'source_schema.source_table3': ReplicationStream(
            object='file://C:/Temp/my_json_folder/',  # Windows Path format
            target_options=TargetOptions(file_max_bytes=4000000)
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()

# Schema wildcard with disabled stream
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.JSON),
    streams={
        'my_schema.*': ReplicationStream(
            object='file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Local Storage (JSON Lines)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_file.jsonl' \
  --tgt-options '{format: jsonlines}'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_json_folder/*.jsonl'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_json_folder/' \
  --tgt-options '{file_max_bytes: 4000000, format: jsonlines}'

# Windows Path format
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file://C:/Temp/my_json_folder/' \
  --tgt-options '{file_max_bytes: 4000000, format: jsonlines}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: LOCAL

defaults:
  target_options:
    format: jsonlines

streams:
  source_schema.source_table:
    object: file:///tmp/my_file.jsonl

  source_schema.source_table1:
    object: file:///tmp/my_jsonlines_folder/*.jsonl

  source_schema.source_table2:
    object: file:///tmp/my_jsonlines_folder/
    target_options:
      file_max_bytes: 4000000

  source_schema.source_table3:
    object: file://C:/Temp/my_jsonlines_folder/ # Windows Path format
    target_options:
      file_max_bytes: 4000000

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'

# Single JSON Lines file export
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.jsonl',
            target_options=TargetOptions(format=Format.JSONLINES)
        )
    }
)

# Run the replication
replication.run()

# Multiple streams with target options
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.JSONLINES),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.jsonl'
        ),
        'source_schema.source_table1': ReplicationStream(
            object='file:///tmp/my_jsonlines_folder/*.jsonl'
        ),
        'source_schema.source_table2': ReplicationStream(
            object='file:///tmp/my_jsonlines_folder/',
            target_options=TargetOptions(file_max_bytes=4000000)
        ),
        'source_schema.source_table3': ReplicationStream(
            object='file://C:/Temp/my_jsonlines_folder/',  # Windows Path format
            target_options=TargetOptions(file_max_bytes=4000000)
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()

# Schema wildcard with disabled stream
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.JSONLINES),
    streams={
        'my_schema.*': ReplicationStream(
            object='file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Local Storage (Parquet)</summary>

See also [Incremental Examples](/examples/database-to-file/incremental).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_file.parquet'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_parquet_folder/*.parquet'

$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file:///tmp/my_parquet_folder/' \
  --tgt-options '{file_max_rows: 4000000, format: parquet}'

# Windows Path format
$ sling run --src-conn MY_SOURCE_DB \
  --src-stream 'source_schema.source_table' \
  --tgt-object 'file://C:/Temp/my_parquet_folder/' \
  --tgt-options '{file_max_rows: 4000000, format: parquet}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: LOCAL

defaults:
  target_options:
    format: parquet

streams:
  source_schema.source_table:
    object: file://C:/Temp/my_file.parquet # Windows Path format

  source_schema.source_table1:
    object: file://C:/Temp/my_parquet_folder/*.parquet # Windows Path format

  source_schema.source_table2:
    object: file:///tmp/my_parquet_folder/
    target_options:
      file_max_rows: 1000000

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/
    target_options:
      file_max_rows: 400000 # will split files into folder
  my_schema.forbidden_table:
    disabled: true

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'

# Single Parquet file export
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file:///tmp/my_file.parquet'
        )
    }
)

# Run the replication
replication.run()

# Multiple streams with target options
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.PARQUET),
    streams={
        'source_schema.source_table': ReplicationStream(
            object='file://C:/Temp/my_file.parquet'  # Windows Path format
        ),
        'source_schema.source_table1': ReplicationStream(
            object='file://C:/Temp/my_parquet_folder/*.parquet'  # Windows Path format
        ),
        'source_schema.source_table2': ReplicationStream(
            object='file:///tmp/my_parquet_folder/',
            target_options=TargetOptions(file_max_rows=1000000)
        )
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()

# Schema wildcard with disabled stream
replication = Replication(
    source='MY_SOURCE_DB',
    target='LOCAL',
    defaults=TargetOptions(format=Format.PARQUET),
    streams={
        'my_schema.*': ReplicationStream(
            object='file:///tmp/{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        )
    },
    env={'SLING_THREADS': '3'}
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Local Storage (GeoJSON)</summary>

Export spatial data from PostgreSQL/PostGIS to GeoJSON format. Sling will convert geometry columns to RFC 7946 compliant GeoJSON. Available from *v1.5.2*.

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
# Export with auto-detected geometry column (looks for column named "geometry")
$ sling run --src-conn MY_POSTGIS \
  --src-stream 'public.locations' \
  --tgt-object 'file:///tmp/locations.geojson' \
  --tgt-options '{format: geojson}'

# Specify which column contains geometry data
$ sling run --src-conn MY_POSTGIS \
  --src-stream 'public.parcels' \
  --tgt-object 'file:///tmp/parcels.geojson' \
  --tgt-options '{format: geojson, columns: {geom: geometry}}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_POSTGIS
target: LOCAL

defaults:
  target_options:
    format: geojson

streams:
  # Auto-detect geometry column (must be named "geometry")
  public.locations:
    object: file:///tmp/locations.geojson

  # Specify geometry column explicitly
  public.parcels:
    sql: |
      -- Create test data inline using CTE
      WITH test_data AS (
        SELECT
          1 as id,
          'Point 1' as name,
          ST_GeomFromText('POINT(9.09425263416477 53.4920035631827)', 4326)::geometry as geom
        UNION ALL
        SELECT
          2 as id,
          'Point 2' as name,
          ST_GeomFromText('POINT(13.0532270916455 49.199065154883)', 4326)::geometry as geom
        UNION ALL
        SELECT
          3 as id,
          'Point 3' as name,
          ST_GeomFromText('POINT(7.81573202029895 52.6718611999912)', 4326)::geometry as geom
      )
      SELECT
        id,
        name,
        geom
      FROM test_data
    object: file:///tmp/parcels.geojson
    columns:
      geom: geometry  # designate 'geom' as the geometry column

  # Rename geometry column in output
  public.boundaries:
    object: file:///tmp/boundaries.geojson
    columns:
      shape: geometry  # 'shape' column will be used as geometry
```

{% endcode %}

{% hint style="info" %}
**Note:** GeoJSON format supports only one geometry column per stream, per RFC 7946. If your table has multiple geometry columns, specify which one to use via the `columns` configuration.
{% endhint %}

</details>

<details>

<summary>Database ⇨ Cloud Storage (CSV)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_file.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_csv_folder/*.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_csv_folder/' --tgt-options '{file_max_rows: 100000, format: csv}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_file.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_csv_folder/*.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_csv_folder/' --tgt-options '{file_max_rows: 100000, format: csv}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_file.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_csv_folder/*.csv'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_csv_folder/' --tgt-options '{file_max_rows: 100000, format: csv}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_CLOUD_STORAGE

defaults:
  object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.csv.gz
  target_options:
    format: csv
    compression: gzip

streams:

  # all tables in schema
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.csv
    target_options:
      file_max_rows: 400000 # will split files into folder

  other_schema.source_table: # will use defaults

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format, Compression
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'
os.environ['MY_CLOUD_STORAGE'] = '...'

# Cloud storage export with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_CLOUD_STORAGE',
    defaults={
        'object': '{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.csv.gz',
        'target_options': TargetOptions(
            format=Format.CSV,
            compression=Compression.GZIP
        )
    },
    streams={
        # all tables in schema
        'my_schema.*': ReplicationStream(
            object='{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.csv',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'other_schema.source_table': {}  # will use defaults
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Cloud Storage (JSON)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_file.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_json_folder/*.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: json}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_file.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_json_folder/*.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: json}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_file.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_json_folder/*.json'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: json}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_CLOUD_STORAGE

defaults:
  object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.json.gz
  target_options:
    format: json
    compression: gzip

streams:

  # all tables in schema
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.json
    target_options:
      file_max_rows: 400000 # will split files into folder

  other_schema.source_table: # will use defaults
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format, Compression
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'
os.environ['MY_CLOUD_STORAGE'] = '...'

# Cloud storage JSON export with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_CLOUD_STORAGE',
    defaults={
        'object': '{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.json.gz',
        'target_options': TargetOptions(
            format=Format.JSON,
            compression=Compression.GZIP
        )
    },
    streams={
        # all tables in schema
        'my_schema.*': ReplicationStream(
            object='{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.json',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'other_schema.source_table': {}  # will use defaults
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Cloud Storage (JSON Lines)</summary>

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_file.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_json_folder/*.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: jsonlines}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_file.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_json_folder/*.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: jsonlines}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_file.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_json_folder/*.jsonl'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_json_folder/' --tgt-options '{file_max_rows: 100000, format: jsonlines}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_CLOUD_STORAGE

defaults:
  object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.jsonl.gz
  target_options:
    format: jsonlines
    compression: gzip

streams:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.jsonl
    target_options:
      file_max_rows: 400000 # will split files into folder

  my_schema.forbidden_table:
    disabled: true

  other_schema.source_table: # will use defaults

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format, Compression
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'
os.environ['MY_CLOUD_STORAGE'] = '...'

# Cloud storage JSON Lines export with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_CLOUD_STORAGE',
    defaults={
        'object': '{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.jsonl.gz',
        'target_options': TargetOptions(
            format=Format.JSONLINES,
            compression=Compression.GZIP
        )
    },
    streams={
        # all tables in schema, except "forbidden_table"
        'my_schema.*': ReplicationStream(
            object='{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.jsonl',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        ),
        'other_schema.source_table': {}  # will use defaults
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Database ⇨ Cloud Storage (Parquet)</summary>

See also [Incremental Examples](/examples/database-to-file/incremental).

**Using** [**CLI Flags**](/sling-cli/run#cli-flags-overview)

{% code title="sling.sh" overflow="wrap" %}

```bash
$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_file.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_parquet_folder/*.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_S3_BUCKET --tgt-object 's3://my-bucket/my_parquet_folder/' --tgt-options '{file_max_rows: 100000, format: parquet}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_file.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_parquet_folder/*.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_GS_BUCKET --tgt-object 'gs://my-bucket/my_parquet_folder/' --tgt-options '{file_max_rows: 100000, format: parquet}'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_file.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_parquet_folder/*.parquet'

$ sling run --src-conn MY_SOURCE_DB --src-stream 'source_schema.source_table' --tgt-conn MY_AZURE_CONT --tgt-object 'https://my_account.blob.core.windows.net/my-container/my_parquet_folder/' --tgt-options '{file_max_rows: 100000, format: parquet}'
```

{% endcode %}

***

**Using** [**Replication**](/concepts/replication)

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" fullWidth="false" %}

```yaml
source: MY_SOURCE_DB
target: MY_CLOUD_STORAGE

defaults:
  object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.parquet
  target_options:
    format: parquet

streams:

  # all tables in schema, except "forbidden_table"
  my_schema.*:
    object: {stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.parquet
    target_options:
      file_max_rows: 400000 # will split files into folder
  my_schema.forbidden_table:
    disabled: true

  other_schema.source_table: # will use defaults

env:
  SLING_THREADS: 3 # run streams concurrently
```

{% endcode %}

***

**Using** [**Python**](/examples/sling-python)

{% code title="replication.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, TargetOptions, Format
import os

# Set environment variables
os.environ['MY_SOURCE_DB'] = '...'
os.environ['MY_CLOUD_STORAGE'] = '...'

# Cloud storage Parquet export with defaults
replication = Replication(
    source='MY_SOURCE_DB',
    target='MY_CLOUD_STORAGE',
    defaults={
        'object': '{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}.parquet',
        'target_options': TargetOptions(format=Format.PARQUET)
    },
    streams={
        # all tables in schema, except "forbidden_table"
        'my_schema.*': ReplicationStream(
            object='{stream_schema}/{stream_table}/{YYYY}_{MM}_{DD}/*.parquet',
            target_options=TargetOptions(file_max_rows=400000)
        ),
        'my_schema.forbidden_table': ReplicationStream(
            disabled=True
        ),
        'other_schema.source_table': {}  # will use defaults
    },
    env={'SLING_THREADS': '3'}  # run streams concurrently
)

replication.run()
```

{% endcode %}

</details>


# Incremental

Examples of using Sling to incrementally load data from databases to files

In order to write to files incrementally, we need to provide an update\_key (which will be used as the partition key) to partition the files into a time resolution. Therefore, specific runtime variables are required to be part of the target object path.

Here are the list of [runtime variables](/concepts/replication/runtime-variables#partition-patterns) for partitioning the file chunks: `part_year`, `part_month`, `part_year_month`, `part_day`, `part_week`, `part_hour`, `part_minute`.

Additionally, we need to provide a environment variable called `SLING_STATE`, which is a location where sling will store the respective incremental values. See [Global Variables](/sling-cli/variables#global-environment-variables) for more details.

Here is an example of incrementally writing to an S3 bucket.

```yaml
source: postgres
target: aws_s3

# applies to all streams
defaults:
  object: tables/{stream_schema}/{stream_table}/{part_year}/{part_month}
  mode: incremental   # mode applies to all streams
  target_options:     # target_options applies to all streams
    format: parquet

streams:
  # all tables in schema main
  main.*:
    primary_key: id
    update_key: created_dt
    target_options:     # overwrites default target_options (write as csv)
      format: csv

  public.transactions:
    primary_key: tx_id
    update_key: created_dt

  public.orders:
    primary_key: order_id
    update_key: timestamp
    sql: |
      select *
      from public.orders
      where status not in ('voided')
        and {incremental_where_cond}

env:
  # uses the `path/to/folder` in the same AWS_S3 connection
  SLING_STATE: AWS_S3/path/to/folder
```

This will write data from tables `public.transactions` and `public.orders` (custom SQL) into the respective paths (such as `tables/public/transactions/2024/11/data_0.parquet` and `tables/public/orders/2024/11/data_0.parquet`) at the year and month level:

* `tables/public/transactions/created_dt_year=2024/created_dt_month=01/data_0.parquet`
* `tables/public/transactions/created_dt_year=2024/created_dt_month=02/data_0.parquet`
* ...
* `tables/public/orders/timestamp_year=2024/timestamp_month=01/data_0.parquet`
* `tables/public/orders/timestamp_year=2024/timestamp_month=01/data_0.parquet`
* ...

The first time the replication is ran, it will select all the data (unless limited), and write the last incremental value into the `SLING_STATE` location. This incremental value will be truncated to the lowest partition level. For example, if the lowest level is `part_day`, a value of `2024-11-04 04:05:06` will be truncated to `2024-11-04`. If the lowest level was `part_month`, the truncated value would be `2024-11-01`.

Once the replication is ran again, it will read this truncated incremental value from the `SLING_STATE`, and use it to obtain the complete partition so that no data is missed.


# API to Database

Examples of using Sling to load data from APIs to databases using API specifications

Sling can extract data from REST and GraphQL APIs using YAML specification files. These specifications define how to authenticate, paginate, and extract data from API endpoints.

See [API Spec](/concepts/api-specs) for detailed information about building API specifications.

<details>

<summary>Basic REST API with Offset Pagination</summary>

This example shows a simple API with offset-based pagination.

Learn more: [Pagination](/concepts/api-specs/advanced#pagination) • [Response Processing](/concepts/api-specs/response)

**Spec File** (`users_api.yaml`)

{% code title="users\_api.yaml" overflow="wrap" %}

```yaml
name: "Users API"

defaults:
  state:
    base_url: https://api.example.com
  request:
    headers:
      Accept: "application/json"

endpoints:
  users:
    state:
      limit: 100
      offset: 0  # Start at the beginning

    request:
      url: '{state.base_url}/users'
      method: GET
      parameters:
        limit: '{state.limit}'
        offset: '{state.offset}'  # Pass offset as query parameter

    pagination:
      next_state:
        # Increment offset by limit for next page
        offset: '{state.offset + state.limit}'
      # Stop when fewer records returned than requested
      stop_condition: length(response.records) < state.limit

    response:
      records:
        # Extract records array from response using JMESPath
        jmespath: "data[]"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  users:
    object: public.users
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'users': ReplicationStream(
            object='public.users',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Cursor-Based Pagination API</summary>

This example demonstrates cursor-based pagination similar to Stripe's API pattern.

Learn more: [Pagination](/concepts/api-specs/advanced#pagination) • [Authentication](/concepts/api-specs/authentication)

**Spec File** (`transactions_api.yaml`)

{% code title="transactions\_api.yaml" overflow="wrap" %}

```yaml
name: "Transactions API"

defaults:
  state:
    base_url: https://api.example.com/v1
  request:
    headers:
      Accept: "application/json"
      # Bearer token from secrets defined in env.yaml
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  transactions:
    state:
      limit: 100
      starting_after: null  # First page has no cursor

    request:
      url: '{state.base_url}/transactions'
      method: GET
      parameters:
        limit: '{state.limit}'
        starting_after: '{state.starting_after}'

    pagination:
      next_state:
        # Use ID of last record as cursor for next page
        starting_after: '{response.records[-1].id}'
      # Stop when API indicates no more pages or no records returned
      stop_condition: response.json.has_more == false || length(response.records) < 1

    response:
      records:
        jmespath: "data[]"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  transactions:
    object: public.transactions
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

# Set API credentials
os.environ['API_KEY'] = 'your_api_key'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'transactions': ReplicationStream(
            object='public.transactions',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Incremental Sync with Timestamps</summary>

This example shows how to fetch only new or updated records using timestamp-based incremental synchronization.

Learn more: [Incremental Sync](/concepts/api-specs/advanced#incremental-synchronization) • [Processors](/concepts/api-specs/advanced#data-processors)

**Spec File** (`orders_api.yaml`)

{% code title="orders\_api.yaml" overflow="wrap" %}

```yaml
name: "Orders API"

defaults:
  state:
    base_url: https://api.example.com
  request:
    headers:
      Accept: "application/json"
      Authorization: "Bearer {secrets.api_token}"

endpoints:
  orders:
    state:
      # Use last sync time if available, otherwise default to 30 days ago
      updated_at_min: '{coalesce(sync.last_updated_at, date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z"))}'
      current_max_updated_at: '{state.updated_at_min}'
      limit: 100
      offset: 0

    # Persist last_updated_at for next run
    sync: [last_updated_at]

    request:
      url: '{state.base_url}/orders'
      method: GET
      parameters:
        # Filter API to only return records updated after this timestamp
        updated_at_min: '{state.updated_at_min}'
        limit: '{state.limit}'
        offset: '{state.offset}'

    pagination:
      next_state:
        offset: '{state.offset + state.limit}'
      stop_condition: length(response.records) < state.limit

    response:
      records:
        jmespath: "orders[]"
      processors:
        # Track the maximum updated_at timestamp across all records
        - expression: "record.updated_at"
          output: "state.last_updated_at"
          aggregation: "maximum"  # Save highest timestamp for next sync
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: incremental
  primary_key: [id]
  update_key: updated_at

streams:
  orders:
    object: public.orders

env:
  SLING_STATE: postgres/sling_state.my_api # one state table per replication to persist
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['API_TOKEN'] = 'your_api_token'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'orders': ReplicationStream(
            object='public.orders',
            mode=Mode.INCREMENTAL,
            primary_key=['id'],
            update_key='updated_at'
        )
    },
    env={'SLING_STATE': 'postgres/sling_state.my_api'}
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Nested Data Extraction with JMESPath</summary>

This example demonstrates extracting nested data from complex API responses.

Learn more: [Response Processing](/concepts/api-specs/response) • [JMESPath Expressions](/concepts/api-specs/response#projection-and-transformation)

**Spec File** (`products_api.yaml`)

{% code title="products\_api.yaml" overflow="wrap" %}

```yaml
name: "Products API"

defaults:
  state:
    base_url: https://api.store.com/v2
  request:
    headers:
      Accept: "application/json"
      X-API-Key: '{secrets.api_key}'

endpoints:
  products:
    state:
      page: 1
      per_page: 50

    request:
      url: '{state.base_url}/products'
      method: GET
      parameters:
        page: '{state.page}'
        per_page: '{state.per_page}'

    pagination:
      next_state:
        page: '{state.page + 1}'
      stop_condition: length(response.records) < state.per_page

    response:
      records:
        # JMESPath projection to flatten and reshape nested data
        # Maps: product_id -> id, product_name -> name, pricing.amount -> price, etc.
        jmespath: >
          result.items[].{
            id: product_id,
            name: product_name,
            price: pricing.amount,
            currency: pricing.currency,
            category: metadata.category,
            tags: tags[].name
          }
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  products:
    object: public.products
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['API_KEY'] = 'your_api_key'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'products': ReplicationStream(
            object='public.products',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>POST Request with JSON Payload</summary>

This example shows how to send POST requests with a JSON body to an API.

Learn more: [Request Configuration](/concepts/api-specs/request) • [Dynamic Values](/concepts/api-specs/structure#state-variables)

**Spec File** (`analytics_api.yaml`)

{% code title="analytics\_api.yaml" overflow="wrap" %}

```yaml
name: "Analytics API"

defaults:
  state:
    base_url: https://analytics.example.com/api
  request:
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {secrets.access_token}"

endpoints:
  events:
    state:
      # Use env vars if available, otherwise default to last 7 days
      start_date: '{coalesce(env.START_DATE, date_format(date_add(now(), -7, "day"), "%Y-%m-%d"))}'
      end_date: '{coalesce(env.END_DATE, date_format(now(), "%Y-%m-%d"))}'

    request:
      url: '{state.base_url}/events/query'
      method: POST
      payload:
        # Request body sent as JSON
        date_range:
          start: '{state.start_date}'
          end: '{state.end_date}'
        metrics: ["views", "clicks", "conversions"]
        dimensions: ["event_name", "user_id"]

    response:
      records:
        jmespath: "data.events[]"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  events:
    object: analytics.events

env:
  START_DATE: ${START_DATE}  # passed as env var
  END_DATE: ${END_DATE}      # passed as env var
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['ACCESS_TOKEN'] = 'your_access_token'
os.environ['START_DATE'] = '2024-01-01'
os.environ['END_DATE'] = '2024-01-31'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'events': ReplicationStream(
            object='analytics.events',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Dynamic URL Path Parameters</summary>

This example demonstrates building URLs with dynamic path parameters.

Learn more: [Request Configuration](/concepts/api-specs/request) • [State Variables](/concepts/api-specs/structure#state-variables)

**Spec File** (`customer_details_api.yaml`)

{% code title="customer\_details\_api.yaml" overflow="wrap" %}

```yaml
name: "Customer Details API"

defaults:
  state:
    base_url: https://api.crm.com/v1
  request:
    headers:
      Accept: "application/json"
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  customer_profile:
    state:
      # require() will fail if env var is not set
      customer_id: '{require(env.CUSTOMER_ID)}'

    request:
      # Use state variable as part of URL path
      url: '{state.base_url}/customers/{state.customer_id}/profile'
      method: GET

    response:
      records:
        # Wrap single object in array
        jmespath: "[customer]"
      processors:
        # Add customer_id to the record for reference
        - expression: "state.customer_id"
          output: "record.customer_id"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  customer_profile:
    object: public.customer_profiles

env:
  CUSTOMER_ID: 'cust_12345'
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['API_KEY'] = 'your_api_key'
os.environ['CUSTOMER_ID'] = 'cust_12345'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'customer_profile': ReplicationStream(
            object='public.customer_profiles',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>GraphQL API</summary>

This example shows how to query a GraphQL API.

Learn more: [Request Configuration](/concepts/api-specs/request) • [Response Processing](/concepts/api-specs/response)

**Spec File** (`github_graphql_api.yaml`)

{% code title="github\_graphql\_api.yaml" overflow="wrap" %}

```yaml
name: "GitHub GraphQL API"

defaults:
  state:
    base_url: https://api.github.com
  request:
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {secrets.github_token}"

endpoints:
  repositories:
    state:
      cursor: null  # First page has no cursor
      org_name: '{require(env.GITHUB_ORG)}'

    request:
      url: '{state.base_url}/graphql'
      method: POST
      payload:
        # GraphQL query as a string
        query: |
          query($org: String!, $cursor: String) {
            organization(login: $org) {
              repositories(first: 50, after: $cursor) {
                pageInfo {
                  hasNextPage
                  endCursor
                }
                nodes {
                  name
                  description
                  stargazerCount
                  forkCount
                  createdAt
                  updatedAt
                }
              }
            }
          }
        # GraphQL variables from state
        variables:
          org: '{state.org_name}'
          cursor: '{state.cursor}'

    pagination:
      next_state:
        # Extract next cursor from GraphQL response
        cursor: '{jmespath("response.json", "data.organization.repositories.pageInfo.endCursor")}'
      # GraphQL provides hasNextPage in pageInfo
      stop_condition: 'jmespath("response.json", "data.organization.repositories.pageInfo.hasNextPage") == false'

    response:
      records:
        # Extract nodes array from GraphQL response
        jmespath: "data.organization.repositories.nodes[]"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  repositories:
    object: public.github_repositories

env:
  GITHUB_ORG: 'your_organization'
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['GITHUB_TOKEN'] = 'your_github_token'
os.environ['GITHUB_ORG'] = 'your_organization'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'repositories': ReplicationStream(
            object='public.github_repositories',
            mode=Mode.FULL_REFRESH
        )
    }
)

replication.run()
```

{% endcode %}

</details>

<details>

<summary>Multi-Endpoint Queue Pattern</summary>

This example demonstrates passing data between endpoints using queues.

Learn more: [Queues](/concepts/api-specs/queues) • [Iteration](/concepts/api-specs/request#iteration-looping-requests) • [Processors](/concepts/api-specs/advanced#data-processors)

**Spec File** (`ecommerce_api.yaml`)

{% code title="ecommerce\_api.yaml" overflow="wrap" %}

```yaml
name: "E-commerce API"

# Declare queues to pass data between endpoints
queues:
  - order_ids

defaults:
  state:
    base_url: https://api.shop.com/v1
  request:
    headers:
      Accept: "application/json"
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  # Step 1: List all orders and collect their IDs
  list_orders:
    state:
      page: 1
      limit: 100

    request:
      url: '{state.base_url}/orders'
      method: GET
      parameters:
        page: '{state.page}'
        limit: '{state.limit}'

    pagination:
      next_state:
        page: '{state.page + 1}'
      stop_condition: length(response.records) < state.limit

    response:
      records:
        jmespath: "orders[]"
      processors:
        # Send each order ID to the queue for the next endpoint
        - expression: "record.id"
          output: "queue.order_ids"

  # Step 2: Fetch details for each order from the queue
  order_details:
    description: "Fetch detailed information for each order"

    iterate:
      # Process each order ID from the queue
      over: "queue.order_ids"
      into: "state.order_id"
      concurrency: 10  # Process 10 orders concurrently

    request:
      # Use the current order ID from iteration in URL
      url: '{state.base_url}/orders/{state.order_id}/details'
      method: GET

    response:
      records:
        jmespath: "[order_detail]"
      processors:
        # Add the order_id to the detail record for reference
        - expression: "state.order_id"
          output: "record.order_id"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  list_orders:
    object: public.orders

  order_details:
    object: public.order_details

env:
  SLING_THREADS: 2
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream, Mode
import os

os.environ['API_KEY'] = 'your_api_key'

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'list_orders': ReplicationStream(
            object='public.orders',
            mode=Mode.FULL_REFRESH
        ),
        'order_details': ReplicationStream(
            object='public.order_details',
            mode=Mode.FULL_REFRESH
        )
    },
    env={'SLING_THREADS': '2'}
)

replication.run()
```

{% endcode %}

</details>


# Incremental

Examples of incrementally loading data from APIs to databases using sync state

Incremental loading from APIs allows you to fetch only new or updated data since the last run, reducing API calls and improving performance. Sling uses the `sync` feature to persist state between runs.

Learn more: [Incremental Sync](/concepts/api-specs/advanced#incremental-synchronization) " [State Variables](/concepts/api-specs/structure#state-variables)

## How API Incremental Sync Works

API incremental sync uses the `sync` key to persist values between runs:

1. Define which state variables to persist using `sync: [variable_name]`
2. On first run, use a default value (e.g., 30 days ago)
3. Track the maximum timestamp/ID in each response using processors
4. On subsequent runs, use the persisted value from `sync.variable_name`

The state is stored in the target database by default, or in a location specified by `SLING_STATE`.

## Timestamp-Based Incremental Sync

This is the most common pattern - fetching records updated since the last run.

**Spec File** (`orders_api.yaml`)

{% code title="orders\_api.yaml" overflow="wrap" %}

```yaml
name: "Orders API"

defaults:
  state:
    base_url: https://api.shop.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  orders:
    description: "Get orders updated since last sync"

    # Persist last_updated_at for next run
    sync: [last_updated_at]

    state:
      # Use last sync time if available, otherwise default to 30 days ago
      updated_at_min: >
        {coalesce(
          sync.last_updated_at,
          date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z")
        )}
      limit: 100
      offset: 0

    request:
      url: "{state.base_url}/orders"
      parameters:
        # Filter API to only return records updated after this timestamp
        updated_at_min: "{state.updated_at_min}"
        limit: "{state.limit}"
        offset: "{state.offset}"

    pagination:
      next_state:
        offset: "{state.offset + state.limit}"
      stop_condition: length(response.records) < state.limit

    response:
      records:
        jmespath: "orders[]"
        primary_key: ["order_id"]

      processors:
        # Track the maximum updated_at timestamp across all records
        - expression: "record.updated_at"
          output: "state.last_updated_at"
          aggregation: "maximum"  # Save highest timestamp for next sync

    overrides:
      mode: incremental  # Upsert based on primary_key
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  orders:
    object: public.orders

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}

On first run, this will fetch orders from the last 30 days. On subsequent runs, it will only fetch orders updated since the last run.

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'orders': ReplicationStream(
            object='public.orders'
        )
    },
    env={'SLING_STATE': 'postgres/sling_state.my_api'}
)

# First run: fetches last 30 days
replication.run()

# Subsequent runs: fetches only new/updated orders
replication.run()
```

{% endcode %}

## ID-Based Incremental Sync

For APIs with monotonically increasing IDs, you can sync based on the highest ID.

**Spec File** (`events_api.yaml`)

{% code title="events\_api.yaml" overflow="wrap" %}

```yaml
name: "Events API"

defaults:
  state:
    base_url: https://api.events.com/v2
  request:
    headers:
      X-API-Key: "{secrets.api_key}"

endpoints:
  events:
    description: "Get events with ID greater than last sync"

    # Persist last event ID
    sync: [last_event_id]

    state:
      # Start from last ID, or 0 if first run
      min_id: "{coalesce(sync.last_event_id, 0)}"
      limit: 1000

    request:
      url: "{state.base_url}/events"
      parameters:
        # Only fetch events with ID greater than last sync
        id_gt: "{state.min_id}"
        limit: "{state.limit}"
        sort: "id asc"  # Ensure ascending order

    pagination:
      next_state:
        # Use last record's ID for next page
        min_id: "{response.records[-1].id}"
      stop_condition: length(response.records) < state.limit

    response:
      records:
        jmespath: "events[]"
        primary_key: ["id"]

      processors:
        # Track the highest event ID
        - expression: "record.id"
          output: "state.last_event_id"
          aggregation: "maximum"

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}

## Date-Based Incremental with Iteration

For APIs that require a date parameter, use iteration with sync state.

**Spec File** (`analytics_api.yaml`)

{% code title="analytics\_api.yaml" overflow="wrap" %}

```yaml
name: "Analytics API"

defaults:
  state:
    base_url: https://analytics.example.com/api
  request:
    headers:
      Authorization: "Bearer {secrets.access_token}"

endpoints:
  daily_stats:
    description: "Get daily statistics since last sync"

    # Persist last processed date
    sync: [last_date]

    iterate:
      # Generate dates from last sync to yesterday
      over: >
        range(
          coalesce(sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          date_format(date_add(now(), -1, "day"), "%Y-%m-%d"),
          "1d"
        )
      into: "state.current_date"
      concurrency: 5

    state:
      date: '{date_format(state.current_date, "%Y-%m-%d")}'

    request:
      url: "{state.base_url}/stats/daily/{state.date}"

    response:
      records:
        jmespath: "data[]"
        primary_key: ["metric_id", "date"]

      processors:
        # Track the latest date processed
        - expression: "state.date"
          output: "state.last_date"
          aggregation: "maximum"

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  daily_stats:
    object: analytics.daily_statistics

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}

## Multiple Sync Variables

You can persist multiple values for complex sync scenarios.

**Spec File** (`multi_sync_api.yaml`)

{% code title="multi\_sync\_api.yaml" overflow="wrap" %}

```yaml
name: "Multi-Sync API"

defaults:
  state:
    base_url: https://api.example.com
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  products:
    description: "Track both timestamp and version for sync"

    # Persist multiple sync variables
    sync: [last_updated_at, last_version]

    state:
      updated_at_min: >
        {coalesce(sync.last_updated_at, date_format(date_add(now(), -30, "day"), "%Y-%m-%d"))}
      version_min: "{coalesce(sync.last_version, 0)}"

    request:
      url: "{state.base_url}/products"
      parameters:
        updated_since: "{state.updated_at_min}"
        version_gt: "{state.version_min}"

    pagination:
      next_state:
        offset: "{state.offset + 100}"
      stop_condition: length(response.records) < 100

    response:
      records:
        jmespath: "products[]"
        primary_key: ["product_id"]

      processors:
        # Track both timestamp and version
        - expression: "record.updated_at"
          output: "state.last_updated_at"
          aggregation: "maximum"

        - expression: "record.version"
          output: "state.last_version"
          aggregation: "maximum"

    overrides:
      mode: incremental
```

{% endcode %}

## Using SLING\_STATE

By default, sync state is stored in the target database. You can store it externally using `SLING_STATE`.

**Using Replication with External State**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  orders:
    object: public.orders

  events:
    object: analytics.events

  products:
    object: public.products

env:
  # Store state in S3 instead of target database
  SLING_STATE: AWS_S3/sling/state
```

{% endcode %}

This creates state files at `s3://your-bucket/sling/state/` for each stream.

## Incremental with Full Refresh Fallback

You can mix incremental and full-refresh streams based on your needs.

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  # Incremental sync for large, frequently updated table
  orders:
    object: public.orders
    # Uses incremental mode from spec overrides

  # Full refresh for small, infrequently updated table
  product_categories:
    object: public.categories
    mode: full-refresh  # Override spec's incremental mode

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}

## Best Practices

### 1. Always Provide Sensible Defaults

Use `coalesce()` to handle the first run gracefully:

```yaml
state:
  updated_at_min: >
    {coalesce(
      sync.last_updated_at,
      date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z")
    )}
```

### 2. Use Appropriate Aggregation

Match the aggregation to your sync variable type:

```yaml
processors:
  # For timestamps and IDs
  - expression: "record.updated_at"
    output: "state.last_updated_at"
    aggregation: "maximum"

  # For first/last values
  - expression: "record.cursor"
    output: "state.next_cursor"
    aggregation: "last"
```

### 3. Include Primary Keys

Always specify primary keys for proper upserting:

```yaml
response:
  records:
    primary_key: ["id"]  # Or composite: ["user_id", "event_id"]
```

### 4. Handle Edge Cases

Account for APIs that might return stale data:

```yaml
state:
  # Add a small lookback window to catch late-arriving updates
  updated_at_min: >
    {coalesce(
      date_format(date_add(date_parse(sync.last_updated_at), -1, "hour"), "%Y-%m-%dT%H:%M:%S%z"),
      date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z")
    )}
```

### 5. Use Incremental Mode Override

Set the mode in the spec to ensure consistent behavior:

```yaml
overrides:
  mode: incremental
```

## Combining Incremental with Backfill

You can use both incremental sync and backfill capabilities together.

**First: Backfill Historical Data**

{% code title="backfill.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  orders:
    object: public.orders
    source_options:
      # Backfill 2023 data
      range: '2023-01-01,2023-12-31'

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}

**Then: Switch to Incremental**

{% code title="incremental.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  orders:
    object: public.orders
    # No range specified - uses sync state for incremental loading

env:
  SLING_STATE: postgres/sling_state.my_api
```

{% endcode %}


# Backfill

Examples of backfilling historical data from APIs to databases using range parameters

Backfill mode allows you to load historical data from APIs within a specific date or value range. This is useful when you need to reload data for a particular time period or catch up on historical data.

Learn more: [Incremental Sync](/concepts/api-specs/advanced#sync-state-for-incremental-loads) | [Iteration](/concepts/api-specs/request#iteration-looping-requests)

## How API Backfill Works

API backfill uses the `range` parameter in combination with `context.range_start` and `context.range_end` to iterate over date or value ranges:

1. Define a range in your replication config using `source_options.range`
2. Access the range values in your spec using `context.range_start` and `context.range_end`
3. Use these values with the `range()` function to generate iterations
4. Each iteration fetches data for that specific date or value

> **Note:** API backfill does not support chunking (like database backfill does). The range is processed through iteration.

## Date Range Backfill

This is the most common use case - backfilling API data for a specific date range.

**Spec File** (`analytics_api.yaml`)

{% code title="analytics\_api.yaml" overflow="wrap" %}

```yaml
name: "Analytics API"

defaults:
  state:
    base_url: https://api.analytics.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  daily_events:
    description: "Get daily event data for a date range"

    # Persist the last processed date for incremental runs
    sync: [last_date]

    iterate:
      # Generate date range using context values from replication config
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(date_add(now(), -1, "day"), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"
      concurrency: 5  # Process 5 dates concurrently

    state:
      # Format the date for the API request
      date: '{date_format(state.current_date, "%Y-%m-%d")}'

    request:
      url: "{state.base_url}/events/daily/{state.date}"
      method: GET

    response:
      records:
        jmespath: "data.events[]"
        primary_key: ["event_id"]

      processors:
        # Track the latest date processed for incremental sync
        - expression: "state.date"
          output: "state.last_date"
          aggregation: "maximum"

    overrides:
      mode: incremental  # Use incremental mode for proper upserting
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

defaults:
  mode: full-refresh

streams:
  daily_events:
    object: analytics.events
    source_options:
      # Backfill data for January 2024
      range: '2024-01-01,2024-01-31'
```

{% endcode %}

> **Tip:** The spec uses `overrides.mode: incremental` to ensure data is properly upserted based on `primary_key`, while the replication config can use `mode: full-refresh` or leave it unset.

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'daily_events': ReplicationStream(
            object='analytics.events',
            source_options={
                'range': '2024-01-01,2024-01-31'  # Backfill January 2024
            }
        )
    }
)

replication.run()
```

{% endcode %}

## Month Range Backfill

For APIs that accept month-level granularity, you can iterate by month.

**Spec File** (`sales_api.yaml`)

{% code title="sales\_api.yaml" overflow="wrap" %}

```yaml
name: "Sales API"

defaults:
  state:
    base_url: https://api.sales.com/v2
  request:
    headers:
      X-API-Key: "{secrets.api_key}"

endpoints:
  monthly_sales:
    description: "Get monthly sales data"

    sync: [last_month]

    iterate:
      # Generate monthly range
      over: >
        range(
          coalesce(context.range_start, sync.last_month, date_format(date_add(now(), -6, "month"), "%Y-%m-01")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-01")),
          "1M"
        )
      into: "state.current_month"
      concurrency: 3

    state:
      # Format as YYYY-MM for the API
      month: '{date_format(state.current_month, "%Y-%m")}'

    request:
      url: "{state.base_url}/sales/summary"
      parameters:
        month: "{state.month}"

    response:
      records:
        jmespath: "sales[]"
        primary_key: ["sale_id"]

      processors:
        - expression: "state.month"
          output: "state.last_month"
          aggregation: "maximum"

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  monthly_sales:
    object: sales.monthly_summary
    source_options:
      # Backfill 6 months of data
      range: '2023-07-01,2023-12-31'
```

{% endcode %}

## Numeric ID Range Backfill

For APIs that support ID-based pagination or filtering, you can backfill by ID range.

**Spec File** (`orders_api.yaml`)

{% code title="orders\_api.yaml" overflow="wrap" %}

```yaml
name: "Orders API"

defaults:
  state:
    base_url: https://api.shop.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  orders_by_id:
    description: "Get orders by ID range"

    sync: [last_order_id]

    iterate:
      # Generate numeric range (process 1000 IDs at a time)
      over: >
        range(
          coalesce(context.range_start, sync.last_order_id, "1"),
          coalesce(context.range_end, "999999"),
          "1000"
        )
      into: "state.start_id"
      concurrency: 5

    state:
      # Calculate end ID for this batch
      end_id: '{to_int(state.start_id) + 999}'

    request:
      url: "{state.base_url}/orders"
      parameters:
        id_min: "{state.start_id}"
        id_max: "{state.end_id}"

    response:
      records:
        jmespath: "orders[]"
        primary_key: ["order_id"]

      processors:
        # Track the highest order ID processed
        - expression: "record.order_id"
          output: "state.last_order_id"
          aggregation: "maximum"

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  orders_by_id:
    object: ecommerce.orders
    source_options:
      # Backfill orders with IDs from 10000 to 50000
      range: '10000,50000'
```

{% endcode %}

## Multiple Endpoints with Different Ranges

You can backfill multiple endpoints with different date ranges in a single replication.

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  # Backfill events for last month
  daily_events:
    object: analytics.events
    source_options:
      range: '2024-01-01,2024-01-31'

  # Backfill sales for last quarter
  monthly_sales:
    object: sales.monthly_summary
    source_options:
      range: '2023-10-01,2023-12-31'

  # Backfill specific order ID range
  orders_by_id:
    object: ecommerce.orders
    source_options:
      range: '10000,50000'

env:
  SLING_THREADS: 3  # Process 3 streams concurrently
```

{% endcode %}

## Open-Ended Range

You can specify only the start of a range, leaving the end open to process up to the current date.

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  daily_events:
    object: analytics.events
    source_options:
      # Backfill from 2024-01-01 to yesterday
      range: '2024-01-01,'
```

{% endcode %}

In your spec, handle the open end with a default:

```yaml
iterate:
  over: >
    range(
      coalesce(context.range_start, sync.last_date, "2024-01-01"),
      coalesce(context.range_end, date_format(date_add(now(), -1, "day"), "%Y-%m-%d")),
      "1d"
    )
```

## Best Practices

### 1. Always Use Sync State

Persist the last processed value so subsequent runs can pick up where you left off:

```yaml
sync: [last_date]

processors:
  - expression: "state.date"
    output: "state.last_date"
    aggregation: "maximum"
```

### 2. Set Appropriate Concurrency

Balance API rate limits with performance:

```yaml
iterate:
  concurrency: 5  # Adjust based on API rate limits

request:
  rate: 10  # Max requests per second
```

### 3. Use Incremental Mode Override

Even when backfilling, use incremental mode to properly handle upserts:

```yaml
overrides:
  mode: incremental
```

### 4. Handle API Limits Gracefully

Add retry logic for rate limits and errors:

```yaml
response:
  rules:
    - condition: "response.status == 429"
      action: retry
      max_attempts: 5
      backoff: exponential
```

### 5. Provide Reasonable Defaults

Use `coalesce()` to provide sensible defaults when range is not specified:

```yaml
over: >
  range(
    coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -30, "day"), "%Y-%m-%d")),
    coalesce(context.range_end, date_format(date_add(now(), -1, "day"), "%Y-%m-%d")),
    "1d"
  )
```


# Dynamic Endpoints ⚡

Examples of using dynamic endpoints to programmatically generate multiple endpoint configurations

Dynamic endpoints allow you to programmatically generate multiple endpoint configurations based on runtime data. This is powerful for APIs where the list of available endpoints or resources isn't known until you query the API itself, or when you need to create many similar endpoints without repeating configuration.

Learn more: [Dynamic Endpoints](/concepts/api-specs/dynamic-endpoints) " [State Variables](/concepts/api-specs/structure#state-variables) " [Processors](/concepts/api-specs/advanced#data-processors)

## How Dynamic Endpoints Work

Dynamic endpoints use the `dynamic_endpoints` key to generate multiple endpoint configurations:

1. **Setup Phase** (optional): Execute API calls to fetch data needed for iteration
2. **Iteration Phase**: Loop over a list (from setup or predefined)
3. **Generation Phase**: Create one endpoint configuration per iteration item
4. **Execution Phase**: Run the generated endpoints like normal static endpoints

The generated endpoints are combined with any static endpoints defined in the `endpoints` section.

## Simple Iteration Over Static List

The simplest use case - iterate over a predefined list to create multiple similar endpoints.

**Spec File** (`resources_api.yaml`)

{% code title="resources\_api.yaml" overflow="wrap" %}

```yaml
name: "Resources API"

defaults:
  state:
    base_url: https://api.example.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

dynamic_endpoints:
  # Iterate over a static list of resource types
  - iterate: '["users", "orders", "products"]'
    into: "state.resource_type"

    endpoint:
      # Generate endpoint name from resource type
      name: "{state.resource_type}"
      description: "Fetch {state.resource_type} data"

      request:
        url: "{state.base_url}/{state.resource_type}"
        parameters:
          limit: 100
          offset: 0

      pagination:
        next_state:
          offset: "{state.offset + 100}"
        stop_condition: length(response.records) < 100

      response:
        records:
          jmespath: "data[]"
          primary_key: ["id"]

        processors:
          # Add resource type to each record for tracking
          - expression: "state.resource_type"
            output: "record._resource_type"
```

{% endcode %}

***

**Using Replication**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  # The dynamic endpoints generate: users, orders, products
  '*':
    object: public.{stream_name}
```

{% endcode %}

This creates three endpoints from a single configuration, avoiding repetition.

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'users': ReplicationStream(object='public.users'),
        'orders': ReplicationStream(object='public.orders'),
        'products': ReplicationStream(object='public.products')
    }
)

replication.run()
```

{% endcode %}

## Dynamic Discovery from API

Fetch the list of available resources from the API itself, then create endpoints dynamically.

**Spec File** (`database_api.yaml`)

{% code title="database\_api.yaml" overflow="wrap" %}

```yaml
name: "Database API"

defaults:
  state:
    base_url: https://api.database.com/v2
  request:
    headers:
      X-API-Key: "{secrets.api_key}"

dynamic_endpoints:
  - setup:
      # Fetch list of available tables from the API
      - request:
          url: "{state.base_url}/metadata/tables"
        response:
          processors:
            # Extract table names from response
            - expression: 'jmespath(response.json, "tables[].name")'
              output: "state.available_tables"
              aggregation: last

    # Create one endpoint per table discovered
    iterate: "state.available_tables"
    into: "state.table_name"

    endpoint:
      name: "table_{state.table_name}"
      description: "Data from {state.table_name} table"

      request:
        url: "{state.base_url}/tables/{state.table_name}/rows"
        parameters:
          limit: 1000
          offset: 0

      pagination:
        next_state:
          offset: "{state.offset + 1000}"
        stop_condition: length(response.records) < 1000

      response:
        records:
          jmespath: "rows[]"
          primary_key: ["id"]

        processors:
          # Tag each record with source table name
          - expression: "state.table_name"
            output: "record._source_table"
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# If API returns tables: ["customers", "invoices", "payments"]
# Then these endpoints are generated: table_customers, table_invoices, table_payments

streams:
  table_customers:
    object: staging.customers

  table_invoices:
    object: staging.invoices

  table_payments:
    object: staging.payments
```

{% endcode %}

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

# The API will discover available tables automatically
replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        'table_customers': ReplicationStream(object='staging.customers'),
        'table_invoices': ReplicationStream(object='staging.invoices'),
        'table_payments': ReplicationStream(object='staging.payments')
    }
)

replication.run()
```

{% endcode %}

## Multi-Organization Endpoints

Create separate endpoints for each organization the authenticated user has access to.

**Spec File** (`multi_org_api.yaml`)

{% code title="multi\_org\_api.yaml" overflow="wrap" %}

```yaml
name: "Multi-Organization API"

defaults:
  state:
    base_url: https://api.saas.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.access_token}"

dynamic_endpoints:
  - setup:
      # Fetch organizations the user can access
      - request:
          url: "{state.base_url}/user/organizations"
        response:
          processors:
            # Store the list of organizations
            - expression: 'jmespath(response.json, "organizations")'
              output: "state.org_list"
              aggregation: last

    # Create one endpoint per organization
    iterate: "state.org_list"
    into: "state.org"

    endpoint:
      # Use organization ID in endpoint name
      name: "org_{state.org.id}_events"
      description: "Events for {state.org.name}"

      request:
        url: "{state.base_url}/organizations/{state.org.id}/events"
        parameters:
          limit: 500
          cursor: "{state.cursor}"

      pagination:
        next_state:
          cursor: '{jmespath(response.json, "pagination.next_cursor")}'
        stop_condition: 'is_null(jmespath(response.json, "pagination.next_cursor"))'

      response:
        records:
          jmespath: "events[]"
          primary_key: ["event_id"]

        processors:
          # Add organization context to each record
          - expression: "state.org.id"
            output: "record.organization_id"
          - expression: "state.org.name"
            output: "record.organization_name"
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# If user has access to orgs 123 and 456, endpoints generated:
# org_123_events, org_456_events

streams:
  '*':
    object: public.{stream_name}
```

{% endcode %}

## Geographic Regions with Metadata

Create endpoints for predefined geographic regions with additional metadata.

**Spec File** (`regional_api.yaml`)

{% code title="regional\_api.yaml" overflow="wrap" %}

```yaml
name: "Regional Sales API"

defaults:
  state:
    base_url: https://api.global-sales.com

    # Define regions with metadata
    regions:
      - code: "us-east"
        name: "US East Coast"
        timezone: "America/New_York"
      - code: "us-west"
        name: "US West Coast"
        timezone: "America/Los_Angeles"
      - code: "eu-west"
        name: "Europe West"
        timezone: "Europe/London"
      - code: "apac"
        name: "Asia Pacific"
        timezone: "Asia/Tokyo"

  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

dynamic_endpoints:
  # No setup needed - iterate over predefined regions
  - iterate: "state.regions"
    into: "state.region"

    endpoint:
      name: "sales_{state.region.code}"
      description: "Sales data for {state.region.name}"

      state:
        # Default to last 7 days
        start_date: '{date_format(date_add(now(), -7, "day"), "%Y-%m-%d")}'
        end_date: '{date_format(now(), "%Y-%m-%d")}'

      request:
        url: "{state.base_url}/regions/{state.region.code}/sales"
        parameters:
          date_from: "{state.start_date}"
          date_to: "{state.end_date}"
          page: "{state.page}"
          page_size: 250

      pagination:
        next_state:
          page: "{state.page + 1}"
        stop_condition: 'jmespath(response.json, "has_more") == false'

      response:
        records:
          jmespath: "sales[]"
          primary_key: ["sale_id"]

        processors:
          # Add region metadata to each record
          - expression: "state.region.code"
            output: "record.region_code"
          - expression: "state.region.name"
            output: "record.region_name"
          - expression: "state.region.timezone"
            output: "record.region_timezone"
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# Dynamic endpoints generated: sales_us-east, sales_us-west, sales_eu-west, sales_apac

streams:
  '*':
    object: public.{stream_name}
```

{% endcode %}

## Combining Static and Dynamic Endpoints

Mix static endpoints (for unique resources) with dynamic endpoints (for similar resources).

**Spec File** (`hybrid_api.yaml`)

{% code title="hybrid\_api.yaml" overflow="wrap" %}

```yaml
name: "Hybrid API"

defaults:
  state:
    base_url: https://api.example.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

# Static endpoints for unique resources
endpoints:
  metadata:
    description: "API metadata and configuration"
    request:
      url: "{state.base_url}/metadata"
    response:
      records:
        jmespath: "."
        primary_key: ["api_version"]

  user_profile:
    description: "Authenticated user profile"
    request:
      url: "{state.base_url}/user/profile"
    response:
      records:
        jmespath: "."
        primary_key: ["user_id"]

# Dynamic endpoints for similar resources
dynamic_endpoints:
  - iterate: '["transactions", "accounts", "categories"]'
    into: "state.entity"

    endpoint:
      name: "{state.entity}"
      description: "Fetch {state.entity} data"

      request:
        url: "{state.base_url}/{state.entity}"
        parameters:
          limit: 100

      pagination:
        next_state:
          offset: "{state.offset + 100}"
        stop_condition: length(response.records) < 100

      response:
        records:
          jmespath: "data[]"
          primary_key: ["id"]
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# Total 5 endpoints: 2 static + 3 dynamic

streams:
  # Static endpoints
  metadata:
    object: config.api_metadata

  user_profile:
    object: config.user_profile

  # Dynamic endpoints
  '*':
    object: public.{stream_name}
```

{% endcode %}

## Filtering During Setup

Only generate endpoints for resources matching specific criteria.

**Spec File** (`filtered_api.yaml`)

{% code title="filtered\_api.yaml" overflow="wrap" %}

```yaml
name: "Filtered Tables API"

defaults:
  state:
    base_url: https://api.example.com
  request:
    headers:
      X-API-Key: "{secrets.api_key}"

dynamic_endpoints:
  - setup:
      - request:
          url: "{state.base_url}/schemas/public/tables"
        response:
          processors:
            # Get all tables
            - expression: 'jmespath(response.json, "tables")'
              output: "state.all_tables"
              aggregation: last

            # Filter to only production tables (starting with "prod_")
            - expression: >
                filter(
                  state.all_tables,
                  "starts_with(name, 'prod_')"
                )
              output: "state.filtered_tables"

    # Only create endpoints for filtered tables
    iterate: "state.filtered_tables"
    into: "state.table"

    endpoint:
      name: "{state.table.name}"
      description: "Production table: {state.table.name}"

      request:
        url: "{state.base_url}/tables/{state.table.name}/data"
        parameters:
          limit: 1000

      pagination:
        next_state:
          offset: "{state.offset + 1000}"
        stop_condition: length(response.records) < 1000

      response:
        records:
          jmespath: "rows[]"
          primary_key: ["id"]
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# Only tables matching "prod_*" pattern are available

streams:
  prod_customers:
    object: public.customers

  prod_orders:
    object: public.orders

  # dev_* and test_* tables are excluded
```

{% endcode %}

## Pattern Matching Dynamic Endpoints

Use wildcards to run multiple dynamic endpoints at once.

**Using Replication with Wildcards**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

# Run all organization endpoints for a specific org
streams:
  org_123_*:
    object: analytics.{stream}
    # This matches: org_123_events, org_123_users, etc.

  # Run all regional sales endpoints
  sales_*:
    object: sales.{stream}
    # This matches: sales_us-east, sales_us-west, etc.
```

{% endcode %}

***

**Using Python with Pattern Matching**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream

# You need to know the actual endpoint names or fetch them
replication = Replication(
    source='MY_API',
    target='MY_TARGET_DB',
    streams={
        # Run all org_123 endpoints
        'org_123_events': ReplicationStream(object='analytics.org_123_events'),
        'org_123_users': ReplicationStream(object='analytics.org_123_users'),
        'org_123_metrics': ReplicationStream(object='analytics.org_123_metrics'),
    }
)

replication.run()
```

{% endcode %}

## Best Practices

### 1. Use Meaningful Endpoint Names

Generate descriptive names that clearly identify what each endpoint does:

```yaml
# Good: Clear and descriptive
name: "region_{state.region.code}_daily_sales"

# Avoid: Generic or unclear
name: "endpoint_{state.index}"
```

### 2. Filter in Setup Phase

Only generate endpoints you'll actually use:

```yaml
setup:
  - request:
      url: "{state.base_url}/resources"
    response:
      processors:
        # Filter to only active resources
        - expression: >
            filter(
              jmespath(response.json, "resources"),
              "status == 'active'"
            )
          output: "state.active_resources"
```

### 3. Add Context to Records

Use processors to tag records with metadata from the iteration:

```yaml
processors:
  - expression: "state.org.id"
    output: "record.organization_id"
  - expression: "state.region.code"
    output: "record.region_code"
```

### 4. Validate Iteration Data

Ensure the data you're iterating over is valid:

```yaml
setup:
  - request:
      url: "{state.base_url}/orgs"
    response:
      processors:
        - expression: 'jmespath(response.json, "organizations")'
          output: "state.orgs"
          aggregation: last

        # Log for debugging
        - expression: log("Found " + string(length(state.orgs)) + " organizations")
          output: ""
```

### 5. Use Static Endpoints for Unique Resources

Don't force everything into dynamic endpoints. Use static endpoints for one-off resources:

```yaml
# Static for unique resources
endpoints:
  metadata:
    request:
      url: "{state.base_url}/metadata"

# Dynamic for similar resources
dynamic_endpoints:
  - iterate: "state.tables"
    into: "state.table"
    endpoint:
      name: "{state.table.name}"
```

### 6. Consider Performance

Be mindful of how many endpoints you generate:

```yaml
# Be cautious with large lists
# Generating 1000+ endpoints can impact performance

setup:
  - request:
      url: "{state.base_url}/all_resources"
    response:
      processors:
        # Consider pagination or filtering large lists
        - expression: 'jmespath(response.json, "resources[0:100]")'
          output: "state.limited_resources"
```

## Combining Dynamic Endpoints with Other Features

### With Incremental Sync

```yaml
dynamic_endpoints:
  - iterate: '["users", "orders", "products"]'
    into: "state.resource"

    endpoint:
      name: "{state.resource}"

      # Enable incremental sync for each endpoint
      sync: [last_updated_at]

      state:
        updated_since: >
          {coalesce(
            sync.last_updated_at,
            date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z")
          )}

      request:
        url: "{state.base_url}/{state.resource}"
        parameters:
          updated_since: "{state.updated_since}"

      response:
        records:
          jmespath: "data[]"
          primary_key: ["id"]

        processors:
          - expression: "record.updated_at"
            output: "state.last_updated_at"
            aggregation: "maximum"
```

### With Backfill

```yaml
dynamic_endpoints:
  - iterate: "state.regions"
    into: "state.region"

    endpoint:
      name: "region_{state.region.code}_events"

      sync: [last_date]

      iterate:
        # Support backfill with context.range_start/end
        over: >
          range(
            coalesce(context.range_start, sync.last_date, "2024-01-01"),
            coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
            "1d"
          )
        into: "state.current_date"

      request:
        url: "{state.base_url}/regions/{state.region.code}/events"
        parameters:
          date: "{state.current_date}"
```


# Use Hooks Data

Examples of using data from hooks/steps to drive API iteration

Hooks can be used to query data and pass it to API endpoints for iteration. This pattern is powerful when you need to:

* Fetch a list of IDs or parameters from a database
* Use those values to make corresponding API calls
* Coordinate data between multiple sources

Learn more: [Query Hook](/concepts/hooks/query) | [Store Hook](/concepts/hooks/store) | [API Iteration](/concepts/api-specs/request#iteration-looping-requests)

## How Hook-Driven API Iteration Works

API endpoints can iterate over data provided via hooks using `context.store`:

1. Use a `query` hook with `into` parameter to fetch records and store them in the store
2. Reference the stored records in your API spec using `context.store.variable_name`
3. The API endpoint iterates over each record, making one API call per record
4. Each record's fields are accessible as `state.record_field`

The data flow:

```
Database Query ➡️ Hook Store ➡️ context.store ➡️ API Iteration ➡️ Target Database
```

## Query-Driven Ticker Data Collection

Fetch ticker symbols from a database and retrieve market data for each one from an API.

**Spec File** (`polygon.yaml`)

{% code title="polygon.yaml" overflow="wrap" %}

```yaml
name: Polygon
description: Polygon.io provides real-time and historical market data

defaults:
  state:
    base_url: https://api.polygon.io
  request:
    headers:
      Authorization: 'Bearer {require(secrets.api_key, "Polygon API key required")}'
    rate: 10
    concurrency: 3

endpoints:
  # This endpoint iterates over records provided via context.store
  options_daily_ticker_summary:
    description: "Daily open/close summary for options tickers"
    docs: https://polygon.io/docs/rest/options/aggregates/daily-ticker-summary

    # Require the hook to provide ticker_date_records
    iterate:
      over: require(context.store.ticker_date_records, "Must provide ticker_date_records via hook")
      into: "state.ticker_date_record"
      concurrency: 10

    state:
      # Extract fields from each record
      date: '{date_format(state.ticker_date_record.date, "%Y-%m-%d")}'
      ticker: '{require(state.ticker_date_record.ticker)}'

    request:
      url: '{state.base_url}/v1/open-close/{state.ticker}/{state.date}'

    response:
      records:
        jmespath: "@"  # Single object response
        primary_key: ["symbol", "from"]

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication with Query Hook**

Running with Sling: `sling run -r /path/to/replication.yaml`

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_POLYGON_API
target: MY_TARGET_DB

hooks:
  start:
    # Query database to get list of tickers and dates to fetch
    - type: query
      connection: MY_TARGET_DB
      query: |
        SELECT
          ticker,
          date
        FROM public.active_tickers
        WHERE date >= CURRENT_DATE - INTERVAL '7 days'
        AND data_fetched = false
        ORDER BY date DESC, ticker
        LIMIT 100
      # Store results in the store for API to consume
      into: "ticker_date_records"

streams:
  # The endpoint will iterate over each ticker/date combination
  options_daily_ticker_summary:
    object: market_data.options_daily
```

{% endcode %}

Each record from the query (ticker + date) triggers one API call. The endpoint makes 100 API calls in this example.

***

**Using Python**

{% code title="api\_to\_database.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream
from sling.hooks import HookQuery, HookMap

replication = Replication(
    source='MY_POLYGON_API',
    target='MY_TARGET_DB',
    streams={
        'options_daily_ticker_summary': ReplicationStream(
            object='market_data.options_daily'
        )
    },
    hooks=HookMap(
        start=[
            HookQuery(
                connection='MY_TARGET_DB',
                query='''
                    SELECT ticker, date
                    FROM public.active_tickers
                    WHERE date >= CURRENT_DATE - INTERVAL '7 days'
                    AND data_fetched = false
                    LIMIT 100
                ''',
                into='ticker_date_records'
            )
        ]
    )
)

replication.run()
```

{% endcode %}

## Date Range Generation with Store Hook

Use store hooks to build date ranges that drive API iteration.

**Spec File** (`analytics_api.yaml`)

{% code title="analytics\_api.yaml" overflow="wrap" %}

```yaml
name: "Analytics API"

defaults:
  state:
    base_url: https://api.analytics.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  daily_metrics:
    description: "Get daily metrics for dates provided via context.store"

    iterate:
      over: require(context.store.date_list, "Must provide date_list via hook")
      into: "state.current_date"
      concurrency: 5

    state:
      date: '{date_format(state.current_date, "%Y-%m-%d")}'

    request:
      url: "{state.base_url}/metrics/daily"
      parameters:
        date: "{state.date}"

    response:
      records:
        jmespath: "data.metrics[]"
        primary_key: ["metric_id", "date"]

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication with Store Hook**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

hooks:
  start:
    # Generate list of dates to process
    - type: store
      key: date_list
      value: >
        {range(
          date_format(date_add(now(), -7, "day"), "%Y-%m-%d"),
          date_format(now(), "%Y-%m-%d"),
          "1d"
        )}

streams:
  daily_metrics:
    object: analytics.daily_metrics
```

{% endcode %}

The `range()` function generates an array of dates, stored in `context.store.date_list`, which the API endpoint iterates over.

## Customer IDs with Record Enrichment

Query customer IDs and use them to fetch detailed customer data from an API.

**Spec File** (`customers_api.yaml`)

{% code title="customers\_api.yaml" overflow="wrap" %}

```yaml
name: "Customers API"

defaults:
  state:
    base_url: https://api.customers.com/v2
  request:
    headers:
      X-API-Key: "{secrets.api_key}"

endpoints:
  customer_details:
    description: "Get detailed customer information by ID"

    iterate:
      over: require(context.store.customer_records, "Must provide customer_records via hook")
      into: "state.customer_record"
      concurrency: 10

    state:
      customer_id: '{state.customer_record.customer_id}'
      # Can also access other fields from the query
      region: '{state.customer_record.region}'

    request:
      url: "{state.base_url}/customers/{state.customer_id}"
      parameters:
        include_details: "true"
        region: "{state.region}"

    response:
      records:
        jmespath: "@"
        primary_key: ["customer_id"]

      processors:
        # Add the region from our query to the response
        - expression: state.region
          output: record.source_region

    overrides:
      mode: incremental
```

{% endcode %}

***

**Using Replication**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  customer_details:
    object: public.customer_details

    hooks:
      pre:
        # Query customers that need enrichment
        - type: query
          connection: MY_TARGET_DB
          query: |
            SELECT
              c.customer_id,
              c.region,
              c.last_updated
            FROM public.customers c
            LEFT JOIN public.customer_details cd
              ON c.customer_id = cd.customer_id
            WHERE cd.customer_id IS NULL
              OR c.last_updated > cd.details_fetched_at
            LIMIT 500
          into: "customer_records"

      post:
        # Mark customers as enriched
        - type: query
          connection: MY_TARGET_DB
          if: run.status == "success"
          query: |
            UPDATE public.customer_details
            SET details_fetched_at = CURRENT_TIMESTAMP
            WHERE customer_id IN (
              SELECT DISTINCT customer_id
              FROM public.customer_details
              WHERE details_fetched_at > CURRENT_TIMESTAMP - INTERVAL '1 hour'
            )
```

{% endcode %}

## Processor Output to Hooks Integration

Use processor outputs (`env.*` and `context.store.*`) to pass aggregated data from API responses to hooks for validation, logging, or conditional logic.

**Spec File** (`orders_api.yaml`)

{% code title="orders\_api.yaml" overflow="wrap" %}

```yaml
name: "Orders API"

defaults:
  state:
    base_url: https://api.orders.com/v1
  request:
    headers:
      Authorization: "Bearer {secrets.api_key}"

endpoints:
  orders:
    description: "Fetch orders with metadata tracking"

    request:
      url: "{state.base_url}/orders"
      parameters:
        status: "completed"
        limit: 100

    response:
      records:
        jmespath: "data.orders[]"
        primary_key: ["order_id"]

      processors:
        # Store the maximum timestamp in environment variable
        - expression: "record.updated_at"
          output: "env.MAX_ORDER_TIMESTAMP"
          aggregation: "maximum"

        # Store the first order ID in replication store
        - expression: "record.order_id"
          output: "context.store.first_order_id"
          aggregation: "first"

        # Store the last order ID in replication store
        - expression: "record.order_id"
          output: "context.store.last_order_id"
          aggregation: "last"

        # Track total order count
        - expression: "1"
          output: "context.store.order_count"
          aggregation: "collect"

    overrides:
      mode: full-refresh
```

{% endcode %}

***

**Using Replication with End Hooks**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_ORDERS_API
target: MY_TARGET_DB

streams:
  orders:
    object: sales.orders

hooks:
  start:
    - type: log
      message: "Starting order sync..."

  end:
    # Validate that we got data
    - type: check
      check: store.first_order_id != nil
      on_failure: fail
      message: "No orders were processed!"

    # Log processing summary using values from processors
    - type: log
      message: |
        Orders sync completed successfully!
        ========================================
        First Order ID: {store.first_order_id}
        Last Order ID:  {store.last_order_id}
        Total Orders:   {len(store.order_count)}
        Max Timestamp:  {env.MAX_ORDER_TIMESTAMP}

    # Update metadata table with sync information
    - type: query
      connection: MY_TARGET_DB
      if: execution.status.error == 0
      query: |
        INSERT INTO sales.sync_metadata (
          sync_date,
          first_order_id,
          last_order_id,
          order_count,
          max_timestamp
        ) VALUES (
          CURRENT_TIMESTAMP,
          '{store.first_order_id}',
          '{store.last_order_id}',
          {len(store.order_count)},
          '{env.MAX_ORDER_TIMESTAMP}'
        )

    # Conditional validation based on order count
    - type: check
      check: len(store.order_count) >= 10
      on_failure: log
      message: "Warning: Fewer than 10 orders processed ({len(store.order_count)})"
```

{% endcode %}

***

**Using Python**

{% code title="orders\_sync.py" overflow="wrap" %}

```python
from sling import Replication, ReplicationStream
from sling.hooks import HookLog, HookCheck, HookQuery, HookMap

replication = Replication(
    source='MY_ORDERS_API',
    target='MY_TARGET_DB',
    streams={
        'orders': ReplicationStream(
            object='sales.orders'
        )
    },
    hooks=HookMap(
        start=[
            HookLog(message="Starting order sync...")
        ],
        end=[
            HookCheck(
                check='store.first_order_id != nil',
                on_failure='fail',
                message='No orders were processed!'
            ),
            HookLog(
                message='''Orders sync completed successfully!
First Order ID: {store.first_order_id}
Last Order ID:  {store.last_order_id}
Total Orders:   {len(store.order_count)}
Max Timestamp:  {env.MAX_ORDER_TIMESTAMP}'''
            ),
            HookQuery(
                connection='MY_TARGET_DB',
                if_='execution.status.error == 0',
                query='''INSERT INTO sales.sync_metadata
                         (sync_date, first_order_id, last_order_id, order_count, max_timestamp)
                         VALUES (CURRENT_TIMESTAMP, '{store.first_order_id}',
                                '{store.last_order_id}', {len(store.order_count)},
                                '{env.MAX_ORDER_TIMESTAMP}')'''
            )
        ]
    )
)

replication.run()
```

{% endcode %}

This pattern is useful for:

* Tracking metadata about API responses (record counts, date ranges, etc.)
* Validating data quality before committing to the target
* Logging detailed sync information
* Conditional hook execution based on aggregated values
* Updating audit or metadata tables with sync statistics

## Setting Environment Variables for API Authentication

Use store hooks with `env.*` prefix to set environment variables that are available during API spec rendering. This is particularly powerful when you need to:

* Dynamically compute authentication parameters before API calls
* Inject values into authentication blocks before spec compilation
* Pass computed values to dynamic endpoints
* Use values from database queries or previous API calls in authentication

The key advantage: environment variables set via `env.*` hooks are available **before** the API spec is compiled/rendered, making them usable in authentication blocks and dynamic\_endpoints blocks.

**Spec File** (`api_with_dynamic_auth.yaml`)

{% code title="api\_with\_dynamic\_auth.yaml" overflow="wrap" %}

```yaml
name: "API with Dynamic Auth"
description: "Demonstrates using environment variables set by hooks in authentication"

# Environment variables set by hooks are available here during rendering
authentication:
  type: basic
  username: "{env.COMPUTED_USERNAME}"
  password: "{env.COMPUTED_PASSWORD}"

defaults:
  state:
    base_url: https://api.example.com/v1
  request:
    headers:
      X-Environment: "{env.DEPLOYMENT_ENV}"

endpoints:
  users:
    description: "Fetch users with dynamically configured authentication"

    request:
      url: "{state.base_url}/users"
      parameters:
        # Can also use env vars in request parameters
        api_version: "{env.API_VERSION}"

    response:
      records:
        jmespath: "data.users[]"
        primary_key: ["user_id"]

# Environment variables are also available in dynamic endpoints
dynamic_endpoints:
  - iterate: '{split(env.RESOURCE_LIST, ",")}'
    into: "state.resource_name"
    endpoint:
      name: "data_{state.resource_name}"
      request:
        url: "{state.base_url}/{state.resource_name}"
        headers:
          X-Resource-Token: "{env.RESOURCE_TOKEN}"
      response:
        records:
          jmespath: "data[]"
```

{% endcode %}

***

**Using Pipeline with Environment Variable Setup**

Running with Sling: `sling run -p /path/to/pipeline.yaml`

{% code title="pipeline.yaml" overflow="wrap" %}

```yaml
steps:
  # Step 1: Set environment variables before replication runs
  - type: store
    key: env.COMPUTED_USERNAME
    value: "api_user_prod"

  - type: store
    key: env.COMPUTED_PASSWORD
    value: >
      {base64_encode("secure_password_123")}

  - type: store
    key: env.DEPLOYMENT_ENV
    value: "production"

  - type: store
    key: env.API_VERSION
    value: "v2"

  # Step 2: Query database to get resource list
  - type: query
    connection: MY_CONFIG_DB
    query: |
      SELECT string_agg(resource_name, ',') as resources
      FROM api_resources
      WHERE enabled = true
    into: resource_query

  # Step 3: Set resource list/token as environment variable
  - type: store
    map:
      env.RESOURCE_LIST: '{store.resource_query[0].resources}'
      env.RESOURCE_TOKEN: '{sha256(env.COMPUTED_USERNAME + "-" + now())}'

  # Step 4: Log the configuration
  - type: log
    message: |
      API Configuration:
      - Username: {env.COMPUTED_USERNAME}
      - Environment: {env.DEPLOYMENT_ENV}
      - API Version: {env.API_VERSION}
      - Resources: {env.RESOURCE_LIST}

  # Step 5: Run replication with dynamically configured API
  - type: replication
    path: /path/to/replication.yaml
```

{% endcode %}

***

**Replication File**

{% code title="replication.yaml" overflow="wrap" %}

```yaml
source: MY_API_WITH_DYNAMIC_AUTH
target: MY_TARGET_DB

streams:
  users:
    object: public.api_users

  # Dynamic endpoints will be created based on RESOURCE_LIST
  # e.g., data_customers, data_products, data_orders
```

{% endcode %}

***

**Key Benefits**

1. **Dynamic Authentication**: Compute credentials at runtime (e.g., from database queries, secrets managers, or transformations)
2. **Pre-Compilation Configuration**: Environment variables set via `env.*` hooks are available during API spec rendering, allowing use in:
   * Authentication blocks
   * Dynamic endpoint definitions
   * Default configurations
3. **Separation of Concerns**: Keep sensitive or dynamic values out of spec files
4. **Database-Driven Configuration**: Query databases to determine which endpoints to call or which credentials to use
5. **Multi-Environment Support**: Dynamically configure API behavior based on deployment environment without changing spec files

**Common Use Cases:**

* Rotating API keys fetched from a secrets manager
* Environment-specific endpoints (dev/staging/prod)
* Database-driven resource lists for dynamic endpoints
* Computed authentication tokens based on current state
* Multi-tenant API configurations


# Sling + Python 🚀

Examples of using Sling with Python

## Installation

`pip install sling` or `pip install sling[arrow]` for streaming.

Then you should be able to run `sling --help` from command line or use it in your application as shown below.

See the wrapper source code at <https://github.com/slingdata-io/sling-python>.

## Using the `Replication` class

Run a replication from file:

```python
import yaml
from sling import Replication

# From a YAML file
replication = Replication(file_path="path/to/replication.yaml")
replication.run()

# Or load into object
with open('path/to/replication.yaml') as file:
  config = yaml.load(file, Loader=yaml.FullLoader)

replication = Replication(**config)

replication.run()
```

### Build a replication dynamically

```python
from sling import Replication, ReplicationStream, Mode
from sling.hooks import HookHttp, HookQuery

# build sling replication
streams = {}
for (folder, table_name) in list(folders):
  streams[folder] = ReplicationStream(
    mode=Mode.FULL_REFRESH,
    object=table_name,
    primary_key='_hash_id',
    hooks=dict(
      post=[
        HookQuery(connection='postgres', query='insert ... '),
      ],
    ),
  )

  # or just use dictionaries
  streams[folder] = {
    'mode': Mode.FULL_REFRESH,
    'object': table_name,
    'primary_key': '_hash_id',
    'hooks': {
      'post': [
        {
          'connection': 'postgres',
          'query': 'insert ... ',
        }
      ]
    }
  }

replication = Replication(
  source='aws_s3',
  target='snowflake',
  streams=streams,
  hooks=dict(
    end=[
      HookHTTP(url='http://my.webhook.com/my-stream/log-end'),
    ],
  ),
  env=dict(
    SLING_STREAM_URL_COLUMN='true',
    SLING_LOADED_AT_COLUMN='true',
    SLING_CLI_TOKEN='xxxxxx-xxxxxxx-xxxxxx',
  ),
  debug=True,
)

replication.run()
```

## Using the `Sling` Class

For more direct control and streaming capabilities, you can use the `Sling` class, which mirrors the CLI interface. Available in latest version.

### Basic Usage with `run()` method

```python
import os
from sling import Sling, Mode

# Set postgres & snowflake connection
# see https://docs.slingdata.io/connections/database-connections
os.environ["POSTGRES"] = 'postgres://...'
os.environ["SNOWFLAKE"] = 'snowflake://...'

# Database to database transfer
Sling(
    src_conn="postgres",
    src_stream="public.users",
    tgt_conn="snowflake",
    tgt_object="public.users_copy",
    mode=Mode.FULL_REFRESH
).run()

# Database to file
Sling(
    src_conn="postgres", 
    src_stream="select * from users where active = true",
    tgt_object="file:///tmp/active_users.csv"
).run()

# File to database
Sling(
    src_stream="file:///path/to/data.csv",
    tgt_conn="snowflake",
    tgt_object="public.imported_data"
).run()
```

### Input Streaming - Python Data to Target

> **💡 Tip:** Install `pip install sling[arrow]` for better streaming performance and improved data type handling.

{% hint style="warning" %}
Be careful with large numbers of `Sling` invocations using `input` or `stream()` methods when working with external systems (databases, file systems). Each call re-opens the connection since it invokes the underlying sling binary. For better performance and connection reuse, consider using the `Replication` class instead, which maintains open connections across multiple operations.
{% endhint %}

```python
import os
from sling import Sling
from sling.enum import Format

# Set postgres and SQL Server connection
# see https://docs.slingdata.io/connections/database-connections
os.environ["POSTGRES"] = 'postgres://...'
os.environ["MSSQL"] = 'sqlserver://...'

# Stream Python data to CSV file
data = [
    {"id": 1, "name": "John", "age": 30},
    {"id": 2, "name": "Jane", "age": 25},
    {"id": 3, "name": "Bob", "age": 35}
]

Sling(input=data, tgt_object="file:///tmp/output.csv").run()

# Stream Python data to database
Sling(input=data, tgt_conn="postgres", tgt_object="public.users").run()

# Stream Python data to JSON Lines file
Sling(
    input=data,
    tgt_object="file:///tmp/output.jsonl",
    tgt_options={"format": Format.JSONLINES}
).run()

# Stream from generator (memory efficient for large datasets)
def data_generator():
    for i in range(10000):
        yield {"id": i, "value": f"item_{i}", "timestamp": "2023-01-01"}

Sling(input=data_generator(), tgt_object="file:///tmp/large_dataset.csv").run()
```

> **📊 DataFrame Support:** The `input` parameter accepts lists of dictionaries, pandas DataFrames, or polars DataFrames. DataFrame support preserves data types when using Arrow format.

```python
# Stream pandas DataFrame to database
import pandas as pd

df = pd.DataFrame({
    "id": [1, 2, 3, 4],
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "age": [25, 30, 35, 28],
    "salary": [50000, 60000, 70000, 55000]
})

Sling(input=df, tgt_conn="postgres", tgt_object="public.employees").run()

# Stream polars DataFrame to CSV file
import polars as pl

df = pl.DataFrame({
    "product_id": [101, 102, 103],
    "product_name": ["Laptop", "Mouse", "Keyboard"],
    "price": [999.99, 25.50, 75.00],
    "in_stock": [True, False, True]
})

Sling(input=df,  tgt_object="file:///tmp/products.csv").run()

# DataFrame with column selection
Sling(
    input=df,
    select=["product_name", "price"],  # Only export specific columns
    tgt_conn="mssql",
    tgt_object="dbo.product_prices"
).run()
```

### Output Streaming with `stream()`

```python
import os
from sling import Sling

# Set postgres connection
# see https://docs.slingdata.io/connections/database-connections
os.environ["POSTGRES"] = 'postgres://...'

# Stream data from database
sling = Sling(src_conn="postgres", src_stream="public.users", limit=1000)

for record in sling.stream():
    print(f"User: {record['name']}, Age: {record['age']}")

# Stream data from file
sling = Sling(src_stream="file:///path/to/data.csv")

# Process records one by one (memory efficient)
for record in sling.stream():
    # Process each record
    processed_data = transform_record(record)
    # Could save to another system, send to API, etc.

# Stream with parameters
sling = Sling(
    src_conn="postgres",
    src_stream="public.orders",
    select=["order_id", "customer_name", "total"],
    where="total > 100",
    limit=500
)

records = list(sling.stream())
print(f"Found {len(records)} high-value orders")
```

### High-Performance Streaming with `stream_arrow()`

> **🚀 Performance:** The `stream_arrow()` method provides the highest performance streaming with full data type preservation by using Apache Arrow's columnar format. Requires `pip install sling[arrow]`.

> **📊 Type Safety:** Unlike `stream()` which may convert data types during CSV serialization, `stream_arrow()` preserves exact data types including integers, floats, timestamps, and more.

```python
import os
from sling import Sling

# Set postgres connection  
# see https://docs.slingdata.io/connections/database-connections
os.environ["POSTGRES"] = 'postgres://...'

# Basic Arrow streaming from database
sling = Sling(src_conn="postgres", src_stream="public.users", limit=1000)

# Get Arrow RecordBatchStreamReader for maximum performance
reader = sling.stream_arrow()

# Convert to Arrow Table for analysis
table = reader.read_all()
print(f"Received {table.num_rows} rows with {table.num_columns} columns")
print(f"Column names: {table.column_names}")
print(f"Schema: {table.schema}")

# Convert to pandas DataFrame with preserved types
if table.num_rows > 0:
    df = table.to_pandas()
    print(df.dtypes)  # Shows preserved data types

# Stream Arrow file with type preservation
sling = Sling(
    src_stream="file:///path/to/data.arrow",
    src_options={"format": "arrow"}
)

reader = sling.stream_arrow()
table = reader.read_all()

# Access columnar data directly (very efficient)
for column_name in table.column_names:
    column = table.column(column_name)
    print(f"{column_name}: {column.type}")

# Process Arrow batches for large datasets (memory efficient)
sling = Sling(src_conn="postgres", src_stream="select * from large_table")

reader = sling.stream_arrow()
for batch in reader:
    # Process each batch separately to manage memory
    print(f"Processing batch with {batch.num_rows} rows")
    # Convert batch to pandas if needed
    batch_df = batch.to_pandas()
    # Process batch_df...

# Round-trip with Arrow format preservation
import pandas as pd

# Write DataFrame to Arrow file with type preservation
df = pd.DataFrame({
    "id": [1, 2, 3],
    "amount": [100.50, 250.75, 75.25],
    "timestamp": pd.to_datetime(["2023-01-01", "2023-01-02", "2023-01-03"]),
    "active": [True, False, True]
})

Sling(
    input=df,
    tgt_object="file:///tmp/data.arrow",
    tgt_options={"format": "arrow"}
).run()

# Read back with full type preservation
sling = Sling(
    src_stream="file:///tmp/data.arrow",
    src_options={"format": "arrow"}
)

reader = sling.stream_arrow()
restored_table = reader.read_all()
restored_df = restored_table.to_pandas()

# Types are exactly preserved (no string conversion)
print(restored_df.dtypes)
assert restored_df['active'].dtype == 'bool'
assert 'datetime64' in str(restored_df['timestamp'].dtype)
```

### Round-trip Examples

```python
import os
from sling import Sling

# Set postgres connection
# see https://docs.slingdata.io/connections/database-connections
os.environ["POSTGRES"] = 'postgres://...'

# Python → File → Python
original_data = [
    {"id": 1, "name": "Alice", "score": 95.5},
    {"id": 2, "name": "Bob", "score": 87.2}
]

# Step 1: Python data to file
sling_write = Sling(input=original_data, tgt_object="file:///tmp/scores.csv")
sling_write.run()

# Step 2: File back to Python
sling_read = Sling(src_stream="file:///tmp/scores.csv")
loaded_data = list(sling_read.stream())

# Python → Database → Python (with transformations)
sling_to_db = Sling(
    input=original_data,
    tgt_conn="postgres",
    tgt_object="public.temp_scores"
)
sling_to_db.run()

sling_from_db = Sling(
    src_conn="postgres", 
    src_stream="select *, score * 1.1 as boosted_score from public.temp_scores",
)
transformed_data = list(sling_from_db.stream())
```

```python
# DataFrame → Database → DataFrame (with pandas/polars)
import pandas as pd

# Start with pandas DataFrame
df = pd.DataFrame({
    "user_id": [1, 2, 3],
    "purchase_amount": [100.50, 250.75, 75.25],
    "category": ["electronics", "clothing", "books"]
})

# Write DataFrame to database
Sling(input=df, tgt_conn="postgres", tgt_object="public.purchases").run()

# Read back with SQL transformations as pandas DataFrame
sling_query = Sling(
    src_conn="postgres",
    src_stream="""
        SELECT category, 
               COUNT(*) as purchase_count,
               AVG(purchase_amount) as avg_amount
        FROM public.purchases 
        GROUP BY category
    """
)
summary_data = list(sling_query.stream())
summary_df = pd.DataFrame(summary_data)
print(summary_df)
```

## Using the `Pipeline` class

Run a [Pipeline](https://docs.slingdata.io/concepts/pipeline):

```python
from sling import Pipeline
from sling.hooks import StepLog, StepCopy, StepReplication, StepHTTP, StepCommand

# From a YAML file
pipeline = Pipeline(file_path="path/to/pipeline.yaml")
pipeline.run()

# Or using Hook objects for type safety
pipeline = Pipeline(
    steps=[
        StepLog(message="Hello world"),
        StepCopy(from_="sftp//path/to/file", to="aws_s3/path/to/file"),
        StepReplication(path="path/to/replication.yaml"),
        StepHTTP(url="https://trigger.webhook.com"),
        StepCommand(command=["ls", "-l"], print_output=True)
    ],
    env={"MY_VAR": "value"}
)
pipeline.run()

# Or programmatically using dictionaries
pipeline = Pipeline(
    steps=[
        {"type": "log", "message": "Hello world"},
        {"type": "copy", "from": "sftp//path/to/file", "to": "aws_s3/path/to/file"},
        {"type": "replication", "path": "path/to/replication.yaml"},
        {"type": "http", "url": "https://trigger.webhook.com"},
        {"type": "command", "command": ["ls", "-l"], "print": True}
    ],
    env={"MY_VAR": "value"}
)
pipeline.run()
```

## Building API Specs with `ApiSpec`

Build [API Spec](https://docs.slingdata.io/concepts/api-specs) YAML files programmatically with type checking and validation. API specs define how Sling extracts data from REST APIs.

See the full [Python SDK reference](https://docs.slingdata.io/concepts/api/python-sdk) for all classes and enums.

```python
from sling.api_spec import (
    ApiSpec, Endpoint, Request, Pagination, Response, Records,
    Processor, Rule, Iterate, RuleAction, AggregationType, BackoffType,
)

spec = ApiSpec(
    name="My API",
    description="Extract data from My API",
    queues=["user_ids"],
    defaults=Endpoint(
        state={"base_url": "https://api.example.com/v1", "limit": 100},
        request=Request(
            headers={
                "Accept": "application/json",
                "Authorization": 'Bearer {require(secrets.api_key, "API key required")}',
            },
            rate=5,
            concurrency=3,
        ),
        response=Response(
            rules=[
                Rule(
                    action=RuleAction.RETRY,
                    condition="response.status == 429",
                    max_attempts=5,
                    backoff=BackoffType.EXPONENTIAL,
                    backoff_base=2,
                ),
            ],
        ),
    ),
    endpoints={
        "users": Endpoint(
            description="List all users",
            state={"offset": 0},
            request=Request(
                url="{state.base_url}/users",
                parameters={"limit": "{state.limit}", "offset": "{state.offset}"},
            ),
            pagination=Pagination(
                next_state={"offset": "{state.offset + state.limit}"},
                stop_condition="length(response.records) < state.limit",
            ),
            response=Response(
                records=Records(jmespath="data[]", primary_key=["id"]),
                processors=[
                    Processor(expression="record.id", output="queue.user_ids"),
                ],
            ),
        ),
        "user_details": Endpoint(
            description="Get details for each user",
            depends_on=["users"],
            iterate=Iterate(over="queue.user_ids", into="state.user_id", concurrency=5),
            request=Request(url="{state.base_url}/users/{state.user_id}"),
            response=Response(
                records=Records(jmespath="data", primary_key=["id"]),
            ),
        ),
    },
)

# Validate
errors = spec.validate()
assert errors == [], errors

# Write to file
spec.to_yaml_file("my_api.yaml")
```

### Parse and Modify an Existing Spec

```python
from sling.api_spec import ApiSpec, Endpoint, Request, Response, Records

spec = ApiSpec.parse_file("path/to/spec.yaml")
print(spec.name)
print(list(spec.endpoints.keys()))

# Add a new endpoint
spec.endpoints["new_endpoint"] = Endpoint(
    description="A new endpoint",
    request=Request(url="{state.base_url}/new"),
    response=Response(records=Records(jmespath="data[]", primary_key=["id"])),
)

spec.to_yaml_file("updated_spec.yaml")
```

### Incremental Sync Example

```python
from sling.api_spec import (
    ApiSpec, Endpoint, Request, Pagination, Response, Records,
    Processor, AggregationType,
)

spec = ApiSpec(
    name="Incremental API",
    defaults=Endpoint(
        state={"base_url": "https://api.example.com/v1", "limit": 100},
        request=Request(
            headers={"Authorization": 'Bearer {require(secrets.api_key, "API key required")}'},
        ),
    ),
    endpoints={
        "orders": Endpoint(
            description="Incremental order sync",
            state={
                "offset": 0,
                "updated_since": '{coalesce(sync.last_updated, date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%SZ"))}',
            },
            sync=["last_updated"],
            request=Request(
                url="{state.base_url}/orders",
                parameters={
                    "updated_since": "{state.updated_since}",
                    "limit": "{state.limit}",
                    "offset": "{state.offset}",
                },
            ),
            pagination=Pagination(
                next_state={"offset": "{state.offset + state.limit}"},
                stop_condition="length(response.records) < state.limit",
            ),
            response=Response(
                records=Records(
                    jmespath="data[]",
                    primary_key=["id"],
                    update_key="updated_at",
                ),
                processors=[
                    Processor(
                        expression="record.updated_at",
                        output="state.last_updated",
                        aggregation=AggregationType.MAXIMUM,
                    ),
                ],
            ),
        ),
    },
)

spec.to_yaml_file("incremental_api.yaml")
```


# Database Connections

## Supported Connections

* [ADBC (Arrow)](/connections/database-connections/adbc)
* [Azure Table](/connections/database-connections/azuretable)
* [Clickhouse](/connections/database-connections/clickhouse)
* [Cloudflare D1](/connections/database-connections/d1)
* [Databricks](/connections/database-connections/databricks)
* [DB2](/connections/database-connections/db2)
* [DuckDB](/connections/database-connections/duckdb)
* [Elasticsearch](/connections/database-connections/elasticsearch)
* [Exasol](/connections/database-connections/exasol)
* [Fabric](/connections/database-connections/fabric)
* [Google BigQuery](/connections/database-connections/bigquery)
* [Google BigTable](/connections/database-connections/bigtable)
* [MariaDB](/connections/database-connections/mariadb)
* [MongoDB](/connections/database-connections/mongodb)
* [MotherDuck](/connections/database-connections/motherduck)
* [MySQL](/connections/database-connections/mysql)
* [ODBC](/connections/database-connections/odbc)
* [Oracle](/connections/database-connections/oracle)
* [PostgreSQL](/connections/database-connections/postgres)
* [Prometheus](/connections/database-connections/prometheus)
* [Proton](/connections/database-connections/proton)
* [Redshift](/connections/database-connections/redshift)
* [ScyllaDB](/connections/database-connections/scylladb)
* [Snowflake](/connections/database-connections/snowflake)
* [SQL Server](/connections/database-connections/sqlserver)
* [SQLite](/connections/database-connections/sqlite)
* [StarRocks](/connections/database-connections/starrocks)
* [Trino](/connections/database-connections/trino)


# ADBC (Arrow)

Connect & Ingest data from / to multiple databases via Arrow Database Connectivity (ADBC)

Arrow Database Connectivity (ADBC) provides a standardized interface for accessing various databases using the Apache Arrow columnar format. ADBC enables efficient, high-performance data movement with zero-copy semantics and native Arrow support (*v1.5.2+*)

## Supported Database Types

Sling supports the following databases via ADBC drivers:

* **PostgreSQL** - Full support via ADBC PostgreSQL driver
* **MySQL** - Support via ADBC MySQL driver (v9.4+)
* **SQL Server** - Full support via ADBC SQL Server driver
* **Snowflake** - Full support via ADBC Snowflake driver
* **SQLite** - Support via ADBC SQLite driver
* **DuckDB** - Full support via ADBC DuckDB driver
* **BigQuery** - Full support via ADBC BigQuery driver
* **Trino** - Support via ADBC Trino driver (v4.0+)

## Setup

Using ADBC requires two components:

1. **ADBC Driver Manager** — a shared library (`libadbc_driver_manager.so` / `.dylib` / `.dll`) that loads and manages drivers
2. **Database Driver** — the native ADBC driver for your specific database (e.g., DuckDB, PostgreSQL)

**Sling downloads both automatically the first time you use an ADBC connection** (*v1.5.24+*), so in most cases no setup is needed — just add `use_adbc: true` to your connection and run it:

```bash
$ sling conns test POSTGRES
INF downloading dbc 0.3.0 for linux/amd64
INF installing ADBC driver postgresql via dbc
INF downloading ADBC driver manager 1.12.0 for linux/amd64
INF success!
```

Downloads happen once and are cached under `~/.sling/bin/`:

| Component        | Location                                                                     |
| ---------------- | ---------------------------------------------------------------------------- |
| `dbc` CLI        | `~/.sling/bin/dbc/<version>/`                                                |
| Driver manager   | `~/.sling/bin/adbc/<version>/`                                               |
| Database drivers | standard `dbc` locations (see [Step 2](#step-2-install-the-database-driver)) |

Anything already installed on the system is preferred over downloading. Sling only fetches what is missing.

{% hint style="info" %}
To disable automatic downloads (e.g. in an air-gapped environment), set `SLING_DISABLE_DBC_AUTO_INSTALL=true` and install the components manually using the instructions below.
{% endhint %}

The rest of this section covers **manual installation**, which you need only if you have disabled auto-install, are offline, or want to pin a specific build.

### Step 1: Install the ADBC Driver Manager

The driver manager is a shared library that Sling loads at runtime.

{% tabs %}
{% tab title="macOS" %}
Using Conda (recommended):

```bash
conda install -c conda-forge libadbc-driver-manager
```

Or with Homebrew + Conda:

```bash
brew install --cask mambaforge
mamba install -c conda-forge libadbc-driver-manager
```

The library is installed to your conda environment's `lib/` directory (e.g., `~/mambaforge/lib/libadbc_driver_manager.dylib`). Sling auto-detects common conda paths.
{% endtab %}

{% tab title="Linux (apt — amd64 only)" %}
The Apache Arrow apt repository provides pre-built packages for **amd64** (x86\_64):

```bash
# Add Apache Arrow apt repository
sudo apt update
sudo apt install -y ca-certificates lsb-release wget
wget -q https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb -O /tmp/arrow.deb
sudo apt install -y /tmp/arrow.deb && rm /tmp/arrow.deb
sudo apt update

# Install the driver manager
sudo apt install -y libadbc-driver-manager110
```

{% hint style="warning" %}
The `libadbc-driver-manager` apt package is only available for **amd64**. For arm64/aarch64, use conda-forge (see below).
{% endhint %}
{% endtab %}

{% tab title="Linux (conda — amd64 & arm64)" %}
Conda-forge provides packages for both amd64 and arm64:

```bash
conda install -c conda-forge libadbc-driver-manager
```

The library is installed to your conda environment's `lib/` directory. Sling auto-detects common conda paths (`~/mambaforge/lib/`, `~/miniforge3/lib/`, `~/miniconda3/lib/`).
{% endtab %}

{% tab title="Windows" %}
Using Conda:

```powershell
conda install -c conda-forge libadbc-driver-manager
```

If Sling cannot find the library, set the `ADBC_DRIVER_MANAGER_LIB` environment variable to the full path of `adbc_driver_manager.dll`.
{% endtab %}
{% endtabs %}

{% hint style="info" %}
Sling auto-detects the driver manager from common installation paths. If it cannot find the library, you can set the `ADBC_DRIVER_MANAGER_LIB` environment variable to the full path:

```bash
export ADBC_DRIVER_MANAGER_LIB=/path/to/libadbc_driver_manager.so    # Linux
export ADBC_DRIVER_MANAGER_LIB=/path/to/libadbc_driver_manager.dylib  # macOS
```

{% endhint %}

### Step 2: Install the Database Driver

Install the native ADBC driver for your target database using the [`dbc` CLI tool](https://docs.columnar.tech/dbc/):

```bash
# Install dbc
# macOS (Homebrew)
brew install columnar-tech/tap/dbc

# Linux/macOS (shell script)
curl -LsSf https://dbc.columnar.tech/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://dbc.columnar.tech/install.ps1 | iex"
```

Then install drivers for your target databases:

```bash
dbc install duckdb
dbc install postgresql
dbc install mysql
dbc install snowflake
dbc install trino
# ... etc
```

Drivers are installed to:

* **macOS**: `~/Library/Application Support/ADBC/Drivers/`
* **Linux**: `~/.config/adbc/drivers/`
* **Windows**: `%APPDATA%\adbc\drivers\`

Sling auto-discovers installed drivers from these locations.

### Quick Start Example (DuckDB)

With auto-install, the complete setup on any platform is:

```bash
# 1. Set up connection
sling conns set DUCKDB type=duckdb instance=/tmp/test.db use_adbc=true

# 2. Test it — the driver manager and DuckDB driver download on first use
sling conns test DUCKDB
```

If you have disabled auto-install, install the components first:

```bash
# 1. Install driver manager
conda install -c conda-forge libadbc-driver-manager

# 2. Install DuckDB driver
curl -LsSf https://dbc.columnar.tech/install.sh | sh   # macOS/Linux
dbc install duckdb

# 3. Set up connection and test
sling conns set DUCKDB type=duckdb instance=/tmp/test.db use_adbc=true
sling conns test DUCKDB
```

### Manual Driver Installation

If you prefer not to use `dbc`, you can install ADBC driver libraries manually:

{% tabs %}
{% tab title="macOS" %}

```bash
# Install via Conda
conda install conda-forge::libadbc-driver-postgresql

# Or download .dylib from ADBC releases and place in a known location
cp libadbc_driver_postgresql.dylib ~/Library/Application\ Support/ADBC/Drivers/
```

{% endtab %}

{% tab title="Linux" %}

```bash
# Ubuntu/Debian amd64 (after adding Apache Arrow APT repository)
sudo apt install libadbc-driver-postgresql-dev

# Or via Conda (amd64 and arm64)
conda install conda-forge::libadbc-driver-postgresql
```

{% endtab %}

{% tab title="Windows" %}
Download the appropriate `.dll` from the [ADBC releases](https://github.com/apache/arrow-adbc/releases) or install via Conda:

```powershell
conda install conda-forge::libadbc-driver-postgresql
```

{% endtab %}
{% endtabs %}

### Driver Discovery

Sling searches for ADBC driver libraries automatically in the following order:

1. **Explicit `driver` property** in the connection configuration
2. **`ADBC_DRIVER_PATH` environment variable** — additional directories to search (colon-separated on Unix, semicolon-separated on Windows)
3. **Standard installation paths:**

| Platform | Paths                                                                                                                          |
| -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| macOS    | `~/Library/Application Support/ADBC/Drivers`, `~/.dbc/drivers`, `/usr/local/lib`, `/opt/homebrew/lib`                          |
| Linux    | `~/.local/share/ADBC/Drivers`, `~/.config/adbc/drivers`, `~/.dbc/drivers`, `/usr/lib`, `/usr/local/lib`                        |
| Windows  | `%APPDATA%\adbc\drivers`, `%LOCALAPPDATA%\ADBC\Drivers`, `~/.config/adbc/drivers`, `~/.dbc/drivers`, `%ProgramFiles%\ADBC\lib` |

4. **Automatic install** — if the driver is still not found, Sling installs it with `dbc` (downloading `dbc` itself if needed)

Sling looks for driver files matching `*{driver_name}*` (e.g., `libduckdb.so`, `libadbc_driver_postgresql.dylib`).

If the driver is in a non-standard location, you can either set `ADBC_DRIVER_PATH` or specify the full path directly in your connection:

```bash
# Add custom search paths
export ADBC_DRIVER_PATH="/opt/custom/lib:/another/path"
```

```yaml
# Or specify the driver path directly in the connection
connections:
  POSTGRES:
    type: postgres
    host: localhost
    use_adbc: true
    driver: /opt/custom/lib/libadbc_driver_postgresql.dylib
```

## Enabling ADBC on Connections

To use ADBC with your existing database connections, simply add `use_adbc: true` to your connection configuration. This allows you to keep your original connection properties and format while enabling ADBC's high-performance data transfer.

### Connection Properties

The following ADBC-specific properties can be added to any supported database connection:

* `use_adbc` (optional) -> Enable ADBC driver for this connection (`true` or `false`). Default is `false`.
* `adbc_uri` (optional) -> Override the automatically constructed ADBC URI. Sling automatically builds the ADBC URI from your connection properties, but you can specify a custom URI if needed.
* `driver` (optional) -> Explicit path to the ADBC driver library file (e.g., `/usr/local/lib/libadbc_driver_postgresql.dylib`). If not set, Sling auto-discovers the driver.

## ADBC Environment Variables

These control how Sling locates and downloads the ADBC components. (To define a *connection* with an environment variable, see [Environment Variable](#environment-variable) below.)

| Variable                         | Purpose                                                                                   |
| -------------------------------- | ----------------------------------------------------------------------------------------- |
| `ADBC_DRIVER_MANAGER_LIB`        | Full path to the driver manager library. Overrides auto-detection and auto-download.      |
| `ADBC_DRIVER_PATH`               | Extra directories to search for database drivers (`:`-separated on Unix, `;` on Windows). |
| `ADBC_DRIVER_MANAGER_VERSION`    | Driver manager version to download. Defaults to `1.12.0`.                                 |
| `DBC_PATH`                       | Full path to an existing `dbc` binary, instead of downloading one.                        |
| `DBC_VERSION`                    | `dbc` version to download. Defaults to `0.3.0`.                                           |
| `SLING_DISABLE_DBC_AUTO_INSTALL` | Set to `true` to disable all automatic downloads.                                         |

## Troubleshooting

### "failed to load ADBC driver manager library"

This means the driver manager shared library could not be loaded. Sling normally downloads it automatically, so this usually indicates auto-install is disabled, the machine is offline, or the library is present but incompatible with the system (see the next two sections).

**Fix:** Install the driver manager (see [Step 1](#step-1-install-the-adbc-driver-manager) above), or set the `ADBC_DRIVER_MANAGER_LIB` environment variable to the full path of the library.

```bash
# Find the library
find / -name "libadbc_driver_manager*" 2>/dev/null

# Set the path
export ADBC_DRIVER_MANAGER_LIB=/path/to/libadbc_driver_manager.so
```

### Linux: "version \`GLIBCXX\_3.4.29' not found"

The prebuilt driver manager is built with GCC 11 and needs `GLIBCXX_3.4.29`. Older distributions ship an older C++ runtime — Ubuntu 20.04 and Debian 11 provide `GLIBCXX_3.4.28`, one version short.

Distributions from 2021 onward (Ubuntu 22.04+, Debian 12+, RHEL/Rocky 9+, Amazon Linux 2023) are unaffected and need nothing here.

Sling cannot fix this from inside a running process: the dynamic loader binds the driver manager's dependency to whichever `libstdc++.so.6` is already loaded, and by then the system copy is in place. So Sling downloads a compatible library and prints the command to use it:

```
The ADBC driver manager needs a newer C++ runtime (libstdc++) than this system
provides (requires GLIBCXX_3.4.29). A compatible libstdc++ has been downloaded
to ~/.sling/bin/adbc/1.12.0/libstdc++.so.6 — re-run with it preloaded:

    LD_PRELOAD=~/.sling/bin/adbc/1.12.0/libstdc++.so.6 sling conns test POSTGRES
```

**Fix:** re-run with `LD_PRELOAD` as shown. To make it permanent, export it in your shell profile:

```bash
export LD_PRELOAD="$HOME/.sling/bin/adbc/1.12.0/libstdc++.so.6"
```

Alternatively, install a newer system `libstdc++` (`conda install -c conda-forge libstdcxx`), or point `ADBC_DRIVER_MANAGER_LIB` at a build compatible with your system.

You can check what your system provides with:

```bash
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep -o 'GLIBCXX_3\.4\.[0-9]*' | sort -V | tail -1
```

### Linux: "version \`GLIBC\_2.xx' not found"

The C runtime (glibc) itself is older than the driver manager requires. Unlike the `libstdc++` case above, this cannot be worked around with a downloaded library — glibc cannot be preloaded safely.

**Fix:** run Sling on a newer distribution, or set `ADBC_DRIVER_MANAGER_LIB` to a driver manager built for your system.

### "Must provide 'driver' parameter"

This means Sling could not find or install the database-specific ADBC driver (e.g., the DuckDB or PostgreSQL driver). Sling normally installs it automatically with `dbc`, so this usually means auto-install is disabled or the machine is offline.

**Fix:** Install the driver with `dbc install <driver_name>`, or set `ADBC_DRIVER_PATH` to the directory containing the driver, or set the `driver` property in your connection config.

```bash
# Install the driver
dbc install duckdb

# Or point Sling to the driver location
export ADBC_DRIVER_PATH=/path/to/drivers
```

### Windows: driver not found after dbc install

Sling searches both `%APPDATA%\adbc\drivers` (Roaming, where `dbc` installs) and `%LOCALAPPDATA%\ADBC\Drivers` (Local). If your driver is somewhere else, set `ADBC_DRIVER_PATH`:

```powershell
$env:ADBC_DRIVER_PATH = "C:\path\to\drivers"
```

Or specify the driver path directly in your connection:

```yaml
connections:
  DUCKDB:
    type: duckdb
    instance: C:/path/to/file.db
    use_adbc: true
    driver: C:/Users/me/AppData/Roaming/adbc/drivers/duckdb_windows_amd64_v1.4.4/duckdb.dll
```

## Database-Specific Examples

### PostgreSQL with ADBC

```yaml
connections:
  POSTGRES:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    port: 5432
    use_adbc: true
    # adbc_uri: "postgresql://myuser:mypass@localhost:5432/mydatabase"  # optional override
```

ADBC URI format: `postgresql://user:password@host:port/database`

**Official Documentation:** [Apache ADBC PostgreSQL Driver](https://arrow.apache.org/adbc/current/driver/postgresql.html)

### MySQL with ADBC

```yaml
connections:
  MYSQL:
    type: mysql
    host: localhost
    user: root
    password: mypass
    database: mydb
    port: 3306
    use_adbc: true
    # adbc_uri: "root@tcp(localhost:3306)/mydb"  # optional override
```

ADBC URI format: `user@tcp(host:port)/database`

**Official Documentation:** [Apache ADBC MySQL Driver](https://docs.adbc-drivers.org/drivers/mysql/index.html)

### SQL Server with ADBC

```yaml
connections:
  MSSQL:
    type: sqlserver
    host: localhost
    user: sa
    password: mypass
    database: master
    port: 1433
    use_adbc: true
    # adbc_uri: "mssql://sa:mypass@localhost:1433/master"  # optional override
```

ADBC URI format: `mssql://user:password@host:port/database`

**Official Documentation:** [Apache ADBC SQL Server Driver](https://docs.adbc-drivers.org/drivers/mssql/index.html)

### Snowflake with ADBC

```yaml
connections:
  SNOWFLAKE:
    type: snowflake
    account: myaccount.us-east-1
    user: myuser
    password: mypass
    database: mydb
    schema: myschema
    use_adbc: true
    # adbc_uri: "snowflake://myuser:mypass@myaccount/mydb/myschema"  # optional override
```

ADBC URI format: `snowflake://user:password@account/database/schema`

**Official Documentation:** [Apache ADBC Snowflake Driver](https://arrow.apache.org/adbc/current/driver/snowflake.html)

### DuckDB with ADBC

```yaml
connections:
  DUCKDB:
    type: duckdb
    instance: /path/to/file.db
    use_adbc: true
    # adbc_uri: "duckdb:///path/to/file.db"  # optional override
```

ADBC URI format: `duckdb:///path/to/file.db` or `duckdb://:memory:`

**Official Documentation:** [Apache ADBC DuckDB Driver](https://arrow.apache.org/adbc/current/driver/duckdb.html)

### SQLite with ADBC

```yaml
connections:
  SQLITE:
    type: sqlite
    instance: /path/to/file.db
    use_adbc: true
    # adbc_uri: "sqlite:///path/to/file.db"  # optional override
```

ADBC URI format: `sqlite:///path/to/file.db`

**Official Documentation:** [Apache ADBC SQLite Driver](https://arrow.apache.org/adbc/current/driver/sqlite.html)

### BigQuery with ADBC

```yaml
connections:
  BIGQUERY:
    type: bigquery
    project: myproject
    dataset: mydataset
    key_file: /path/to/service.account.json
    use_adbc: true
    # adbc_uri: "bigquery://myproject"  # optional override
```

ADBC URI format: `bigquery://project`

**Official Documentation:** [Apache ADBC BigQuery Driver](https://docs.adbc-drivers.org/drivers/bigquery/index.html)

### Trino with ADBC

```yaml
connections:
  TRINO:
    type: trino
    http_url: "http://myuser@localhost:8080?catalog=hive&schema=default"
    use_adbc: true
    # adbc_uri: "http://myuser@localhost:8080?catalog=hive&schema=default"  # optional override
```

ADBC URI format: `http://user@host:port?catalog=catalog&schema=schema`

**Official Documentation:** [Apache ADBC Trino Driver](https://docs.adbc-drivers.org/drivers/trino/index.html)

## Using `sling conns`

Here are examples of enabling ADBC on existing connections:

{% code overflow="wrap" %}

```bash
# PostgreSQL with ADBC
$ sling conns set POSTGRES type=postgres host=localhost user=myuser password=mypass database=mydb use_adbc=true

# MySQL with ADBC
$ sling conns set MYSQL type=mysql host=localhost user=root password=mypass database=mydb use_adbc=true

# SQL Server with ADBC
$ sling conns set MSSQL type=sqlserver host=localhost user=sa password=mypass database=master use_adbc=true

# Snowflake with ADBC
$ sling conns set SNOWFLAKE type=snowflake account=myaccount user=myuser password=mypass database=mydb use_adbc=true

# With custom ADBC URI override
$ sling conns set POSTGRES type=postgres host=localhost user=myuser password=mypass database=mydb use_adbc=true adbc_uri="postgresql://myuser:mypass@localhost:5432/mydb"
```

{% endcode %}

## Environment Variable

{% code overflow="wrap" %}

```bash
export POSTGRES='{ type: postgres, host: localhost, user: myuser, password: mypass, database: mydb, use_adbc: true }'

export MYSQL='{ type: mysql, host: localhost, user: root, password: mypass, database: mydb, use_adbc: true }'

export SNOWFLAKE='{ type: snowflake, account: myaccount, user: myuser, password: mypass, database: mydb, use_adbc: true }'
```

{% endcode %}

## Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  # PostgreSQL with ADBC
  POSTGRES:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    use_adbc: true

  # MySQL with ADBC
  MYSQL:
    type: mysql
    host: localhost
    user: root
    password: mypass
    database: mydb
    use_adbc: true

  # SQL Server with ADBC
  MSSQL:
    type: sqlserver
    host: localhost
    user: sa
    password: mypass
    database: master
    use_adbc: true

  # Snowflake with ADBC
  SNOWFLAKE:
    type: snowflake
    account: myaccount
    user: myuser
    password: mypass
    database: mydb
    schema: myschema
    use_adbc: true

  # DuckDB with ADBC
  DUCKDB:
    type: duckdb
    instance: /path/to/file.db
    use_adbc: true

  # With custom ADBC URI override
  POSTGRES_CUSTOM:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    use_adbc: true
    adbc_uri: "postgresql://myuser:mypass@localhost:5432/mydatabase?sslmode=require"
```

## Additional Resources

* [Apache ADBC Project](https://arrow.apache.org/adbc/)
* [ADBC Driver Documentation](https://docs.adbc-drivers.org/)
* [dbc CLI Documentation](https://docs.columnar.tech/dbc/)
* [Apache Arrow Documentation](https://arrow.apache.org/)

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# Azure Table

Extract data from Azure Table Storage

## Setup

The following credentials keys are accepted:

* `conn_str` (optional) -> The full connection string
* `account_name` (optional) -> The Azure Storage account name
* `account_key` (optional) -> The account key for authentication
* `sas_token` (optional) -> The SAS token for authentication

{% hint style="info" %}
You must provide one of: `account_key`, `sas_token`, `conn_str`, or use Azure's DefaultAzureCredential (when none are provided).
{% endhint %}

### Using `sling conns`

Here are examples of setting a connection named `AZURE_TABLE`. We must provide the `type=azuretable` property:

{% code overflow="wrap" %}

```bash
# Using connection string
$ sling conns set AZURE_TABLE type=azuretable conn_str="<connection_string>"

# Using account key
$ sling conns set AZURE_TABLE type=azuretable account_name=<account_name> account_key=<account_key>

# Using SAS token
$ sling conns set AZURE_TABLE type=azuretable account_name=<account_name> sas_token=<sas_token>
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
# Using connection string
export AZURE_TABLE='{ type: azuretable, conn_str: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net" }'

# Using account key
export AZURE_TABLE='{ type: azuretable, account_name: "myaccount", account_key: "mykey" }'

# Using SAS token
export AZURE_TABLE='{ type: azuretable, account_name: "myaccount", sas_token: "?sv=2020-08-04&ss=t&srt=sco..." }'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  AZURE_TABLE:
    type: azuretable
    account_name: myaccount
    account_key: <account_key>

  AZURE_TABLE_SAS:
    type: azuretable
    account_name: myaccount
    sas_token: '?sv=2020-08-04&ss=t&srt=sco...'

  AZURE_TABLE_CONN_STR:
    type: azuretable
    conn_str: DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net

  AZURE_TABLE_DEFAULT_AUTH:
    type: azuretable
    account_name: myaccount
    # Uses DefaultAzureCredential when no key/sas/conn_str provided
```

## Examples

### Extract data from Azure Table Storage

```yaml
source: azure_table
target: postgres

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  default.customers:
  default.orders:
    select: [PartitionKey, RowKey, OrderId, CustomerId, Amount, OrderDate]
  default.products:
    limit: 1000
```

### Incremental Loading

```yaml
source: postgres
target: azure_table

defaults:
  mode: incremental
  primary_key: [id]
  update_key: Timestamp

streams:
  public.events:
    object: default.events
```

### Working with Filters

```yaml
source: azure_table
target: snowflake

streams:
  default.logs:
    object: public.logs
    # Use OData filter syntax
    where: "PartitionKey eq '2024-01' and Status eq 'active'"
  
  default.transactions:
    object: public.transactions
    # Filter by timestamp
    where: "Timestamp ge datetime'2024-01-01T00:00:00Z'"
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# BigTable

Connect & Ingest data from a BigTable instance

## Setup

The following credentials keys are accepted:

* `instance` **(required)** -> The BigTable instance id
* `project` **(required)** -> The GCP project ID for the project
* `key_file` **(required)** -> The Service Account JSON

### Using `sling conns`

Here are examples of setting a connection named `BIGTABLE`. We must provide the `type=bigtable` property:

{% code overflow="wrap" %}

```bash
$ sling conns set BIGTABLE type=bigtable instance=<instance> project=<project> gc_key_file=/path/to/service.account.json
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export BIGTABLE='{type: bigtable, project: my-google-project, instance: my-instance, key_file: /path/to/service.account.json}'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  BIGTABLE:
    type: bigtable
    project: <project>
    instance: <instance>
    key_file: '<key_file>'
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# BigQuery

Connect & Ingest data from / to a BigQuery database

## Setup

The following credentials keys are accepted:

* `project` **(required)** -> The GCP project ID for the project
* `dataset` **(required)** -> The default dataset (like a schema)
* `gc_bucket` (optional) -> The Google Cloud Storage Bucket to use for loading (Recommended)
* `key_file` (optional) -> The path of the Service Account JSON. If not provided, the Google [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) will be used.
* `key_body` (optional) -> The Service Account JSON key content as a string. You can also provide the JSON content in env var `GC_KEY_BODY`.
* `location` (optional) -> The location of the account, such as `US` or `EU`. Default is `US`.
* `extra_scopes` (optional) -> An array of strings, which represent scopes to use in addition to `https://www.googleapis.com/auth/bigquery`. e.g. `["https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/spreadsheets"]`
* `use_adbc` (optional) -> Enable Arrow Database Connectivity (ADBC) driver for high-performance data transfer. See [ADBC](/connections/database-connections/adbc) for setup and details. (*v1.5.2+*)
* `adbc_uri` (optional) -> Override the automatically constructed ADBC connection URI when using `use_adbc=true`.

{% hint style="warning" %}
If you'd like to have sling use the machine's Google Cloud Application Default Credentials (usually with `cloud auth application-default login`), don't specify a `key_file` (or the env var `GC_KEY_BODY`).
{% endhint %}

### Using `sling conns`

Here are examples of setting a connection named `BIGQUERY`. We must provide the `type=bigquery` property:

{% code overflow="wrap" %}

```bash
$ sling conns set BIGQUERY type=bigquery project=<project> dataset=<dataset> gc_bucket=<gc_bucket> key_file=/path/to/service.account.json location=<location>
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export BIGQUERY='{type: bigquery, project: my-google-project, gc_bucket: my_gc_bucket, dataset: public, location: US, key_file: /path/to/service.account.json}'
```

{% endcode %}

You can also provide Sling the Service Account JSON in `key_body` as a string, or via environment variable `GC_KEY_BODY`, instead of a `key_file`.

{% code overflow="wrap" %}

```bash
export GC_KEY_BODY='{"type": "service_account","project_id": ...........}'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  BIGQUERY:
    type: bigquery
    project: <project>
    dataset: <dataset>
    gc_bucket: <gc_bucket>
    key_file: '<key_file>'

  # using with `key_body` instead of `key_file`
  BIGQUERY:
    type: bigquery
    project: <project>
    dataset: <dataset>
    gc_bucket: <gc_bucket>
    key_body: |
      { "type": "service_account", ... } 
```

### BigQuery Table Partitioning

```yaml
streams:
  my_schema.another_table:
    object: my_dataset.{stream_table}
    target_options:
      table_keys:
        partition: [ DATE_TRUNC(transaction_date, MONTH) ]

# OR
streams:
  my_schema.another_table:
    object: my_dataset.{stream_table}
    target_options:
      table_ddl: |
         CREATE TABLE my_dataset.{stream_table} ({col_types}) 
          PARTITION BY
            DATE_TRUNC(transaction_date, MONTH)
            OPTIONS (
              partition_expiration_days = 3,
              require_partition_filter = TRUE)
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# Cloudflare D1

Connect & Ingest data from / to a Cloudflare D1 database

## Setup

The following credentials keys are accepted:

* `account_id` **(required)** -> The Cloudflare account ID
* `api_token` **(required)** -> The API Token to access D1 resources
* `database` **(required)** -> The database name (not UUID)
* `insert_concurrency` (optional) -> The max number of concurrent requests when inserting data. Default is `50`.

### Using `sling conns`

Here are examples of setting a connection named `D1`. We must provide the `type=d1` property:

{% code overflow="wrap" %}

```bash
# for local files 
$ sling conns set D1 type=d1 account_id=<account_id> api_token=<api_token> database=<database>

# Or use url
$ sling conns set D1 url="d1://any_user:<api_token>@<account_id>/<database>"
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export D1='d1://any_user:<api_token>@<account_id>/<database>'
export D1='{ type: d1, api_token: "<api_token>", account_id: "<account_id>", database: "<database>" }'

$env:D1="d1://any_user:<api_token>@<account_id>/<database>" # For Windows PowerShell
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  D1:
    type: d1
    account_id: <account_id>
    api_token: <api_token>
    database: <database>
    insert_concurrency: 100
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# Clickhouse

Connect & Ingest data from / to a Clickhouse database

## Setup

The following credentials keys are accepted:

* `host` **(required)** -> The hostname / ip of the instance
* `database` **(required)** -> The database name of the instance
* `user` (optional) -> The username to access the instance
* `schema` (optional) -> The default schema to use
* `password` (optional) -> The password to access the instance
* `port` (optional) -> The port of the instance. Default is `9000`.
* `secure` (optional) -> Whether to use TLS for connecting. Default is `false`. **Note: This is required (`secure=true`) when connecting to ClickHouse Cloud.**
* `export_stream_format` (optional) -> Whether to specify the `FORMAT` when exporting (e.g. [`CSVWithNames`](https://clickhouse.com/docs/interfaces/formats/CSVWithNames)). Can help achieve low-memory exports.
* `skip_verify` (optional) -> Whether to skip verification for TLS. Default is `false`.
* `http_url` (optional) -> The HTTP url to override the connection string (see docs at [github.com/ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go?tab=readme-ov-file#http-support-experimental)). When specifying `http_url`, sling will use the HTTP clickhouse interface instead of the native interface. Native is recommended for optimal performance. **HTTP still has some limitations to be aware of for things like batch flushing and session context, so be cautious when switching over code to this protocol.**
* `ssh_tunnel` (optional) -> The URL of the SSH server you would like to use as a tunnel (example `ssh://user:password@db.host:22`)
* `ssh_private_key` (optional) -> The private key to use to access a SSH server (raw string or path to file).
* `ssh_passphrase` (optional) -> The passphrase to use to access a SSH server.
* `tls` -> TLS configuration name (`true`, `false`, `skip-verify`, or `custom` when providing `cert_*` keys below)

Custom TLS Certificates (v1.4.18+):

* `cert_file` (optional) -> the client certificate to use to access the instance via TLS (file path or raw)
* `cert_key_file` (optional) -> the client key to use to access the instance via TLS (file path or raw)
* `cert_ca_file` (optional) -> the client CA certificate to use to access the instance via TLS (file path or raw)

### Using `sling conns`

Here are examples of setting a connection named `CLICKHOUSE`. We must provide the `type=clickhouse` property:

{% code overflow="wrap" %}

```bash
$ sling conns set CLICKHOUSE type=clickhouse host=<host> user=<user> database=<database> password=<password> port=<port> 

# OR use url
$ sling conns set CLICKHOUSE url="clickhouse://myuser:mypass@host.ip:9000/mydatabase"

# connecting via http
$ sling conns set CLICKHOUSE type=clickhouse http_url="http://myuser:mypass@host.ip:8123/default"

# connecting to ClickHouse Cloud (secure=true is required)
$ sling conns set CLICKHOUSE_CLOUD url="clickhouse://myuser:mypass@host.clickhouse.cloud:9440/mydatabase?secure=true"
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export CLICKHOUSE='clickhouse://myuser:mypass@host.ip:9000/mydatabase'
export CLICKHOUSE='{ type: clickhouse, user: "myuser", password: "mypass", host: "host.ip", port: 9000, database: "mydatabase", export_stream_format: "CSVWithNames" }'

# ClickHouse Cloud (secure=true is required)
export CLICKHOUSE_CLOUD='clickhouse://myuser:mypass@host.clickhouse.cloud:9440/mydatabase?secure=true'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  CLICKHOUSE:
    type: clickhouse
    host: <host>
    user: <user>
    port: <port>
    database: <database>
    schema: <schema>
    password: <password>
    export_stream_format: CSVWithNames

  # connecting via http
  CLICKHOUSE_HTTP:
    type: clickhouse
    http_url: http://myuser:mypass@host.ip:8123/default

  # connecting via https
  CLICKHOUSE_HTTP:
    type: clickhouse
    http_url: https://myuser:mypass@host.ip:8123/default?secure=true

  # ClickHouse Cloud (secure=true is required)
  CLICKHOUSE_CLOUD:
    type: clickhouse
    host: host.clickhouse.cloud
    user: <user>
    port: 9440
    database: <database>
    password: <password>
    secure: true
    export_stream_format: CSVWithNames

  CLICKHOUSE_URL:
    url: "clickhouse://myuser:mypass@host.ip:9000/mydatabase"
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# Databricks

Connect & Ingest data from / to a Databricks database

## Setup

The following credentials keys are accepted:

* `host` **(required)** -> The hostname of the Databricks workspace (e.g., `dbc-a1b2c3d4-e5f6.cloud.databricks.com`)
* `token` **(required)** -> The personal access token or password to access the instance
* `warehouse_id` **(required)** -> The SQL warehouse ID to connect to
* `http_path` (optional) -> The HTTP path for the connection (if not using warehouse\_id)
* `catalog` (optional) -> The initial catalog name to use in the session (default: `hive_metastore`)
* `schema` (optional) -> The initial schema name to use in the session (default: `default`)
* `port` (optional) -> The port number (default: `443`)
* `max_rows` (optional) -> Maximum number of rows fetched per request (default: `10000`)
* `internal_volume` (optional) -> Specifies a custom internal volume to use for bulk operations. If not provided, Sling will attempt to create a volume in the default schema named `SLING_SCHEMA.SLING_STAGING`.
* `timeout` (optional) -> Timeout in seconds for server query execution (no timeout by default)
* `user_agent_entry` (optional) -> Used to identify partners
* `ansi_mode` (optional) -> Boolean for ANSI SQL specification adherence (default: `false`)
* `timezone` (optional) -> Timezone setting (default: `UTC`)

### Using `sling conns`

Here are examples of setting a connection named `DATABRICKS`. We must provide the `type=databricks` property:

{% code overflow="wrap" %}

```bash
# Basic connection with warehouse
$ sling conns set DATABRICKS type=databricks host=<workspace-hostname> token=<access-token> warehouse_id=<warehouse-id>

# Connection with custom HTTP path
$ sling conns set DATABRICKS type=databricks host=<workspace-hostname> token=<access-token> http_path=<http-path>

# With catalog and schema
$ sling conns set DATABRICKS type=databricks host=<workspace-hostname> token=<access-token> warehouse_id=<warehouse-id> catalog=<catalog> schema=<schema>

# Or use url
$ sling conns set DATABRICKS url="databricks://token:<access-token>@<workspace-hostname>:443/sql/1.0/warehouses/<warehouse-id>?schema=<schema>"
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export DATABRICKS='databricks://token:<access-token>@<workspace-hostname>:443/sql/1.0/warehouses/<warehouse-id>?schema=<schema>'

# use JSON format
export DATABRICKS_CONN='{ "type": "databricks", "host": "<workspace-hostname>", "token": "<access-token>", "warehouse_id": "<warehouse-id>", "schema": "<schema>" }'

# use YAML format (with new lines)
export DATABRICKS='
type: databricks
host: <workspace-hostname>
token: <access-token>
warehouse_id: <warehouse-id>
schema: <schema>
'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  DATABRICKS:
    type: databricks
    host: <workspace-hostname>
    token: <access-token>
    warehouse_id: <warehouse-id>
    schema: <schema>

  DATABRICKS_URL:
    url: "databricks://token:<access-token>@<workspace-hostname>:443/sql/1.0/warehouses/<warehouse-id>?catalog=<catalog>&schema=<schema>"\
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


# DB2

Connect & Extract data from IBM DB2 databases

IBM DB2 is a family of data management products, including database servers, developed by IBM. Sling supports connecting to DB2 databases via ODBC with a built-in `db2` template that provides optimized SQL syntax and type mappings.

{% hint style="info" %}
**Prerequisites**: DB2 connections use ODBC under the hood. Make sure you have the ODBC driver manager installed on your system before proceeding. See the [ODBC connection guide](/connections/database-connections/odbc) for general ODBC setup instructions and troubleshooting.
{% endhint %}

## Setup

Before using DB2 connections, you must install the IBM DB2 ODBC driver for your operating system.

{% tabs %}
{% tab title="macOS" %}
**Download the Driver**

Download the IBM Data Server Driver for ODBC and CLI from [IBM Fix Central](https://www.ibm.com/support/pages/db2-odbc-cli-driver-download-and-installation-information):

* **Intel Macs**: Download `macos64_odbc_cli.tar.gz` (Db2 11.5)
* **Apple Silicon (M1/M2/M3)**: Download the Db2 12.1 ARM64 driver package

**Install the Driver**

```bash
tar -xzf macos64_odbc_cli.tar.gz
cd odbc_cli/clidriver
./installDSDriver
```

**Set Environment Variables**

Add to `~/.zshrc` or `~/.bash_profile`:

```bash
export DB2_CLI_DRIVER_INSTALL_PATH=/path/to/clidriver
export DYLD_LIBRARY_PATH=$DB2_CLI_DRIVER_INSTALL_PATH/lib:$DYLD_LIBRARY_PATH
```

{% endtab %}

{% tab title="Linux" %}
**Download the Driver**

Download the IBM Data Server Driver for ODBC and CLI from [IBM Fix Central](https://www.ibm.com/support/pages/db2-odbc-cli-driver-download-and-installation-information):

```bash
wget https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz
```

**Install the Driver**

```bash
tar -xzf linuxx64_odbc_cli.tar.gz
cd odbc_cli/clidriver
./installDSDriver
```

**Install unixODBC**

```bash
# Debian/Ubuntu
sudo apt-get install unixodbc unixodbc-dev

# RHEL/CentOS/Fedora
sudo dnf install unixODBC unixODBC-devel
```

**Register the Driver**

Add to `/etc/odbcinst.ini`:

```ini
[IBM DB2 DRIVER]
Description = IBM DB2 ODBC Driver
Driver = /path/to/clidriver/lib/libdb2o.so
FileUsage = 1
```

**Set Environment Variables**

Add to `~/.bashrc`:

```bash
export DB2_CLI_DRIVER_INSTALL_PATH=/path/to/clidriver
export LD_LIBRARY_PATH=$DB2_CLI_DRIVER_INSTALL_PATH/lib:$LD_LIBRARY_PATH
```

{% endtab %}

{% tab title="Windows" %}
**Download the Driver**

Download the IBM Data Server Driver for ODBC and CLI from [IBM Fix Central](https://www.ibm.com/support/pages/db2-odbc-cli-driver-download-and-installation-information)

**Install the Driver**

1. Extract the compressed file to `C:\Program Files\IBM`
2. Open Command Prompt as Administrator and register the driver:

```cmd
cd "C:\Program Files\IBM\clidriver\bin"
db2oreg1 -i
db2oreg1 -setup
```

This registers the DB2 ODBC driver in Windows ODBC Data Sources.
{% endtab %}
{% endtabs %}

## Connection Properties

* `conn_string` **(required)** -> The ODBC connection string for DB2
* `conn_template` **(required)** -> Set to `db2` to use the built-in DB2 template

### Connection String Parameters

Common parameters for the DB2 ODBC connection string:

| Parameter  | Description                                     |
| ---------- | ----------------------------------------------- |
| `Driver`   | The ODBC driver name (e.g., `{IBM DB2 DRIVER}`) |
| `Hostname` | The DB2 server hostname or IP address           |
| `Port`     | The DB2 server port (default: `50000`)          |
| `Database` | The database name                               |
| `Protocol` | Connection protocol (typically `TCPIP`)         |
| `UID`      | Username for authentication                     |
| `PWD`      | Password for authentication                     |

### Using `sling conns`

{% code overflow="wrap" %}

```bash
sling conns set DB2 type=odbc conn_string='Driver={IBM DB2 DRIVER};Hostname=host.ip;Port=50000;Database=testdb;Protocol=TCPIP;UID=db2inst1;PWD=password;' conn_template=db2
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export DB2='{ type: odbc, conn_string: "Driver={IBM DB2 DRIVER};Hostname=host.ip;Port=50000;Database=testdb;Protocol=TCPIP;UID=db2inst1;PWD=password;", conn_template: db2 }'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  DB2:
    type: odbc
    conn_string: Driver={IBM DB2 DRIVER};Hostname=host.ip;Port=50000;Database=testdb;Protocol=TCPIP;UID=db2inst1;PWD=password;
    conn_template: db2
```

## Replication Example

Here's an example replication configuration to extract data from DB2 to PostgreSQL:

```yaml
source: DB2
target: MY_POSTGRES

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  # Extract entire table
  myschema.customers:

  # Extract using SQL query
  orders:
    sql: SELECT * FROM myschema.orders WHERE order_date >= '2024-01-01'
    object: public.orders

  # Extract specific columns
  products:
    sql: SELECT product_id, name, price FROM myschema.products
    object: public.products
```

## Troubleshooting

### Driver Not Found

If you receive a "driver not found" error:

1. Verify the driver is installed by checking the ODBC configuration:
   * **Linux/macOS**: Check `/etc/odbcinst.ini` or run `odbcinst -q -d`
   * **Windows**: Open "ODBC Data Sources" from Control Panel
2. Ensure the driver name in your connection string matches exactly (including brackets), e.g., `{IBM DB2 DRIVER}`
3. Verify environment variables are set correctly (`DB2_CLI_DRIVER_INSTALL_PATH`, `LD_LIBRARY_PATH`/`DYLD_LIBRARY_PATH`)

### Connection Timeout

For slow connections, add timeout parameters to your connection string:

```
Driver={IBM DB2 DRIVER};Hostname=host.ip;Port=50000;Database=testdb;Protocol=TCPIP;UID=db2inst1;PWD=password;ConnectTimeout=30;
```

### SSL/TLS Connections

For SSL-enabled DB2 connections, add security parameters:

```
Driver={IBM DB2 DRIVER};Hostname=host.ip;Port=50000;Database=testdb;Protocol=TCPIP;UID=db2inst1;PWD=password;Security=SSL;
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).




---

[Next Page](/llms-full.txt/1)