Target Options

Specification

Here we have various keys accepted for target options:

Key	Description
add_new_columns	(Only for database target) Whether to add new columns from stream not found in target table (when mode is not full-refresh). Default is true.
adjust_column_type	(Only for database target) Whether to adjust the column type when needed. Default is false.
batch_limit	(Only for database target — since v1.2.11) The maximum number of records per transaction batch.
column_casing	Whether to convert the column name casing. This facilitates querying tables in target databases without using quotes. Accepts normalize (normalizes to target casing, unless column has varied casings, since v1.4.5), source (keep original casing), target (converts casing according to target database), snake (converts snake casing according to target database), upper or lower. Default is normalize (before v1.4.5, default is source). See here for details.
column_typing	This allows precise control over how column types are generated, for example, manipulating string column lengths when transferring between databases with different character encoding requirements. See here for details.
compression	(Only for file target) The type of compression to use when writing files. Valid inputs are none, auto and gzip, zstd, snappy. Default is auto.
datetime_format	(Only for file target) The ISO 8601 date format to use when writing date values. Default is auto
delete_missing	(Only for database target) Whether to soft/hard delete missing primary keys records in the target tables. Accepts soft or hard values. See here for more details
direct_insert	(Only for database target) Whether to bypass temporary table and insert directly into the target table. This can improve performance but has limitations: incremental/backfill modes with primary keys are not supported as they require merge/upsert operations. Default is false.
encoding	(Only for file target) The text encoding to use when writing files. Allows writing files in different encodings for compatibility with legacy systems or international data requirements. Options include: latin1, latin5, latin9, utf8, utf8_bom, utf16, windows1250, windows1252. Default is utf8.
isolation_level	(Only for database target) What transaction isolation level to set when insert/merging into final table. Accepts: default, read_uncommitted, read_committed, write_committed, repeatable_read, snapshot, serializable, linearizable
merge_strategy	(Only for database target — since v1.5.5) The strategy to use when merging data in incremental or backfill mode with a primary key. Accepts update_insert (upsert), delete_insert, insert (append-only), or update (update-only). Default depends on database. See Merge Strategy for details.
delimiter	(Only for file target) The delimiter to use when writing tabular files. Default is ,.
file_max_bytes	(For file target or temp files for DB target) The maximum number of bytes to write to a file. 0 means infinite number of bytes. When a value greater than 0 is specified, the output location will be a folder with many parts in it. Default is 50000000. Does not work with parquet file format (use file_max_rows instead).
file_max_rows	(For file target or temp files for DB target) The maximum number of rows (usually lines) to write to a file. 0 means infinite number of rows. When a value greater than 0 is specified, the output location will be a folder with many parts in it. Default is 500000.
format	(Only for file target) The format of the file(s). Options are: csv, parquet, xlsx, json and jsonlines. On Snowflake, Databricks and Redshift, parquet is what the Arrow Lane uses for the staged files, so a stream on the lane stages Parquet even when the option is not set.
ignore_existing	Ignore existing target file/table if it exists (do not overwrite/modify). Default is false.
header	(Only for file target) Whether to write the first line as header. Default is true.
table_ddl	(Only for database target) The table DDL to use when writing to a database. Default is auto-generated by Sling. Accepts the {col_types} runtime variable. See here for more details
table_keys	(Only for database target) The table keys to define when creating a table, such as partition, cluster, sort, etc. Each entry defines an array of column names, or expression. See here for more details.
table_tmp	(Only for database target) The temporary table name that should be used when loading into a database. Default is auto-generated by Sling.
use_bulk	(Only for database target) Whether to use external bulk loading tools, if installed and available in the PATH environment variable. Sling looks to use bcp for SQL Server, sqlldr for Oracle and mysql for MySQL. If false, traditional batch INSERT loading will be used. Default is true.

Encoding Options

When writing files for different systems or regions, you may need to specify a particular text encoding. This is essential for maintaining data integrity when working with international characters or legacy systems that expect specific encodings.

Supported Encodings

• latin1 (ISO-8859-1): Western European languages

• latin5 (ISO-8859-5): Cyrillic alphabet (Russian, Bulgarian, etc.)

• latin9 (ISO-8859-15): Western European with Euro symbol

• utf8: Unicode UTF-8 (default, most common)

• utf8_bom: UTF-8 with Byte Order Mark

• utf16: Unicode UTF-16

• windows1250: Central European languages (Windows)

• windows1252: Western European languages (Windows)

Arrow Lane. On Snowflake and Databricks the Arrow Lane stages Parquet files by default when it is active, so format: parquet needs no explicit setting there. Set format: csv to keep the delimited files, which also keeps the stream on the row path.

Examples

# Writing data to a Latin-1 encoded CSV file for a legacy French system
streams:
  public.customers:
    object: "file://output/customers_french.csv"
    target_options:
      format: csv
      encoding: latin1
      header: true

# Writing data to a Windows-1252 encoded file for compatibility with older Windows systems
streams:
  public.documents:
    object: "file://output/documents_legacy.csv"
    target_options:
      format: csv
      encoding: windows1252
      header: true

# Writing data to a UTF-8 file with BOM for systems that require it
streams:
  public.international_data:
    object: "file://output/international.csv"
    target_options:
      format: csv
      encoding: utf8_bom
      header: true

CLI Usage

# Write data to a Latin-1 encoded CSV file
sling run \
  --src-conn POSTGRES \
  --src-stream public.customers \
  --tgt-object "file://./output/customers_latin1.csv" \
  --tgt-options '{"format": "csv", "encoding": "latin1", "header": true}'

# Write data to a Windows-1252 encoded file for legacy compatibility
sling run \
  --src-conn MYSQL \
  --src-stream inventory.products \
  --tgt-object "file://./exports/products_windows.csv" \
  --tgt-options '{"format": "csv", "encoding": "windows1252", "header": true}'

Important: Choose the appropriate encoding based on your target system's requirements. Using the wrong encoding may cause special characters to appear corrupted or unreadable. UTF-8 is recommended for most modern systems and international data.

Table DDL

The table_ddl option allows you to define the table DDL to use when creating a table. This is useful when you want to define a table with specific properties. Sling also allows the injection of runtime variables, such as {object_name}, which will be replaced with the actual stream table name. See Runtime Variables for more information. The special {col_types} variable will be replaced with the actual column types of the stream table.

Here is an example of using the table_ddl option:

source: my_source_db
target: clickhouse

# apply to all streams
defaults:
  target_options:
    table_ddl: create table {object_name} ({col_types}) engine=MergeTree

streams:
  # use default target_options
  my_schema.my_table:

  # override default for this stream
  my_schema.another_table:
    target_options:
      table_ddl: create table {object_name} ({col_types}) engine=AggregatingMergeTree

Table Keys

Table keys are used to define the keys of the target table. They are useful to preset keys such as indexes, primary keys, etc.

Here are the accepted keys:

• cluster (used in BigQuery and Snowflake)

• index (used in PostgreSQL, MySQL, MariaDB, Oracle, SQL Server, SQLite, DuckDB and MotherDuck)

• partition (used in PostgreSQL, BigQuery, ClickHouse)

• primary (used in PostgreSQL, MySQL, Oracle, SQL Server, SQLite)

• sort (used in Redshift)

• unique (used in PostgreSQL, MySQL, Oracle, SQL Server, SQLite and DuckDB)

• aggregate, duplicate, distribution and hash (used in StarRocks)

Here is an example of using the table_keys option:

source: my_source_db
target: my_target_db

# apply to all streams
defaults:
  target_options:
    table_keys:
      primary: [id]

streams:
  # use default target_options
  my_schema.my_table:

  # override default for this stream
  my_schema.another_table:
    target_options:
      table_keys:
        index: [col1, col2] # creates two separate indexes: one on col1, one on col2

A flat list under index creates one index per entry, not a single composite index. The example above (index: [col1, col2]) produces two single-column indexes. To create a single composite index over multiple columns, nest the columns in a list (e.g. index: [[col1, col2]]). See Indexes below for all supported shapes.

Indexes

The index key is a list where each entry describes one index. A bare column name (or a flat list of bare column names) therefore creates one single-column index per name — index: [col1, col2] creates two separate indexes. To create a composite index over multiple columns, the entry itself must be a list (i.e. nest it): index: [[col1, col2]].

From v1.5.20+ each entry can take any of these shapes:

target_options:
  table_keys:
    index:
      # a single column -> one index on that column
      - search_col

      # an expression column -> one index on that column
      - LOWER(email)

      # a list -> one composite index over those columns
      - [org_id, project_id]

      # a custom named index
      - idx_lookup: [org_id, project_id]

Covering (INCLUDE) and partial (WHERE) indexes

A named entry can also take a mapped body to define a covering index (extra non-key columns via include) or a partial index (a filter via where). Use expression for an expression index and sort for column order:

target_options:
  table_keys:
    index:
      # covering index: key on org_id, includes name and email in the leaf pages
      - idx_org_cover:
          columns: [org_id]
          include: [name, email]

      # partial index: only rows where status = 'active'
      - idx_active:
          columns: [status]
          where: "status = 'active'"

      # expression index with descending order
      - idx_recent:
          expression: LOWER(email)
          sort: desc

include (covering) and where (partial) are supported on PostgreSQL and SQL Server. SQLite supports where (partial) only. On engines that do not support an attribute, Sling drops it with a warning and still creates the index. Engines without index support (Redshift, ClickHouse, Snowflake, BigQuery, StarRocks) ignore index entries entirely — use cluster, sort, partition, or aggregate for those.

When are keys and indexes applied

Primary keys, unique keys, and indexes from table_keys are part of the CREATE TABLE DDL. They are applied only when Sling creates the table, that is:

• full-refresh mode (the table is dropped and recreated on every run),

• the first incremental, backfill, or snapshot run — when the target table does not yet exist and Sling creates it,

• definition-only mode (creates the empty structure with the keys and indexes).

On a later incremental / backfill / snapshot run against a target table that already exists, Sling does not re-apply or reconcile keys and indexes — the existing table is kept as-is and only data is merged or appended. To change keys or indexes on an existing table, run a full-refresh (which recreates it) or alter the table directly in the database.

Primary & Unique Keys

The primary and unique keys each define one key over the listed columns (a composite key when more than one column is given). Unlike index, a flat list is not split into multiple keys — primary: [user_id, brand_id] creates a single composite primary key over both columns.

From v1.5.21+ the nested (wrapped-list) form is also accepted and is treated identically to the flat form, so you can use the same [[...]] shape as index for consistency:

target_options:
  table_keys:
    # both of these create a single composite PRIMARY KEY (user_id, brand_id)
    primary: [user_id, brand_id]
    # primary: [[user_id, brand_id]]

    # both of these create a single composite UNIQUE key (org_id, project_id)
    unique: [org_id, project_id]
    # unique: [[org_id, project_id]]