Structure

Below is the structure of the replication configuration file.

Root Level

At the root level, we have the following keys:

# 'source', 'target' and 'streams' keys are required
source: <connection name>
target: <connection name>

defaults: <replication stream map>

streams:
  <stream name>: <replication stream map>

env:
  <variable name>: <variable value>

Stream Level

The <stream name> identifies the stream to replicate. This can be either a source table name, a file path, or a wildcard pattern using *. Wildcards allow matching multiple tables within a schema or multiple files within a directory. For example, my_schema.* matches all tables in my_schema, while data/*.csv matches all CSV files in the data directory. See Tags & Wildcards for more details.

The <replication stream map> is a map object which accepts the following keys:

object: <target table or file name>
mode: full-refresh | incremental | truncate | snapshot | backfill
description: <stream description>
disabled: true | false

primary_key: [<array of column names to use as primary key>]
update_key: <column name to use as incremental key>

columns: {<map of column name to data type>}
select: [<array of column names to include or exclude>]
single: true | false
sql: <source custom SQL query>
transforms: [<array of transforms or map of column name to array of transforms>]

source_options: <source options map>
target_options: <target options map>

Source Options

The <source options map> accepts the keys below. See Source Options for more details.

compression: auto | none | zip | gzip | snappy | zstd
datetime_format: auto | <ISO 8601 date format>
delimiter: <character to use as flat file delimiter>
empty_as_null: true | false
escape: <character to use as flat file quote escape>
flatten: true | false
format: csv | xml | xlsx | json | parquet | avro | sas7bdat | jsonlines | iceberg | delta | raw
header: true | false
jmespath: <JMESPath expression>
limit: <integer>
null_if: <null_if expression>
range: <backfill range expression>
sheet: <excel sheet/range expression>
skip_blank_lines: true | false

Target Options

The <target options map> accepts the keys below. See Target Options for more details.

add_new_columns: true | false
adjust_column_type: true | false
batch_limit: <integer>
column_casing: source | target | snake | upper | lower
compression: auto | none | gzip | snappy | zstd
datetime_format: auto | <ISO 8601 date format>
delimiter: <character to use as flat file delimiter>
file_max_bytes: <integer>
file_max_rows: <integer>
format: csv | xlsx | json | parquet | raw
header: true | false
ignore_existing: true | false
post_sql: <sql query>
pre_sql: <sql query>
table_ddl: <ddl sql query>
table_keys: {map of table key type to array of column names}
table_tmp: <name of table>
use_bulk: true | false

Replication Specification

Here we have the definitions for the accepted keys.

Replication Config Key	Description
source	The source database connection (name, conn string or URL).
target	The target database connection (name, conn string or URL).
streams.<key>	The source table (schema.table), local / cloud file path. Use file:// for local paths.
streams.<key>.object or defaults.object	The target table (schema.table) or local / cloud file path. Use file:// for local paths.
streams.<key>.columns or defaults.columns	The columns types map. See here for details.
streams.<key>.transforms or defaults.transforms	The transforms to apply. See here for details.
streams.<key>.mode or defaults.mode	The target load mode to use: incremental, truncate, full-refresh, backfill or snapshot. Default is full-refresh.
streams.<key>.select or defaults.select	Select or exclude specific columns from the source stream. Use - prefix to exclude.
streams.<key>.single or defaults.single	When using a wildcard (*) in the stream name, consider as a single stream (don't expand into many streams).
streams.<key>.sql or defaults.sql	The custom SQL query to use. Accepts file://path/to.query.sql as well.
streams.<key>.primary_key or defaults.primary_key	The column(s) to use as primary key. If composite key, use array.
streams.<key>.update_key or defaults.update_key	The column to use as update key (for incremental mode).
streams.<key>.source_options or defaults.source_options	Options to further configure source. See here for details.
streams.<key>.target_options or defaults.target_options	Options to further configure target. See here for details.
env	Environment variables to use for replication. See here for details.