Replication
Multiple streams in a YAML or JSON file. Best way to scale Sling.
Overview
Replications are the best way to use sling in a reusable manner. The defaults
key allows reusing your inputs with the ability to override any of them in a particular stream. Both YAML or JSON files are accepted. When you run a replication, internally, Sling auto-generates many tasks (one per stream) and runs them in order.
Here is a basic example, where all PostgreSQL tables in the schema my_schema
will be loaded into Snowflake. The my_schema.*
notation as the stream name is a feature possible only in Replications. Also notice how defaults.object
uses runtime variables.
Another example:
We can use a replication config with: sling run -r /path/to/replication.yaml
Interface Specifications
Replication Config Key | Description |
---|---|
| The source database connection (name, conn string or URL). |
| The target database connection (name, conn string or URL). |
| The source table (schema.table), local / cloud file path. Use |
or | The target table (schema.table) or local / cloud file path. Use |
or | The target load mode to use: |
| Select or exclude specific columns from the source stream. Use |
| When using a wildcard ( |
| The custom SQL query to use. Accepts |
or | The column(s) to use as primary key. If composite key, use array. |
or | The column to use as update key (for |
or | Options to further configure source. See here for details. |
or | Options to further configure target. See here for details. |
| Environment variables to use for replication. See here for details. |
Here are the relevant Go structs for replications:
Last updated