CLI Flags

Quick ad-hoc runs from your terminal shell or script.

Overview

For quickly running ad-hoc operations from the terminal, using CLI flags is often best. Here are some examples:

# Pipe in your json file and flatten the nested keys into their own columns
$ cat /tmp/my_file.json | sling run --src-options '{"flatten": "true"}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

# Read folder containing many CSV files
$ sling run \
    --src-stream 'file:///tmp/my_csv_folder/' \
    --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' \
    --mode full-refresh

# Load only latest data from one source DB to another.
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_TARGET_DB \
    --tgt-object 'target_schema.target_table' \
    --mode incremental \
    --primary-key 'id' --update-key 'last_modified_dt' 

# Export / Backup database tables to JSON files
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_S3_BUCKET \
    --tgt-object 's3://my-bucket/my_json_folder/' \
    --tgt-options '{"file_max_rows": 100000, "format": "jsonlines"}'

Interface Specifications

CLI Flag	Description
--src-conn	The source database connection (name, conn string or URL).
--tgt-conn	The target database connection (name, conn string or URL).
--src-stream	The source table (schema.table), local / cloud file path. Can also be the path of sql file or in-line text to use as query. Use file:// for local paths.
--tgt-object	The target table (schema.table) or local / cloud file path. Use file:// for local paths. See here for details on runtime variables.
--mode	The target load mode to use: incremental, truncate, full-refresh, backfill or snapshot. Default is full-refresh.
--primary-key	The column(s) to use as primary key (for incremental mode). If composite key, use a comma-delimited string.
--update-key	The column to use as update key (for incremental mode).
--src-options	In-line options to further configure source (JSON or YAML). See here for details.
--tgt-options	In-line options to further configure target (JSON or YAML). See here for details.
--stdout	Output the stream to standard output (STDOUT).
--select	Select or exclude specific columns from the source stream. (comma separated). Use - prefix to exclude.
--streams	Only run specific streams from a replication. (comma separated)