CLI Flags

Quick ad-hoc runs from your terminal shell or script.

Overview

For quickly running ad-hoc operations from the terminal, using CLI flags is often best. Here are some examples:

# Pipe in your json file and flatten the nested keys into their own columns
$ cat /tmp/my_file.json | sling run --src-options '{"flatten": "true"}' --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' --mode full-refresh

# Read folder containing many CSV files
$ sling run \
    --src-stream 'file:///tmp/my_csv_folder/' \
    --tgt-conn MY_TARGET_DB --tgt-object 'target_schema.target_table' \
    --mode full-refresh

# Load only latest data from one source DB to another.
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_TARGET_DB \
    --tgt-object 'target_schema.target_table' \
    --mode incremental \
    --primary-key 'id' --update-key 'last_modified_dt' 

# Export / Backup database tables to JSON files
$ sling run \
    --src-conn MY_SOURCE_DB \
    --src-stream 'source_schema.source_table' \
    --tgt-conn MY_S3_BUCKET \
    --tgt-object 's3://my-bucket/my_json_folder/' \
    --tgt-options '{"file_max_rows": 100000, "format": "jsonlines"}'

Interface Specifications

CLI FlagDescription

--src-conn

The source database connection (name, conn string or URL).

--tgt-conn

The target database connection (name, conn string or URL).

--src-stream

The source table (schema.table), local / cloud file path. Can also be the path of sql file or in-line text to use as query. Use file:// for local paths.

--tgt-object

The target table (schema.table) or local / cloud file path. Use file:// for local paths. See here for details on runtime variables.

--mode

The target load mode to use: incremental, truncate, full-refresh, backfill or snapshot. Default is full-refresh.

--primary-key

The column(s) to use as primary key (for incremental mode). If composite key, use a comma-delimited string.

--update-key

The column to use as update key (for incremental mode).

--src-options

In-line options to further configure source (JSON or YAML). See here for details.

--tgt-options

In-line options to further configure target (JSON or YAML). See here for details.

--stdout

Output the stream to standard output (STDOUT).

--select

Select or exclude specific columns from the source stream. (comma separated). Use - prefix to exclude.

--streams

Only run specific streams from a replication. (comma separated)

Last updated