Tags & Wildcards

Wildcards

Wildcards are a way to match multiple streams, whether tables or files. They are useful to apply defaults to multiple streams, so you don't have to specify each stream configuration individually.

source: my_source_db
target: my_target_db

defaults:
  # will dynamically create the object name based on the schema and table name
  object: my_schema.{stream_schema}_{stream_table}

streams:
  # match all tables in my_schema
  my_schema.*: 

  # match all tables in another_schema that start with a prefix
  another_schema.prefix_*:

  # match all tables in another_schema that end with a suffix
  another_schema.*_suffix:

Filtering files in a folder:

source: my_source_file
target: my_target_db

defaults:
  # will dynamically create the object name based on the folder and file name
  object: my_schema.{stream_folder}_{stream_file_name}

streams:
  # match all files in folder
  folder/*: 

  # match all files in another_folder that start with a prefix
  another_folder/prefix_*:

  # match all files in another_folder that end with a suffix
  another_folder/*.csv:

  # match all files in another_folder that start with prefix and end with suffix
  another_folder/prefix_*.parquet:

This also works for the CLI:

sling run --src-conn MY_SOURCE_FILE \
  --src-stream another_folder/prefix_*.parquet \
  --tgt-conn MY_TARGET ...

Tags

Tags are a way to categorize your streams. They can be used to filter streams when running a replication, or to create a job in the Sling Platform.

source: MY_SOURCE_FILE
target: MY_TARGET

defaults:
  tags: [ finance ]

streams:
  # use default tags
  path/to/file1.csv:

  # override default tags
  path/to/file2.csv:
    tags: [ marketing ]

Create a Platform Job for a specific Tag

Using the Sling Platform, you can create a job for specific tags. Below is an example of creating a job only running the streams with the tag marketing.

Sling Platform Job Tag

Calling with CLI

# Run all streams with tag:my_tag
sling run -r my_replication.yaml --streams tag:my_tag

# Run all streams with tag:my_tag or tag:another_tag
sling run -r my_replication.yaml --streams tag:my_tag,tag:another_tag