Wildcards are a way to match multiple streams, whether tables or files. They are useful to apply defaults to multiple streams, so you don't have to specify each stream configuration individually.
source: my_source_db
target: my_target_db
defaults:
# will dynamically create the object name based on the schema and table name
object: my_schema.{stream_schema}_{stream_table}
streams:
# match all tables in my_schema
my_schema.*:
# match all tables in another_schema that start with a prefix
another_schema.prefix_*:
# match all tables in another_schema that end with a suffix
another_schema.*_suffix:
Filtering files in a folder:
source: my_source_file
target: my_target_db
defaults:
# will dynamically create the object name based on the folder and file name
object: my_schema.{stream_folder}_{stream_file_name}
streams:
# match all files in folder
folder/*:
# match all files in another_folder that start with a prefix
another_folder/prefix_*:
# match all files in another_folder that end with a suffix
another_folder/*.csv:
# match all files in another_folder that start with prefix and end with suffix
another_folder/prefix_*.parquet:
Using the Sling Platform, you can create a job for specific tags. Below is an example of creating a job only running the streams with the tag marketing.
Calling with CLI
# Run all streams with tag:my_tag
sling run -r my_replication.yaml --streams tag:my_tag
# Run all streams with tag:my_tag or tag:another_tag
sling run -r my_replication.yaml --streams tag:my_tag,tag:another_tag
