> For the complete documentation index, see [llms.txt](https://docs.slingdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.slingdata.io/concepts/replication/tags-wildcards.md). # Tags & Wildcards ## Wildcards Wildcards are a way to match multiple streams, whether tables or files. They are useful to apply defaults to multiple streams, so you don't have to specify each stream configuration individually. ```yaml source: my_source_db target: my_target_db defaults: # will dynamically create the object name based on the schema and table name object: my_schema.{stream_schema}_{stream_table} streams: # match all tables in my_schema my_schema.*: # match all tables in another_schema that start with a prefix another_schema.prefix_*: # match all tables in another_schema that end with a suffix another_schema.*_suffix: ``` Filtering files in a folder: ```yaml source: my_source_file target: my_target_db defaults: # will dynamically create the object name based on the folder and file name object: my_schema.{stream_folder}_{stream_file_name} streams: # match all files in folder folder/*: # match all files in another_folder that start with a prefix another_folder/prefix_*: # match all files in another_folder that end with a suffix another_folder/*.csv: # match all files in another_folder that start with prefix and end with suffix another_folder/prefix_*.parquet: ``` This also works for the CLI: ```bash sling run --src-conn MY_SOURCE_FILE \ --src-stream another_folder/prefix_*.parquet \ --tgt-conn MY_TARGET ... ``` ### Supported wildcard patterns Wildcards are matched with standard glob syntax. `*` (any run of characters) is the common case, and `?`, `[0-9]` character classes, and `{a,b}` alternation also match — for example `events_[0-9]*` or `log_{2023,2024}_*`. {% hint style="warning" %} A pattern is only treated as a wildcard when it contains a `*` or a `?`. A pattern built **only** from a character class or alternation — e.g. `events_[0-9]` with no `*` — is read as a literal table name, not a wildcard, and will not expand. Always include a `*` (or `?`) to make the key a wildcard: `events_[0-9]*`. {% endhint %} ### Merging wildcard tables into one target By default a database wildcard **expands into one stream per matched table**, and each stream needs a distinct target object — so you use a runtime variable like `{stream_table}` in `object` to keep them apart. To instead point every matched table at **one** target object, set `single: true` on the wildcard stream. Sling then keeps the wildcard as a single stream (it does not expand it into per-table streams) and writes all matched tables into the one `object`: ```yaml source: MY_SOURCE_DB target: MY_TARGET_DB streams: # write every table matching schema.events_* into ONE target table schema.events_*: object: analytics.events_all mode: full-refresh single: true ``` {% hint style="info" %} **`full-refresh` + `single: true` does not drop the target once per table.** Because the wildcard stays a single stream, the target is dropped and recreated **once** for the whole match, then all matched tables are written into it — you do not lose all but the last table. (Without `single: true`, each expanded table is its own stream against its own object.) {% endhint %} For **incremental** merges into one target, set `single: true` with a `primary_key` so rows from every matched table upsert into the shared table rather than replacing it. #### Merging dated / sharded tables (e.g. BigQuery `events_YYYYMMDD`) When shard tables must be combined with explicit control over the union — for example BigQuery date-sharded `events_20240101`, `events_20240102`, … — a **custom SQL stream** is the most predictable approach. Write the SQL that unions or wildcard-scans the shards and give it one target object: ```yaml source: BIGQUERY target: MY_SNOWFLAKE streams: events_merged: object: analytics.events_all mode: full-refresh sql: | SELECT * FROM `my_dataset.events_*` WHERE _TABLE_SUFFIX BETWEEN '20240101' AND '20240131' ``` BigQuery's own `events_*` wildcard-table syntax with `_TABLE_SUFFIX` is valid inside the `sql:` — Sling runs the query as-is. This keeps the union rule explicit rather than relying on stream-wildcard expansion. ### Excluding tables from a wildcard To drop a subset of a wildcard match, add a **more-specific** stream key with `disabled: true`. A table that matches a disabled key is skipped, even if a broader wildcard also matches it: ```yaml streams: # replicate everything under schema.events_* schema.events_*: # …but skip the intraday tables and any staging tables schema.events_intraday_*: disabled: true schema.*_stg: disabled: true ``` ### Precedence when several keys match When more than one stream key matches the same source table: * An **explicit (non-wildcard) stream** always wins over a wildcard. If you list `schema.events_2024` explicitly and also have `schema.events_*`, the explicit key's configuration is used for that table and the wildcard does not overwrite it. * A `disabled: true` key wins in the sense that the table is skipped entirely. * When two **wildcards** both match the same table, the order they are defined in the file is preserved. Prefer to disambiguate with an explicit key or a `disabled` exclusion rather than relying on overlap between two wildcards. ## Tags Tags are a way to categorize your streams. They can be used to filter streams when running a replication, or to create a job in the Sling Platform. ```yaml source: MY_SOURCE_FILE target: MY_TARGET defaults: tags: [ finance ] streams: # use default tags path/to/file1.csv: # override default tags path/to/file2.csv: tags: [ marketing ] ``` ### Create a Platform Job for a specific Tag Using the Sling Platform, you can create a job for specific tags. Below is an example of creating a job only running the streams with the tag `marketing`.
Sling Platform Job Tag
### Calling with CLI ```bash # Run all streams with tag:my_tag sling run -r my_replication.yaml --streams tag:my_tag # Run all streams with tag:my_tag or tag:another_tag sling run -r my_replication.yaml --streams tag:my_tag,tag:another_tag ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.slingdata.io/concepts/replication/tags-wildcards.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.