Modes
Here are the various loading modes available. All modes load into a new temporary table prior to final load.
full-refresh
This is the default mode. The target table will be dropped and recreated with the source data.
incremental
The source data will be merged or appended into the target table. If the table does not exist, it will be created. See below for more details.
truncate
Similar to full-refresh
, except that the target table is truncated instead of dropped. This keeps any special DDL / GRANT applied.
snapshot
Appends the full dataset with an added timestamp column. If the target table exists, Sling will insert into / append data with a _sling_loaded_at
column. If it does not, the table will be created.
backfill
Similar to incremental
, but takes a range
input to backfill a specific update_key
range, such as dates or numbers.
Incremental Mode Strategies
New Data Upsert (update/insert)
yes
yes
Only new records after max(update_key)
Full Data Upsert (update/insert)
yes
no
Full data
Append Only (insert, no update)
no
yes
Only new records after max(update_key)
Incremental or Backfill Mode With Custom SQL
When using incremental
or backfill
mode with a custom SQL stream, Sling provides two placeholder options to handle incremental loading:
{incremental_where_cond}
: Injects a complete WHERE condition{incremental_value}
: Injects only the value, allowing for custom condition logic
Using {incremental_where_cond}
This placeholder injects a complete WHERE condition based on the update_key
:
Sling will replace the placeholder as follows:
First run (table doesn't exist):
WHERE 1=1
Subsequent runs:
WHERE my_update_key > '[incremental_value]'
For example, if the last maximum value was '2001-01-01 01:01:01', the query becomes:
Using {incremental_value}
This placeholder gives you more control over the WHERE condition by injecting only the value:
Sling will replace the placeholder as follows:
First run (table doesn't exist):
{incremental_value}
becomesnull
Subsequent runs:
{incremental_value}
becomes the last maximum value
For example, if the last maximum value was 99, the query becomes:
Last updated