Modes

Here are the various loading modes available. All modes load into a new temporary table prior to final load.

Mode
Description

full-refresh

This is the default mode. The target table will be dropped and recreated with the source data.

incremental

The source data will be merged or appended into the target table. If the table does not exist, it will be created. See below for more details.

truncate

Similar to full-refresh, except that the target table is truncated instead of dropped. This keeps any special DDL / GRANT applied.

snapshot

Appends the full dataset with an added timestamp column. If the target table exists, Sling will insert into / append data with a _sling_loaded_at column. If it does not, the table will be created.

backfill

Similar to incremental, but takes a range input to backfill a specific update_key range, such as dates or numbers.

Incremental Mode Strategies

Load Strategy
Primary Key
Update Key
Stream Strategy

New Data Upsert (update/insert)

yes

yes

Only new records after max(update_key)

Full Data Upsert (update/insert)

yes

no

Full data

Append Only (insert, no update)

no

yes

Only new records after max(update_key)

Incremental or Backfill Mode With Custom SQL

When using incremental or backfill mode with a custom SQL stream, Sling provides two placeholder options to handle incremental loading:

  • {incremental_where_cond}: Injects a complete WHERE condition

  • {incremental_value}: Injects only the value, allowing for custom condition logic

Using {incremental_where_cond}

This placeholder injects a complete WHERE condition based on the update_key:

SELECT * FROM my_schema.my_table 
WHERE {incremental_where_cond}

Sling will replace the placeholder as follows:

  • First run (table doesn't exist): WHERE 1=1

  • Subsequent runs: WHERE my_update_key > '[incremental_value]'

For example, if the last maximum value was '2001-01-01 01:01:01', the query becomes:

SELECT * FROM my_schema.my_table 
WHERE my_update_key > '2001-01-01 01:01:01'

Using {incremental_value}

This placeholder gives you more control over the WHERE condition by injecting only the value:

SELECT * FROM my_schema.my_table 
WHERE my_int_key > coalesce({incremental_value}, 0)  -- For numeric columns
-- or
WHERE my_timestamp > coalesce({incremental_value}, '2001-01-01')  -- For timestamp columns

Sling will replace the placeholder as follows:

  • First run (table doesn't exist): {incremental_value} becomes null

  • Subsequent runs: {incremental_value} becomes the last maximum value

For example, if the last maximum value was 99, the query becomes:

SELECT * FROM my_schema.my_table 
WHERE my_int_key > coalesce(99, 0)

Last updated