Hooks / Steps
Execute custom actions throughout your replication or pipeline
Hooks are powerful mechanisms in Sling that allow you to execute custom actions before (pre-hooks) or after (post-hooks) a replication stream, as well as at the start or end of the replication parent cycle (before first stream and/or after last stream). They enable you to extend and customize your data pipeline with various operations such as data validation, notifications, file management, and custom processing.
Some typical operations include:
Stream Level
Pre-Hooks: Execute before the replication stream run starts
Validate prerequisites
Download necessary files
Set up configurations
Perform cleanup operations
Post-Hooks: Execute after the replication stream run completes
Validate results
Send notifications
Upload processed files
Clean up temporary files
Log completion status
Pre/Post-Merge-Hooks: Execute in transaction session, before/after data is loaded/merged into final table
Set specific session setting and configuration
Alter table holding the temporary data, prior to merge
Run Specific SQL queries on other tables
Available Hook Types
Hook Configuration
Hooks can be configured in two locations:
At the
defaultslevel (applies to all streams)At the individual
streamlevel (replaces the default hooks of the same stage, see Merging Stream Hooks with Default Hooks)
Stream level Hooks
We can use the following structure to decare hooks with the hooks key, under the defaults branch or under any stream branch.
Merging Stream Hooks with Default Hooks
Sling applies the defaults hooks one stage at a time. A stage that you set on a stream replaces only the default hooks of that stage. The other default stages stay. For example, a stream that sets its own post hooks still runs the default pre hooks.
To add hooks to the default hooks, instead of replacing them, use a modifier key:
+pre / pre+
Runs the hooks before / after the pre hooks
+post / post+
Runs the hooks before / after the post hooks
+pre_merge / pre_merge+
Runs the hooks before / after the pre_merge hooks
+post_merge / post_merge+
Runs the hooks before / after the post_merge hooks
The modifiers wrap the default hooks of that stage. If the stream also sets the stage (for example post), the modifiers wrap the stream hooks, and the default hooks of that stage do not run.
This is available in v1.6.5+.
Replication level Hooks
We can also define hooks to run at the replication file level, meaning before any of the streams run and/or after all the streams have ran. For replication level hooks, we must declare the start and end hooks at the root of the YAML configuration.
Common Hook Properties
All hook types share some common properties:
type
The type of hook (query/ http/ check/ copy / delete/ log / inspect)
Yes
if
Optional condition to determine if the hook should execute
No
id
a specify identifier to refer to the hook output data.
No
on_failure
What to do if the hook fails (abort/ warn/ quiet/skip/break)
No (defaults to abort)
Variables Available
runtime_state- Contains all state variables availablestate.*- All hooks output state information (keyed by hook id)store.*- All stored values from previous hooksenv.*- All variables defined in theenvtimestamp.*- Various timestamp parts informationexecution.*- Replication run level informationsource.*- Source connection informationtarget.*- Target connection informationstream.*- Current source stream infoobject.*- Current target object inforuns.*- All runs information (keyed by stream run id)run.*- Current stream run information
Nested Fields
runtime_state Payload
The best way to view any available variables is to print the runtime_state variable.
For example, using the log hook as shown below will print all available variables.
Shows something like below JSON payload.
Furthermore, we can access any data-point using a jmespath expression:
state["start-02"].status- Gets the status of a hook (returnssuccess)store.my_key- Gets a stored value from the store (returnsmy_value)run.total_rows- Gets the number of rows processed in the current run (returns34)run.duration- Gets the duration of the current run in seconds (returns8)timestamp.unix- The epoch/unix timestamp (returns1737286051)source.bucket- Gets the source S3 bucket name (returnsmy-bucket-1)target.database- Gets the target database name (returnspostgres)run.config.primary_key[0]- Gets the first primary key column (returnsid)stream.file_path- Gets the current stream's file path (returnstest/public_test1k_postgres_pg_parquet/update_dt_year=2018/update_dt_month=11)stream.file_ext- Gets the file extension (returnsparquet)stream.schema_lower- Gets the stream schema name in lowercasestream.table_upper- Gets the stream table name in uppercaseobject.temp_full_name- Gets the temporary table full name (returns"public"."test1k_postgres_pg_parquet_tmp")execution.status.error- Gets the count of errored streams (returns0)execution.total_bytes- Gets the total bytes processed across all runs (returns6050)runs["test_public_test1k"].status- Gets the status of a specific run by ID (returnssuccess)
Complete Example
Best Practices
Error Handling: Specify appropriate
on_failurebehavior. The default value isabort.Validation: Use
checkhooks to validate prerequisites and resultsLogging: Implement
loghooks for better observabilityCleanup: Use
deletehooks to manage temporary / old filesModularity: Break down complex operations into multiple hooks
Conditions: Use
ifconditions to control hook executionEnvironment Awareness: Consider different environments in hook configurations
Last updated
Was this helpful?