Replication
Replication hooks allow you to trigger nested replication tasks within your workflow. This is particularly useful for orchestrating complex data pipelines, running dependent replications, or managing multi-step data transformations.
Configuration
Properties
path
Yes
Path to the replication configuration file
mode
No
Override the replication mode (full/incremental)
range
No
Override the backfill range for incremental mode
streams
No
List of specific streams to run. If not provided, all streams will be run
env
No
Map of environment variables to set for the replication
on_failure
No
What to do if the replication fails (abort/warn/quiet/skip)
Output
When the replication hook executes successfully, it returns the following output that can be accessed in subsequent hooks:
You can access these values in subsequent hooks using the following syntax (jmespath
):
{state.hook_id.status}
- Status of the hook execution{state.hook_id.path}
- The replication configuration file path{state.hook_id.range}
- The backfill range used{state.hook_id.streams}
- The streams that were processed
Examples
Run Dependent Replication
Execute a dependent replication after successful processing:
Environment-Specific Replication
Run different replication configurations based on environment:
Conditional Stream Selection
Select streams based on runtime conditions:
Last updated