Pipelines
Use Pipelines to orchestrate multiple steps in sequence
A Pipeline in Sling allows you to execute multiple steps in sequence. Each step can be a different type of operation, enabling you to create complex workflows by chaining together various actions like running replications, executing queries, making HTTP requests, and more.
Pipeline Configuration
A pipeline is defined in YAML format with a steps
key at the root level containing an array of steps. Each step supports the same types and configurations as Hooks.
Available Step Types
Pipelines support all the same types as Hooks:
Check
Validate conditions and control flow
Command
Run any command/process
Copy
Transfer files between local or remote storage connections
Delete
Remove files from local or remote storage connections
Group
Run sequences of steps or loop over values
HTTP
Make HTTP requests to external services
Inspect
Inspect a file or folder
List
List files in folder
Log
Output custom messages and create audit trails
Query
Execute SQL queries against any defined connection
Replication
Run a Replication
Common Step Properties
Each step shares the same common properties as hooks:
type
The type of step (query
/ http
/ check
/ copy
/ delete
/ log
/ inspect
)
Yes
if
Optional condition to determine if the step should execute
No
id
A specific identifier to refer to the step output data
No
on_failure
What to do if the step fails (abort
/ warn
/ quiet
/skip
)
No (defaults to abort
)
Variables Available
Pipeline steps have access to the runtime state which includes various variables that can be referenced using curly braces {variable}
. The available variables include:
runtime_state
- Contains all state variables availableenv.*
- All variables defined in theenv
timestamp.*
- Various timestamp parts informationsteps.*
- Output data from previous steps (referenced by theirid
)
You can view all available variables by using a log step:
Example Pipeline
Here's a complete example that demonstrates various pipeline capabilities:
Best Practices
Error Handling: Use appropriate
on_failure
behaviors for each stepValidation: Include check steps to validate critical conditions
Logging: Add log steps for better observability
Modularity: Break down complex operations into multiple steps
Conditions: Use
if
conditions to control step executionVariables: Leverage environment variables and runtime state for dynamic configuration
Identifiers: Use meaningful
id
s for steps when you need to reference their output later
Running a Pipeline
You can run a pipeline using the Sling CLI:
Last updated