Group

Group hooks allow you to execute a sequence of steps, optionally in a loop. This is particularly useful for running multiple operations together, iterating over datasets, or creating reusable step templates.

Configuration

- type: group
  steps:                # Required: Array of steps to execute
    - type: log
      message: "Step 1"
    - type: http
      url: "https://api.example.com"
  loop: [1, 2, 3]      # Optional: Array or jmespath expression for iteration
  env:                 # Optional: Environment variables for all steps
    ENV_VAR1: "value1"
    ENV_VAR2: "value2"
  on_failure: abort    # Optional: abort/warn/quiet/skip
  id: my_id           # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property	Required	Description
steps	Yes	Array of step configurations to execute
loop	No	Array or jmespath expression defining iteration values
env	No	Map of environment variables available to all steps
on_failure	No	What to do if any step fails (abort/warn/quiet/skip)

Output

When the group hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
loop_values: 5   # The rendered values if loop was provided

You can access these values in subsequent hooks using the following syntax (jmespath):

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.loop_values} - Values used in loop (if provided)

Within loop iterations, steps can access:

• {loop.index} - Current iteration index (0-based)

• {loop.value} - Current value from the loop array/expression

Examples

Basic Step Group

Execute multiple steps in sequence:

- type: group
  id: initialization
  steps:
    - type: log
      message: "Starting initialization"
    - type: query
      connection: target_db
      query: "CREATE SCHEMA IF NOT EXISTS processed"
    - type: log
      message: "Initialization complete"

Loop Over Array

Iterate over a fixed array of values:

- type: group
  id: process_regions
  loop: ["us-east", "us-west", "eu-central"]
  steps:
    - type: log
      message: "Processing region: {loop.value}"
    - type: query
      connection: target_db
      query: |
        UPDATE region_status 
        SET last_checked = CURRENT_TIMESTAMP
        WHERE region_name = '{loop.value}'

Loop Over Previous Hook Results

Use results from a previous hook as loop input:

- type: list
  id: file_list
  location: aws_s3/data/daily/
  only: files    # return only files

- type: group
  id: process_files
  loop: state.file_list.result
  steps:
    - type: log
      message: "Processing file: {loop.value.name}"

    - type: download
      connection: aws_s3
      remote_path: "data/daily/{loop.value.name}"

Conditional Step Execution in Loop

Execute steps conditionally within a loop:

- type: group
  id: validate_tables
  loop: ["users", "orders", "products"]
  steps:
    - type: query
      connection: source_db
      id: get_count
      query: |
        SELECT COUNT(*) as row_count 
        FROM {loop.value}
    - type: log
      if: get_count.result[0].row_count > 1000
      message: "Table {loop.value} has {get_count.result[0].row_count} rows"

Environment Variables in Group

Share environment variables across steps:

- type: group
  id: backup_process
  env:
    BACKUP_DATE: "{timestamp.date}"
    BACKUP_PATH: "backups/{target.environment}"
  steps:
    - type: log
      message: "Starting backup for {env.BACKUP_DATE}"
    - type: copy
      from: "source_storage/{run.stream.name}"
      to: "{env.BACKUP_PATH}/{env.BACKUP_DATE}/{run.stream.name}"

Nested Groups

Create hierarchical step organization:

- type: group
  id: main_process
  loop: ["staging", "production"]
  steps:
    - type: log
      message: "Processing environment: {loop.value}"
    - type: group
      steps:
        - type: query
          connection: "{loop.value}_db"
          query: "VACUUM ANALYZE my_table"
        - type: log
          message: "Maintenance complete for {loop.value}"