Store

Store hooks allow you to store and manage values in memory during execution, creating a shared key-value store that can be used by subsequent hooks. This hook is particularly useful for storing processing results, creating reusable variables, or maintaining state across your workflow.

Configuration

- type: store
  key: my_key             # Required: Key to store the value under
  value: some value       # Required (unless deleting): Value to store
  delete: false           # Optional: Set to true to delete the key

Properties

Property	Required	Description
key	Yes	The key name to store the value under
value	Yes (unless deleting)	Value to store at the specified key
delete	No	When set to true, deletes the key and its value instead of storing
on_failure	No	What to do if the operation fails (abort/warn/quiet/skip)

Output

When the store hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success     # Status of the hook execution
operation: "set"    # The operation that was performed (set/delete)
path: "my_key"      # The key that was manipulated
value: {...}        # The value that was set (only for 'set' operation)

You can access these values in subsequent hooks using the following syntax:

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.operation} - The operation performed (set/delete)

• {state.hook_id.path} - The key that was manipulated

• {state.hook_id.value} - The value that was set (only for 'set' operation)

Accessing Stored Values

To access the stored values in other hooks, use the store namespace:

{store.my_key}  # Access a value stored with key "my_key"

For example, if you stored a complex object, you can access nested properties:

{store.user.name}        # Access the "name" property of an object stored under "user"
{store.metrics.count}    # Access the "count" property of an object stored under "metrics"

Examples

Setting a Simple Value

Store a simple value for later use:

hooks:
  pre:
    - type: store
      key: processing_mode
      value: "batch"
    
    # Later, use the stored value
    - type: log
      message: "Processing in {store.processing_mode} mode"

Storing Complex Values

Store an object with multiple properties:

hooks:
  pre:
    - type: store
      key: database_config
      value:
        max_connections: 10
        timeout: 30
        retry: true
    
    # Later, access the stored values
    - type: log
      message: "Max connections: {store.database_config.max_connections}"

Using Variables in Values

Set values using variables from the runtime state:

hooks:
  post:
    - type: store
      key: execution_summary
      value:
        stream: "{run.stream.name}"
        status: "{run.status}"
        processed_rows: "{run.total_rows}"
        timestamp: "{timestamp.datetime}"
    
    # Later, use the stored summary
    - type: log
      message: "Execution summary - Stream: {store.execution_summary.stream}, Status: {store.execution_summary.status}"

Deleting Stored Values

Remove values that are no longer needed:

hooks:
  post:
    - type: store
      key: temp_data
      delete: true

Conditional Storage

Store values based on conditions:

hooks:
  post:
    - type: store
      if: "run.status == 'success' && run.total_rows > 0"
      key: validation_result
      value: "passed"
    
    - type: store
      if: "run.status == 'error' || run.total_rows == 0"
      key: validation_result
      value: "failed"
    
    # Later, use the conditional value
    - type: log
      message: "Validation result: {store.validation_result}"

Storing Results from Other Hooks

Store and access results from other hooks:

hooks:
  pre:
    - type: query
      id: count_query
      connection: postgres
      query: "SELECT COUNT(*) as row_count FROM my_table"
    
    - type: store
      key: initial_count
      value: "{state.count_query.result[0].row_count}"
    
    # Later, use the stored count
    - type: log
      message: "Initial count: {store.initial_count}"

Using Stored Values in Subsequent Hooks

Show how stored values can be used in various hooks:

hooks:
  pre:
    # Store a configuration
    - type: store
      key: config
      value:
        table_name: "customers"
        batch_size: 1000
  
  post:
    # Use in query
    - type: query
      connection: postgres
      query: "SELECT COUNT(*) FROM {store.config.table_name}"
    
    # Use in HTTP hook
    - type: http
      url: "https://api.example.com/metrics"
      method: POST
      payload: |
        {
          "table": "{store.config.table_name}",
          "batch_size": {store.config.batch_size},
          "rows_processed": {run.total_rows}
        }
    
    # Use in log message
    - type: log
      message: "Processed {run.total_rows} rows from {store.config.table_name} in batches of {store.config.batch_size}"

Building Dynamic Values

Combine stored values to build more complex structures:

hooks:
  pre:
    - type: store
      key: user
      value:
        name: "John Doe"
        role: "admin"
    
    - type: store
      key: settings
      value:
        theme: "dark"
        language: "en"
    
    # Later, create a combined report using stored values
    - type: log
      message: |
        User: {store.user.name}
        Role: {store.user.role}
        Theme: {store.settings.theme}
        Language: {store.settings.language}

Temporary Storage for Processing

Use store for intermediate processing:

hooks:
  pre:
    # Extract date components
    - type: store
      key: date_parts
      value:
        year: "{timestamp.YYYY}"
        month: "{timestamp.MM}"
        day: "{timestamp.DD}"
    
    # Build a formatted path using the stored parts
    - type: store
      key: output_path
      value: "reports/{store.date_parts.year}/{store.date_parts.month}/daily_report_{store.date_parts.day}.csv"
    
    # Use the generated path
    - type: log
      message: "Will save report to: {store.output_path}"