Store

Store hooks allow you to store and manage values in memory during execution, creating a shared key-value store that can be used by subsequent hooks and replications. This hook is particularly useful for storing processing results, creating reusable variables, or maintaining state across your workflow.

The store hook supports two storage targets:

• Replication Store (default): Values stored with store.* or plain keys persist only within the current replication execution

• Environment Variables: Values stored with env.* prefix become actual environment variables, available to all subsequent steps, replications, and even API spec rendering (authentication blocks, dynamic endpoints, etc.)

Configuration

Single Key/Value:

- type: store
  key: my_key             # Required: Key to store the value under
  value: some value       # Required (unless deleting): Value to store
  delete: false           # Optional: Set to true to delete the key

Multiple Key/Values:

- type: store
  map:                    # Set multiple key/value pairs at once
    key1: value1
    key2: value2
    env.API_KEY: "abc123"
  delete: false           # Optional: Set to true to delete all keys in the map (and unset env var)

Properties

Property	Required	Description
key	Yes*	The key name to store the value under. Use env.KEY_NAME prefix to set environment variables, or store.key_name (or just key_name) for replication store. *Required unless using map
value	Yes*	Value to store at the specified key. For environment variables, the value is automatically converted to a string. *Required unless using map or deleting
map	No	A map of key/value pairs to set multiple values at once. Use this instead of key and value when setting multiple store values. Supports both env.* and store.* keys
delete	No	When set to true, deletes the key(s) and their value(s) instead of storing. Works with both key/value and map approaches
on_failure	No	What to do if the operation fails (abort/warn/quiet/skip)

Output

When the store hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success     # Status of the hook execution
operation: "set"    # The operation that was performed (set/delete)
path: "my_key"      # The key that was manipulated
value: {...}        # The value that was set (only for 'set' operation)

You can access these values in subsequent hooks using the following syntax:

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.operation} - The operation performed (set/delete)

• {state.hook_id.path} - The key that was manipulated

• {state.hook_id.value} - The value that was set (only for 'set' operation)

Accessing Stored Values

Replication Store Values

To access values stored in the replication store, use the store namespace:

{store.my_key}  # Access a value stored with key "my_key"

For example, if you stored a complex object, you can access nested properties:

{store.user.name}        # Access the "name" property of an object stored under "user"
{store.metrics.count}    # Access the "count" property of an object stored under "metrics"

Environment Variable Values

To access values stored as environment variables, use the env namespace:

{env.MY_API_KEY}     # Access environment variable MY_API_KEY
{env.DATABASE_URL}   # Access environment variable DATABASE_URL

Environment variables set via env.* prefix are:

• Available across all subsequent pipeline steps and replications

• Accessible in API specs during rendering (authentication, dynamic endpoints, etc.)

• Accessible in connection configurations

• Visible to child processes

Storage Target Comparison

Feature	Replication Store (store.*)	Environment Variables (env.*)
Scope	Current replication only	All subsequent steps/replications
API Spec Rendering	❌ Not available during spec compilation	✅ Available before spec compilation
Complex Objects	✅ Supports nested objects/arrays	❌ String values only
Use in Authentication	❌ Cannot use in auth blocks	✅ Can use in auth/dynamic blocks
Performance	Faster (in-memory)	Slightly slower (OS env vars)
Best For	Temporary data, complex objects	Configuration, credentials, tokens

Examples

Setting a Simple Value

Store a simple value for later use:

hooks:
  pre:
    - type: store
      key: processing_mode
      value: "batch"
    
    # Later, use the stored value
    - type: log
      message: "Processing in {store.processing_mode} mode"

Storing Complex Values

Store an object with multiple properties:

hooks:
  pre:
    - type: store
      key: database_config
      value:
        max_connections: 10
        timeout: 30
        retry: true
    
    # Later, access the stored values
    - type: log
      message: "Max connections: {store.database_config.max_connections}"

Using Variables in Values

Set values using variables from the runtime state:

hooks:
  post:
    - type: store
      key: execution_summary
      value:
        stream: "{run.stream.name}"
        status: "{run.status}"
        processed_rows: "{run.total_rows}"
        timestamp: "{timestamp.datetime}"
    
    # Later, use the stored summary
    - type: log
      message: "Execution summary - Stream: {store.execution_summary.stream}, Status: {store.execution_summary.status}"

Deleting Stored Values

Remove values that are no longer needed:

hooks:
  post:
    - type: store
      key: temp_data
      delete: true

Conditional Storage

Store values based on conditions:

hooks:
  post:
    - type: store
      if: "run.status == 'success' && run.total_rows > 0"
      key: validation_result
      value: "passed"
    
    - type: store
      if: "run.status == 'error' || run.total_rows == 0"
      key: validation_result
      value: "failed"
    
    # Later, use the conditional value
    - type: log
      message: "Validation result: {store.validation_result}"

Storing Results from Other Hooks

Store and access results from other hooks:

hooks:
  pre:
    - type: query
      id: count_query
      connection: postgres
      query: "SELECT COUNT(*) as row_count FROM my_table"
    
    - type: store
      key: initial_count
      value: "{state.count_query.result[0].row_count}"
    
    # Later, use the stored count
    - type: log
      message: "Initial count: {store.initial_count}"

Using Stored Values in Subsequent Hooks

Show how stored values can be used in various hooks:

hooks:
  pre:
    # Store a configuration
    - type: store
      key: config
      value:
        table_name: "customers"
        batch_size: 1000
  
  post:
    # Use in query
    - type: query
      connection: postgres
      query: "SELECT COUNT(*) FROM {store.config.table_name}"
    
    # Use in HTTP hook
    - type: http
      url: "https://api.example.com/metrics"
      method: POST
      payload: |
        {
          "table": "{store.config.table_name}",
          "batch_size": {store.config.batch_size},
          "rows_processed": {run.total_rows}
        }
    
    # Use in log message
    - type: log
      message: "Processed {run.total_rows} rows from {store.config.table_name} in batches of {store.config.batch_size}"

Building Dynamic Values

Combine stored values to build more complex structures:

hooks:
  pre:
    - type: store
      key: user
      value:
        name: "John Doe"
        role: "admin"
    
    - type: store
      key: settings
      value:
        theme: "dark"
        language: "en"
    
    # Later, create a combined report using stored values
    - type: log
      message: |
        User: {store.user.name}
        Role: {store.user.role}
        Theme: {store.settings.theme}
        Language: {store.settings.language}

Temporary Storage for Processing

Use store for intermediate processing:

hooks:
  pre:
    # Extract date components
    - type: store
      key: date_parts
      value:
        year: "{timestamp.YYYY}"
        month: "{timestamp.MM}"
        day: "{timestamp.DD}"

    # Build a formatted path using the stored parts
    - type: store
      key: output_path
      value: "reports/{store.date_parts.year}/{store.date_parts.month}/daily_report_{store.date_parts.day}.csv"

    # Use the generated path
    - type: log
      message: "Will save report to: {store.output_path}"

Setting Environment Variables

Use the env.* prefix to set environment variables that persist across pipeline steps and are available during API spec rendering.

Basic Environment Variable

Set a simple environment variable:

steps:
  - type: store
    key: env.API_KEY
    value: "sk_live_abc123xyz"

  - type: store
    key: env.ENVIRONMENT
    value: "production"

  # These env vars are now available in subsequent steps
  - type: log
    message: "Running in {env.ENVIRONMENT} environment"

Dynamic API Authentication with Environment Variables

Set environment variables that are used in API spec authentication blocks:

Pipeline Configuration:

steps:
  # Step 1: Query database to get API credentials
  - type: query
    connection: MY_CONFIG_DB
    query: |
      SELECT
        api_username,
        api_password,
        api_endpoint
      FROM api_credentials
      WHERE environment = 'production'
      LIMIT 1
    into: api_config

  # Step 2: Set credentials as environment variables
  - type: store
    key: env.API_USERNAME
    value: "{store.api_config[0].api_username}"

  - type: store
    key: env.API_PASSWORD
    value: "{store.api_config[0].api_password}"

  - type: store
    key: env.API_ENDPOINT
    value: "{store.api_config[0].api_endpoint}"

  # Step 3: Log configuration (without sensitive data)
  - type: log
    message: "API configured for endpoint: {env.API_ENDPOINT}"

  # Step 4: Run replication that uses these env vars in API spec
  - type: replication
    path: /path/to/api_replication.yaml

API Spec (using environment variables):

name: "My API"

# Environment variables are available during spec rendering
authentication:
  type: basic
  username: "{env.API_USERNAME}"
  password: "{env.API_PASSWORD}"

defaults:
  state:
    base_url: "{env.API_ENDPOINT}"
  request:
    headers:
      X-Environment: "{env.ENVIRONMENT}"

endpoints:
  users:
    request:
      url: "{state.base_url}/users"

Key Advantage: Environment variables set via env.* hooks are available before API specs are compiled/rendered, making them usable in authentication blocks and dynamic endpoint definitions where store.* variables cannot be used.

Setting Multiple Values at Once

Use the map property to set multiple store or environment variables in a single step:

steps:
  # Set multiple environment variables at once
  - type: store
    map:
      env.API_USERNAME: "api_user_prod"
      env.API_PASSWORD: "secure_pass_123"
      env.API_ENDPOINT: "https://api.example.com"
      env.ENVIRONMENT: "production"

  # Set multiple store values at once
  - type: store
    map:
      retry_count: 3
      timeout: 30
      batch_size: 1000

  # Mix environment variables and store values
  - type: store
    map:
      env.DEPLOYMENT: "prod" # set env var
      cache_enabled: true
      max_connections: 10
      database_name: "production_db"

  # Use the stored values
  - type: log
    message: |
      Configuration:
      - Environment: {env.ENVIRONMENT}
      - Endpoint: {env.API_ENDPOINT}
      - Batch Size: {store.config.batch_size}
      - Max Connections: {store.max_connections}

Dynamic Map from Query Results

Build a map dynamically from query results:

steps:
  # Query database for configuration
  - type: query
    connection: MY_CONFIG_DB
    query: |
      SELECT
        'env.API_KEY' as key,
        api_key as value
      FROM api_config
      WHERE environment = 'production'
      UNION ALL
      SELECT
        'env.API_ENDPOINT' as key,
        api_endpoint as value
      FROM api_config
      WHERE environment = 'production'
    into: config_rows

  # Set all config values at once using map
  - type: store
    map:
      env.API_KEY: "{store.config_rows[0].value}"
      env.API_ENDPOINT: "{store.config_rows[1].value}"

Deleting Environment Variables

Remove environment variables when no longer needed:

steps:
  - type: store
    key: env.TEMP_TOKEN
    value: "temporary_value"

  # Use the token...

  - type: store
    key: env.TEMP_TOKEN
    delete: true  # Removes the environment variable

Deleting Multiple Variables

Remove multiple keys at once:

steps:
  # Set temporary variables
  - type: store
    map:
      env.TEMP_TOKEN: "token_123"
      env.TEMP_SESSION: "session_456"
      temp_data: "some_data"

  # Use the temporary variables...

  # Delete all temporary variables at once
  - type: store
    map:
      env.TEMP_TOKEN: ""
      env.TEMP_SESSION: ""
      temp_data: ""
    delete: true