Read

Read hooks allow you to read the contents of files from any file-based storage connection. This is particularly useful for reading configuration files, processing text content, or incorporating file data into your workflow.

Configuration

- type: read
  from: "connection/path/to/file.txt"   # Required: Source Location
  into: "my_content"      # Optional: Store content in variable
  on_failure: abort       # Optional: abort/warn/quiet/skip
  id: my_id      # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property	Required	Description
from	Yes	The source location string. Contains connection name and file path.
into	No	Variable name to store the file content. If not specified, content is included in hook output.
on_failure	No	What to do if the read fails (abort/warn/quiet/skip)

Output

When the read hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
source_url: "s3://bucket/path/to/file.txt"  # The normalized URI of the source file
bytes_read: 1024  # Number of bytes read
content: "file content here"  # File content (only if 'into' is not specified)

You can access these values in subsequent hooks using the following syntax (jmespath):

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.source_url} - The normalized URI of the source file

• {state.hook_id.bytes_read} - Number of bytes read

• {state.hook_id.content} - File content (only if 'into' is not specified)

• {store.variable_name} - Stored content when using 'into' parameter

Examples

Read Configuration File

Read a configuration file and use its content in subsequent hooks:

hooks:
  pre:
    - type: read
      from: "s3/config/database-config.json"
      into: "db_config"
      
    - type: log
      message: "Database config: {store.db_config}"

Read Query Template

Read a SQL query template from a file and use it in a query hook:

hooks:
  pre:
    - type: read
      from: "local/queries/cleanup_template.sql"
      into: "cleanup_query"
      
    - type: query
      connection: target_db
      query: "{store.cleanup_query}"

Process Text File Content

Read a text file and process its content:

hooks:
  post:
    - type: read
      from: "gcs/reports/summary.txt"
      id: read_summary
      
    - type: log
      message: "Report summary: {state.read_summary.content}"
      
    - type: http
      url: "https://webhook.example.com/notify"
      method: POST
      payload: |
        {
          "message": "Data processing complete",
          "summary": "{state.read_summary.content}",
          "bytes_processed": {state.read_summary.bytes_read}
        }