Log

Log hooks allow you to output custom messages during the replication process. This is useful for debugging, monitoring, and creating audit trails of your data pipeline operations.

Configuration

- type: log
  message: "Custom log message"  # Required: The message to log
  level: info                    # Optional: Log level (info/warn/debug)
  on_failure: abort              # Optional: abort/warn/quiet/skip

Properties

Property	Required	Description
message	Yes	The message to log
level	No	Log level (info/warn/debug). Defaults to info
on_failure	No	What to do if logging fails (abort/warn/quiet/skip)

Output

When the log hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
level: "info"    # The log level used
message: "Custom log message"  # The message that was logged

You can access these values in subsequent hooks using the following syntax (jmespath):

• {state.hook_id.status} - Status of the hook execution

• {state.hook_id.level} - The log level used

• {state.hook_id.message} - The message that was logged

Examples

Print out Runtime State

Log the basic status of a stream after completion:

hooks:
  post:
    - type: log
      message: "runtime_state => {runtime_state}"

    - type: log
      message: "Stream {run.stream.name} completed with status {run.status}. Processed {run.total_rows} rows."

Conditional Warning Log

Log a warning when row count is below threshold:

hooks:
  post:
    - type: log
      if: "run.total_rows < 1000"
      message: "⚠️ Warning: Low record count for {run.stream.name}. Only processed {run.total_rows} rows."
      level: warn

Debug Information

Log detailed information before starting the replication:

hooks:
  pre:
    - type: log
      message: |
        Starting replication for:
        Stream: {run.stream.name}
        Source Schema: {run.stream.schema}
        Target Table: {run.object.full_name}
        Environment: {target.environment}
        Timestamp: {timestamp.datetime}
      level: debug

Performance Metrics Logging

Log performance metrics after successful completion:

hooks:
  post:
    - type: log
      if: run.status == "success"
      message: |
        Performance Metrics for {run.stream.name}:
        - Total Rows: {run.total_rows}
        - Total Bytes: {run.total_bytes}
        - Duration: {run.duration} seconds
        - Processing Rate: {run.total_rows / (run.duration)} rows/second
      level: info

Error Context Logging

Log detailed context when errors occur:

hooks:
  post:
    - type: log
      if: run.status == "error"
      message: |
        ❌ Error in stream {run.stream.name}:
        - Source: {source.type}
        - Target: {target.type}
        - Object: {run.object.full_name}
        - Environment: {target.environment}
        - Start Time: {run.start_time}
        - End Time: {run.end_time}
        Please check the logs for more details.
      level: warn

Audit Trail Logging

Create a detailed audit trail of replication activities:

hooks:
  pre:
    - type: log
      message: |
        🚀 Starting replication task:
        - Stream: {run.stream.name}
        - Mode: {run.mode}
        - Environment: {target.environment}
        - User: {source.user}
        - Start Time: {timestamp.datetime}
      level: info
  post:
    - type: log
      message: |
        📋 Replication task completed:
        - Stream: {run.stream.name}
        - Status: {run.status}
        - Rows Processed: {run.total_rows}
        - Duration: {run.end_time - run.start_time} seconds
        - End Time: {timestamp.datetime}
      level: info

Data Quality Logging

Log data quality metrics after processing:

hooks:
  post:
    - type: log
      message: |
        Data Quality Report for {run.stream.name}:
        - Total Records: {run.total_rows}
        - Null Rate: {state.quality_check.result.null_rate}%
        - Duplicate Rate: {state.quality_check.result.duplicate_rate}%
        - Invalid Format Rate: {state.quality_check.result.invalid_rate}%
      level: info

8. Environment-Specific Logging (Pre-Hook)

Adjust log verbosity based on environment:

hooks:
  pre:
    - type: log
      if: "target.environment == 'production'"
      message: "⚠️ Running production replication for {run.stream.name}"
      level: warn
    - type: log
      if: "target.environment != 'production'"
      message: "Starting {target.environment} replication for {run.stream.name}"
      level: debug

9. Resource Usage Logging (Post-Hook)

Log resource usage statistics:

hooks:
  post:
    - type: log
      message: |
        Resource Usage for {run.stream.name}:
        - Memory Used: {state.resource_check.result.memory_used}
        - CPU Usage: {state.resource_check.result.cpu_usage}%
        - Disk IO: {state.resource_check.result.disk_io_bytes} bytes
      level: debug

10. Milestone Logging (Post-Hook)

Log important milestones during processing:

hooks:
  post:
    - type: log
      if: "run.total_rows >= 1000000"
      message: "🏆 Milestone: Processed over 1 million records in {run.stream.name}"
      level: info
    - type: log
      if: "run.total_bytes >= 1073741824"  # 1GB
      message: "🏆 Milestone: Processed over 1GB of data in {run.stream.name}"
      level: info