Log hooks allow you to output custom messages during the replication process. This is useful for debugging, monitoring, and creating audit trails of your data pipeline operations.
Configuration
- type: log
message: "Custom log message" # Required: The message to log
level: info # Optional: Log level (info/warn/debug)
on_failure: abort # Optional: abort/warn/quiet/skip
Properties
Property
Required
Description
message
Yes
The message to log
level
No
Log level (info/warn/debug). Defaults to info
on_failure
No
What to do if logging fails (abort/warn/quiet/skip)
Output
When the log hook executes successfully, it returns the following output that can be accessed in subsequent hooks:
status: success # Status of the hook execution
level: "info" # The log level used
message: "Custom log message" # The message that was logged
You can access these values in subsequent hooks using the following syntax (jmespath):
{state.hook_id.status} - Status of the hook execution
{state.hook_id.level} - The log level used
{state.hook_id.message} - The message that was logged
Examples
Print out Runtime State
Log the basic status of a stream after completion:
hooks:
post:
- type: log
message: |
Resource Usage for {run.stream.name}:
- Memory Used: {state.resource_check.result.memory_used}
- CPU Usage: {state.resource_check.result.cpu_usage}%
- Disk IO: {state.resource_check.result.disk_io_bytes} bytes
level: debug
10. Milestone Logging (Post-Hook)
Log important milestones during processing:
hooks:
post:
- type: log
if: "run.total_rows >= 1000000"
message: "🏆 Milestone: Processed over 1 million records in {run.stream.name}"
level: info
- type: log
if: "run.total_bytes >= 1073741824" # 1GB
message: "🏆 Milestone: Processed over 1GB of data in {run.stream.name}"
level: info