Http

HTTP hooks enable you to make HTTP requests to external services as part of your replication workflow. This is particularly useful for integrating with external APIs, sending notifications, or triggering other systems.

Configuration

- type: http
  url: "https://api.example.com/webhook"  # Required
  method: GET                             # Optional: GET/POST/PUT/DELETE (default: GET)
  payload: '{"status": "{run.status}"}'   # Optional: Request body
  headers:                                # Optional: Request headers
    Authorization: "Bearer token"
  on_failure: abort                       # Optional: abort/warn/quiet/skip
  id: my_id                               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property
Required
Description

url

Yes

The URL to send the request to

method

No

HTTP method (GET/POST/PUT/DELETE). Defaults to GET

payload

No

The request body (for POST/PUT requests)

headers

No

Map of HTTP headers to include in the request

on_failure

No

What to do if the request fails (abort/warn/quiet/skip)

Output

When the HTTP hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
request:  # Details of the request made
  method: "GET"  # HTTP method used
  url: "https://api.example.com/webhook"  # The URL called
  headers:  # Headers sent with the request
    Authorization: "Bearer token"
  payload: '{"status": "success"}'  # The request body sent
response:  # Details of the response received
  headers:  # Response headers
    Content-Type: "application/json"
    # ... other response headers
  status: "200 OK"  # HTTP status message
  status_code: 200  # HTTP status code
  text: "Response body as text"  # Raw response body
  json:  # Parsed JSON response (if response is JSON)
    key: "value"
    # ... rest of JSON structure

You can access these values in subsequent hooks using the following syntax (jmespath):

  • {state.hook_id.status} - Status of the hook execution

  • {state.hook_id.request.method} - HTTP method used

  • {state.hook_id.request.url} - The URL called

  • {state.hook_id.request.headers} - Request headers

  • {state.hook_id.request.payload} - Request body

  • {state.hook_id.response.status} - HTTP status message

  • {state.hook_id.response.status_code} - HTTP status code

  • {state.hook_id.response.text} - Raw response body

  • {state.hook_id.response.json} - Parsed JSON response

Examples

Slack Notification

Send a notification to Slack after replication completes:

hooks:
  post:
    - type: http
      url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
      method: POST
      payload: |
        {
          "text": "Replication Status Update",
          "blocks": [
            {
              "type": "section",
              "text": {
                "type": "mrkdwn",
                "text": "*Stream:* {run.stream.name}\n*Status:* {run.status}\n*Rows Processed:* {run.total_rows}\n*Duration:* {run.duration} seconds"
              }
            }
          ]
        }
      on_failure: warn

Microsoft Teams Alert

Send an alert to Microsoft Teams when replication fails:

hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://your-teams-webhook-url"
      method: POST
      payload: |
        {
          "@type": "MessageCard",
          "@context": "http://schema.org/extensions",
          "themeColor": "FF0000",
          "summary": "Replication Failed",
          "sections": [{
            "activityTitle": "⚠️ Replication Failed",
            "facts": [
              {
                "name": "Stream",
                "value": "{run.stream.name}"
              },
              {
                "name": "Target Table",
                "value": "{run.object.full_name}"
              },
              {
                "name": "Start Time",
                "value": "{run.start_time}"
              },
              {
                "name": "End Time",
                "value": "{run.end_time}"
              }
            ]
          }]
        }

REST API Integration

Fetch configuration from an external API before starting replication:

hooks:
  pre:
    - type: http
      url: "https://api.company.com/v1/config/{run.stream.name}"
      method: GET
      headers:
        Authorization: "Bearer {source.api_key}"
        Content-Type: "application/json"

Trigger External Workflow

Trigger an external workflow system after successful replication:

hooks:
  post:
    - type: http
      if: run.status == "success"
      url: "https://api.workflow-system.com/v1/triggers"
      method: POST
      headers:
        Authorization: "ApiKey {target.api_key}"
      payload: |
        {
          "workflow_id": "data-quality-check",
          "parameters": {
            "table_name": "{run.object.full_name}",
            "row_count": {run.total_rows},
            "execution_date": "{timestamp.date}"
          }
        }

Data Quality Service Integration

Send data quality metrics to a monitoring service:

hooks:
  post:
    - type: http
      url: "https://metrics-api.company.com/v1/metrics"
      method: POST
      headers:
        X-API-Key: "{target.metrics_api_key}"
      payload: |
        {
          "metric_type": "data_quality",
          "timestamp": "{run.end_time}",
          "metrics": {
            "table_name": "{run.object.full_name}",
            "record_count": {run.total_rows},
            "processing_time_seconds": {run.duration},
            "bytes_processed": {run.total_bytes}
          },
          "tags": {
            "environment": "{env.environment}",
            "stream": "{run.stream.name}"
          }
        }

Error Tracking Integration

Send error details to an error tracking service when replication fails:

hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://api.errortrackers.com/v1/errors"
      method: POST
      headers:
        Authorization: "Bearer {target.error_tracking_token}"
      payload: |
        {
          "error": {
            "name": "Replication Failed",
            "environment": "{target.environment}",
            "metadata": {
              "stream": "{run.stream.name}",
              "target_table": "{run.object.full_name}",
              "start_time": "{run.start_time}",
              "end_time": "{run.end_time}",
              "rows_processed": {run.total_rows}
            }
          }
        }
      on_failure: warn

Last updated