BlogGithub

Http

HTTP hooks enable you to make HTTP requests to external services as part of your replication workflow. This is particularly useful for integrating with external APIs, sending notifications, or triggering other systems.

Configuration

- type: http
  url: "https://api.example.com/webhook"  # Required
  method: GET                             # Optional: GET/POST/PUT/DELETE (default: GET)
  payload: '{"status": "{run.status}"}'   # Optional: Request body (also accepts YAML object)
  timeout: 10                             # Optional: Request timeout seconds (default is 30sec)
  headers:                                # Optional: Request headers
    Authorization: "Bearer token"
  auth:                                   # Optional: Authentication configuration
    type: basic                           # basic, aws-sigv4, or hmac
    username: "myuser"
    password: "mypass"
  proxy: "http://user:[email protected]:8080"  # Optional: HTTP/HTTPS proxy URL
  write_to: local/path/to/response.json   # Optional: Save response to a file
  on_failure: abort                       # Optional: abort/warn/quiet/skip
  id: my_id                               # Optional. Will be generated. Use `log` hook with {runtime_state} to view state.

Properties

Property
Required
Description

url

Yes

The URL to send the request to

method

No

HTTP method (GET/POST/PUT/DELETE). Defaults to GET

payload

No

The request body (for POST/PUT requests)

headers

No

Map of HTTP headers to include in the request

auth

No

Authentication configuration (see Authentication section)

proxy

No

HTTP/HTTPS proxy URL (e.g., http://user:[email protected]:8080)

write_to

No

Location to save the response bytes to a file (e.g., local/path/to/response.json, s3/folder/response.csv)

on_failure

No

What to do if the request fails (abort/warn/quiet/skip)

Output

When the HTTP hook executes successfully, it returns the following output that can be accessed in subsequent hooks:

status: success  # Status of the hook execution
request:  # Details of the request made
  method: "GET"  # HTTP method used
  url: "https://api.example.com/webhook"  # The URL called
  headers:  # Headers sent with the request
    Authorization: "Bearer token"
  payload: '{"status": "success"}'  # The request body sent
response:  # Details of the response received
  headers:  # Response headers
    Content-Type: "application/json"
    # ... other response headers
  status: "200 OK"  # HTTP status message
  status_code: 200  # HTTP status code
  size: 123456  # Response bytes size 
  text: "Response body as text"  # Raw response body
  json:  # Parsed JSON response (if response is JSON)
    key: "value"
    # ... rest of JSON structure
written_to:  # Details of saved response file (if write_to is specified)
  location: "local/path/to/response.json"  # The write_to location
  uri: "file:///absolute/path/to/response.json"  # Absolute URI of saved file
  bytes_written: 1234  # Number of bytes written to file

You can access these values in subsequent hooks using the following syntax (jmespath):

  • {state.hook_id.status} - Status of the hook execution

  • {state.hook_id.request.method} - HTTP method used

  • {state.hook_id.request.url} - The URL called

  • {state.hook_id.request.headers} - Request headers

  • {state.hook_id.request.payload} - Request body

  • {state.hook_id.response.status} - HTTP status message

  • {state.hook_id.response.status_code} - HTTP status code

  • {state.hook_id.response.text} - Raw response body

  • {state.hook_id.response.json} - Parsed JSON response

  • {state.hook_id.response.size} - Size of response in bytes

  • {state.hook_id.written_to.location} - Location where response was saved

  • {state.hook_id.written_to.uri} - Absolute URI of saved response file

Examples

Slack Notification

Send a notification to Slack after replication completes:

hooks:
  post:
    - type: http
      url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
      method: POST
      payload: |
        {
          "text": "Replication Status Update",
          "blocks": [
            {
              "type": "section",
              "text": {
                "type": "mrkdwn",
                "text": "*Stream:* {run.stream.name}\n*Status:* {run.status}\n*Rows Processed:* {run.total_rows}\n*Duration:* {run.duration} seconds"
              }
            }
          ]
        }
      on_failure: warn

Microsoft Teams Alert

Send an alert to Microsoft Teams when replication fails:

hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://your-teams-webhook-url"
      method: POST
      payload: |
        {
          "@type": "MessageCard",
          "@context": "http://schema.org/extensions",
          "themeColor": "FF0000",
          "summary": "Replication Failed",
          "sections": [{
            "activityTitle": "⚠️ Replication Failed",
            "facts": [
              {
                "name": "Stream",
                "value": "{run.stream.name}"
              },
              {
                "name": "Target Table",
                "value": "{run.object.full_name}"
              },
              {
                "name": "Start Time",
                "value": "{run.start_time}"
              },
              {
                "name": "End Time",
                "value": "{run.end_time}"
              }
            ]
          }]
        }

REST API Integration

Fetch configuration from an external API before starting replication:

hooks:
  pre:
    - type: http
      url: "https://api.company.com/v1/config/{run.stream.name}"
      method: GET
      headers:
        Authorization: "Bearer {source.api_key}"
        Content-Type: "application/json"

API with Basic Authentication

Call an API endpoint that requires basic authentication:

hooks:
  post:
    - type: http
      url: "https://api.example.com/v1/notify"
      method: POST
      auth:
        type: basic
        username: "{env.API_USER}"
        password: "{env.API_PASS}"
      payload: |
        {
          "event": "replication_complete",
          "stream": "{run.stream.name}",
          "rows": {run.total_rows}
        }

API with HMAC Signature

Call an API that requires HMAC signature authentication:

hooks:
  post:
    - type: http
      url: "https://partner-api.example.com/webhooks/events"
      method: POST
      auth:
        type: hmac
        algorithm: sha256
        secret: "{env.PARTNER_HMAC_SECRET}"
        signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
        request_headers:
          X-Signature: "sha256={signature}"
          X-Timestamp: "{unix_time}"
      payload: |
        {
          "event_type": "replication.complete",
          "stream": "{run.stream.name}",
          "status": "{run.status}",
          "rows_processed": {run.total_rows},
          "timestamp": "{run.end_time}"
        }

Trigger External Workflow

Trigger an external workflow system after successful replication:

hooks:
  post:
    - type: http
      if: run.status == "success"
      url: "https://api.workflow-system.com/v1/triggers"
      method: POST
      headers:
        Authorization: "ApiKey {target.api_key}"
      payload: |
        {
          "workflow_id": "data-quality-check",
          "parameters": {
            "table_name": "{run.object.full_name}",
            "row_count": {run.total_rows},
            "execution_date": "{timestamp.date}"
          }
        }

Data Quality Service Integration

Send data quality metrics to a monitoring service:

hooks:
  post:
    - type: http
      url: "https://metrics-api.company.com/v1/metrics"
      method: POST
      headers:
        X-API-Key: "{target.metrics_api_key}"
      payload: |
        {
          "metric_type": "data_quality",
          "timestamp": "{run.end_time}",
          "metrics": {
            "table_name": "{run.object.full_name}",
            "record_count": {run.total_rows},
            "processing_time_seconds": {run.duration},
            "bytes_processed": {run.total_bytes}
          },
          "tags": {
            "environment": "{env.environment}",
            "stream": "{run.stream.name}"
          }
        }

Error Tracking Integration

Send error details to an error tracking service when replication fails:

hooks:
  post:
    - type: http
      if: run.status == "error"
      url: "https://api.errortrackers.com/v1/errors"
      method: POST
      headers:
        Authorization: "Bearer {target.error_tracking_token}"
      payload: |
        {
          "error": {
            "name": "Replication Failed",
            "environment": "{target.environment}",
            "metadata": {
              "stream": "{run.stream.name}",
              "target_table": "{run.object.full_name}",
              "start_time": "{run.start_time}",
              "end_time": "{run.end_time}",
              "rows_processed": {run.total_rows}
            }
          }
        }
      on_failure: warn

Download and Save API Response

Fetch data from an API and save the response to a file:

hooks:
  pre:
    - type: http
      id: fetch_config
      url: "https://api.example.com/v1/config"
      method: GET
      headers:
        Authorization: "Bearer {env.API_TOKEN}"
      write_to: local/configs/api_response.json
    
    # Use the saved response location in a subsequent hook
    - type: log
      message: "API response saved to {state.fetch_config.written_to.uri} ({state.fetch_config.written_to.bytes_written} bytes)"

Authentication

The HTTP hook supports multiple authentication methods through the auth property:

Basic Authentication

Use HTTP Basic Authentication with username and password:

- type: http
  url: "https://api.example.com/endpoint"
  auth:
    type: basic
    username: "{env.API_USERNAME}"
    password: "{env.API_PASSWORD}"

AWS Signature V4

Sign requests using AWS Signature Version 4 (for AWS services):

- type: http
  url: "https://my-service.us-east-1.amazonaws.com/endpoint"
  auth:
    type: aws-sigv4
    aws_service: execute-api
    aws_region: us-east-1
    aws_access_key_id: "{env.AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "{env.AWS_SECRET_ACCESS_KEY}"
    # Optional:
    # aws_session_token: "{env.AWS_SESSION_TOKEN}"
    # aws_profile: "my-profile"

Note: AWS credentials can also be loaded from environment variables or AWS profiles if not explicitly provided.

HMAC Signature

Sign requests using HMAC (Hash-based Message Authentication Code) for custom API authentication:

- type: http
  url: "https://api.example.com/endpoint"
  auth:
    type: hmac
    algorithm: sha256  # or sha512 (default: sha256)
    secret: "{env.HMAC_SECRET}"
    signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
    request_headers:
      X-Signature: "{signature}"
      X-Timestamp: "{unix_time}"
      # Optional nonce:
      # X-Nonce: "{nonce}"
    # nonce_length: 16  # Optional: generate random nonce (in bytes)

Available template variables for signing_string and request_headers:

  • {http_method} - HTTP method (GET, POST, etc.)

  • {http_path} - Request path including query string

  • {http_body_md5} - MD5 hash of request body (hex)

  • {http_body_sha1} - SHA1 hash of request body (hex)

  • {http_body_sha256} - SHA256 hash of request body (hex)

  • {http_body_sha512} - SHA512 hash of request body (hex)

  • {http_body_raw} - Raw request body as string

  • {http_query} - Canonical query string (sorted, URL-encoded)

  • {http_headers} - Canonical headers string (lowercase, sorted)

  • {unix_time} - Current Unix timestamp (seconds)

  • {unix_time_ms} - Current Unix timestamp (milliseconds)

  • {date_iso} - Current date in ISO 8601 format (RFC3339)

  • {date_rfc1123} - Current date in RFC1123 format

  • {nonce} - Random nonce (if nonce_length is set)

  • {signature} - The computed HMAC signature (available in request_headers)

Example with complete HMAC flow:

- type: http
  url: "https://api.partner.com/v1/orders"
  method: POST
  payload: '{"order_id": "12345"}'
  auth:
    type: hmac
    algorithm: sha256
    secret: "{env.PARTNER_API_SECRET}"
    signing_string: "{http_method}\n{http_path}\n{unix_time}\n{http_body_sha256}"
    nonce_length: 16
    request_headers:
      Authorization: "HMAC-SHA256 {signature}"
      X-Timestamp: "{unix_time}"
      X-Nonce: "{nonce}"

Bearer Token (via Headers)

For Bearer token authentication, use the headers property instead of auth:

- type: http
  url: "https://api.example.com/endpoint"
  headers:
    Authorization: "Bearer {env.API_TOKEN}"

Proxy Configuration

If your network requires routing HTTP requests through a proxy server, you can configure it using the proxy property:

- type: http
  url: "https://api.example.com/endpoint"
  proxy: "http://proxy.company.com:8080"

Proxy with Authentication

For proxies that require authentication, include credentials in the proxy URL:

- type: http
  url: "https://api.example.com/endpoint"
  proxy: "http://{env.PROXY_USER}:{env.PROXY_PASS}@proxy.company.com:8080"

The proxy URL format is: http://[username:password@]host:port

Notes:

  • Both HTTP and HTTPS proxies are supported

  • Authentication credentials in the proxy URL are automatically handled

  • The proxy applies to all HTTP/HTTPS requests made by this hook

PreviousGroupNextInspect

Last updated

Was this helpful?