Requests & Iteration

This page details how to configure HTTP requests and use the iteration feature for looping within Sling API specifications.

Request Configuration

Each endpoint defines its HTTP request details under the request key. These settings merge with and override the defaults.request configuration.

Request Properties

Property	Required	Description	Example
url	Yes	Path (relative to defaults.request.url) or full URL	"users" or "https://api.example.com/users"
method	No	HTTP method (default: "GET")	"POST", "PUT", "PATCH", "DELETE"
headers	No	HTTP headers to send	{"Content-Type": "application/json"}
parameters	No	Query parameters (or form fields for POST)	{"page": 1, "limit": 100}
payload	No	Request body for POST/PUT/PATCH	{"name": "New User"}
timeout	No	Request timeout in seconds (default: 30)	60
rate	No	Max requests per second (default: 2)	5
concurrency	No	Max concurrent requests (default: 5)	10

Example Request Configuration

request:
  state:
    base_url: https://api.example.com/v1
    user_id: 123
  
  url: '{state.base_url}/users/{state.user_id}'
  
  # HTTP Method
  method: "POST"
  
  # Headers (merged with defaults.request.headers)
  headers:
    Content-Type: "application/json"
    Authorization: "Bearer {auth.token}"
    X-Request-ID: "{uuid()}"
    
  # Query Parameters (or form parameters for POST with application/x-www-form-urlencoded)
  parameters:
    active: true
    department: "{state.department}"
    
  # Request Body (for POST/PUT/PATCH methods)
  payload:
    user:
      name: "New User"
      email: "{state.user_email}"
      
  # Request timeout (in seconds)
  timeout: 60
  
  # Rate limiting (max requests per second)
  rate: 5
  
  # Concurrency (max in-flight requests)
  concurrency: 3

📝 Note: When method is GET or DELETE, the parameters are added as URL query parameters. When method is POST, PUT, or PATCH and Content-Type is application/x-www-form-urlencoded, the parameters are sent as form fields.

💡 Tip: Use {...} expressions to make request values dynamic based on state variables, authentication details, or environment variables.

Iteration (Looping Requests)

The iterate section allows an endpoint to make multiple requests based on a list of items (e.g., IDs from a queue, date ranges).

How Iteration Works

Iteration Properties

Property	Required	Description	Example
over	Yes	Expression to get items to iterate over	"queue.user_ids" or "[1, 2, 3]"
into	Yes	State variable to store current item	"state.current_id"
concurrency	No	Max iterations to run in parallel (default: 10)	5
if	No	Condition to evaluate before starting iteration	"state.should_process == true"

Example: Basic Iteration

iterate:
  # Get items from a queue filled by another endpoint
  over: "queue.product_ids"
  
  # Store current item in state variable
  into: "state.current_product_id"
  
  # Run up to 5 iterations concurrently
  concurrency: 5

⚠️ Important: Each iteration gets its own state. Changes to state variables in one iteration don't affect other iterations.

Example: Date Range Iteration

This pattern is great for splitting large data extractions into daily chunks:

endpoints:
  daily_reports:
    state:
      # Configure date range (last 7 days by default)
      start_date: '{coalesce(env.START_DATE, date_format(date_add(now(), -7, "day"), "%Y-%m-%d"))}'
      end_date: '{coalesce(env.END_DATE, date_format(now(), "%Y-%m-%d"))}'
    
    iterate:
      # Generate an array of dates between start_date and end_date, one day at a time
      over: >
        range(
          date_parse(state.start_date, "%Y-%m-%d"), 
          date_parse(state.end_date, "%Y-%m-%d"),
          "1d"
        )
      # Store the current date in state.current_day
      into: state.current_day
      concurrency: 10
    
    request:
      url: '{state.base_url}/reports'
      parameters:
        # Format the date for the API
        date: '{date_format(state.current_day, "%Y-%m-%d")}'

Example: Batch Processing with chunk()

When an API accepts multiple IDs in a single request, use chunk() to process them in batches:

endpoints:
  lookup_variants:
    iterate:
      # Split queue into batches of 50 IDs each
      over: "chunk(queue.variant_ids, 50)"
      # state.variant_id_batch will be an array of up to 50 IDs
      into: "state.variant_id_batch"
      concurrency: 5

    request:
      url: '{state.base_url}/variants/lookup'
      parameters:
        # Join IDs into comma-separated string
        ids: '{join(state.variant_id_batch, ",")}'

Using Context Variables for Backfill Ranges

Context variables are runtime values passed from the replication configuration to the API spec. They're particularly useful for supporting both backfill and incremental modes with the same endpoint.

Key context variables:

• context.range_start - Start of backfill range (from source_options.range)

• context.range_end - End of backfill range (from source_options.range)

Example: Date Range with Context Support

endpoints:
  events:
    # Persist last processed date for incremental runs
    sync: [last_date]

    iterate:
      # Backfill mode: Use context.range_start/range_end from config
      # Incremental mode: Use sync.last_date
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events"
      parameters:
        date: "{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]

      processors:
        # Track last processed date
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"

Replication Config for Backfill:

source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    source_options:
      # Backfill specific date range
      range: '2024-01-01,2024-01-31'

Replication Config for Incremental:

source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    # No range - uses sync.last_date for incremental

This pattern allows a single endpoint to handle both historical backfills and ongoing incremental updates seamlessly. See Context Variables for complete details.

Request Flow Control

You can control the request flow using these features:

Feature	Location	Purpose	Example
rate	request	Limit requests per second	rate: 5
concurrency	request	Limit parallel requests	concurrency: 3
concurrency	iterate	Limit parallel iterations	concurrency: 10
if	iterate	Conditional iteration	if: "state.has_data"
timeout	request	Set request timeout	timeout: 30

💡 Tip: Balance performance and API limits:

• Too low concurrency: slower extraction but gentler on the API

• Too high concurrency: faster extraction but may trigger rate limits

• iterate.concurrency controls parallelism across different IDs/items

• request.concurrency controls parallelism across paginated requests

Sequences: Setup and Teardown

Endpoints can define optional setup and teardown sequences that run before and after the main requests. These are multi-step workflows perfect for initialization, cleanup, or complex processes.

What are Sequences?

A sequence is an ordered array of API calls. Each call in a sequence can:

• Make HTTP requests

• Process responses

• Update state variables

• Use pagination

• Execute conditionally

Sequences are used in three places:

1. Authentication - Custom authentication workflows

2. Setup - Pre-execution initialization

3. Teardown - Post-execution cleanup

Sequence Call Structure

Each call in a sequence has the same structure:

- if: <conditional expression>      # Optional: Execute only if true
  request: <request configuration>   # Same as endpoint request
  pagination: <pagination config>    # Optional: Paginate this call
  response: <response configuration> # Process response

Setup Sequences

The setup sequence runs once before the endpoint's main requests/iterations begin. Perfect for:

• Fetching configuration data

• Initializing session data

• Validating prerequisites

• Loading dynamic parameters

Basic Setup Example

setup:
  - request:
      url: "{state.base_url}/config"
      method: GET
    response:
      processors:
        - expression: 'jmespath(response.json, "config.api_version")'
          output: "state.api_version"
          aggregation: last
        - expression: 'jmespath(response.json, "config.base_url")'
          output: "state.base_url"
          aggregation: last
        - expression: 'jmespath(response.json, "config.rate_limit")'
          output: "state.rate_limit"
          aggregation: last

Multi-Step Setup Example

setup:
  # Step 1: Get session token
  - request:
      url: "{state.base_url}/session"
      method: POST
      payload:
        app_id: "{secrets.app_id}"
    response:
      processors:
        - expression: "response.json.session_token"
          output: "state.session_token"
          aggregation: last

  # Step 2: Use session token to get user info
  - request:
      url: "{state.base_url}/user/me"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: 'jmespath(response.json, "user.id")'
          output: "state.user_id"
          aggregation: last
        - expression: 'jmespath(response.json, "user.permissions")'
          output: "state.user_permissions"
          aggregation: last

  # Step 3: Conditionally load admin data
  - if: contains(state.user_permissions, "admin")
    request:
      url: "{state.base_url}/admin/settings"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.admin_config"
          output: "state.admin_config"
          aggregation: last

Teardown Sequences

The teardown sequence runs once after all main requests complete. Perfect for:

• Closing sessions

• Cleaning up temporary resources

• Logging completion status

• Archiving processed data

Basic Teardown Example

teardown:
  - request:
      url: "{state.base_url}/session/close"
      method: POST
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.status"
          output: "state.session_close_status"
          aggregation: last

Conditional Teardown Example

teardown:
  # Only cleanup if we created temporary data
  - if: state.created_temp_data == true
    request:
      url: "{state.base_url}/cleanup"
      method: POST
      payload:
        session_id: "{state.session_id}"
        temp_ids: "{state.temp_resource_ids}"
    response:
      processors:
        - expression: 'jmespath(response.json, "cleanup.status")'
          output: "state.cleanup_status"
          aggregation: last

  # Always log final statistics
  - request:
      url: "{state.base_url}/analytics/log"
      method: POST
      payload:
        endpoint_name: "{env.ENDPOINT_NAME}"
        records_processed: "{state.total_records}"
        duration_seconds: "{state.duration}"

Conditional Execution with if

Each call in a sequence can execute conditionally using the if field:

setup:
  # Always runs
  - request:
      url: "{state.base_url}/status"
    response:
      processors:
        - expression: "response.json.api_status"
          output: "state.api_status"
          aggregation: last

  # Only runs if API is in maintenance mode
  - if: state.api_status == "maintenance"
    request:
      url: "{state.base_url}/maintenance/info"
    response:
      processors:
        - expression: "response.json.maintenance_until"
          output: "state.maintenance_until"
          aggregation: last
        - expression: log("API in maintenance until: " + state.maintenance_until)
          output: ""

Common if Patterns:

# Check state variable existence
if: '!is_null(state.session_token)'

# Check for specific value
if: state.environment == "production"

# Check array membership
if: contains(state.enabled_features, "advanced_mode")

# Multiple conditions
if: state.user_type == "admin" && !is_null(state.admin_token)

# Check environment variable
if: env.ENABLE_FEATURE == "true"

Pagination in Sequences

Sequence calls support pagination, useful for multi-page setup data:

setup:
  # Load all available categories (paginated)
  - request:
      url: "{state.base_url}/categories"
      parameters:
        limit: 100

    pagination:
      next_state:
        offset: "{state.offset + 100}"
      stop_condition: "length(response.records) < 100"

    response:
      records:
        jmespath: "categories[]"
      processors:
        # Collect all category IDs
        - expression: "record.id"
          output: "queue.category_ids"

State Management in Sequences

Important behaviors:

1. State Isolation: Setup/teardown sequences have their own state copy

2. State Merging: Changes to state persist back to the main endpoint

3. Headers Inherited: Request headers from the main endpoint are copied

4. No Iteration State: Sequences don't have iteration-specific state

endpoints:
  my_endpoint:
    state:
      base_url: "https://api.example.com"
      initial_value: 100

    setup:
      - request:
          url: "{state.base_url}/init"
        response:
          processors:
            # This updates the endpoint's state
            - expression: "response.json.config_value"
              output: "state.config_value"
              aggregation: last

    request:
      # Can now use state.config_value from setup
      url: "{state.base_url}/data"
      parameters:
        config: "{state.config_value}"

Complete Sequence Example

Here's a real-world example showing setup, main execution, and teardown:

endpoints:
  export_data:
    state:
      base_url: "https://api.example.com/v2"

    setup:
      # Step 1: Request export job creation
      - request:
          url: "{state.base_url}/exports"
          method: POST
          payload:
            format: "csv"
            filters:
              date_from: "{state.start_date}"
              date_to: "{state.end_date}"
        response:
          processors:
            - expression: "response.json.export_id"
              output: "state.export_id"
              aggregation: last
            - expression: log("Created export job: " + response.json.export_id)
              output: ""

      # Step 2: Poll until export is ready
      - request:
          url: "{state.base_url}/exports/{state.export_id}/status"

        pagination:
          next_state:
            poll_count: "{coalesce(state.poll_count, 0) + 1}"
          stop_condition: >
            jmespath(response.json, "status") == "completed" ||
            jmespath(response.json, "status") == "failed" ||
            state.poll_count >= 60

        response:
          processors:
            - expression: 'jmespath(response.json, "status")'
              output: "state.export_status"
              aggregation: last
            - if: 'jmespath(response.json, "status") == "processing"'
              expression: log("Export still processing, poll " + string(state.poll_count))
              output: ""

      # Step 3: Get download URL (only if completed)
      - if: state.export_status == "completed"
        request:
          url: "{state.base_url}/exports/{state.export_id}"
        response:
          processors:
            - expression: 'jmespath(response.json, "download_url")'
              output: "state.download_url"
              aggregation: last

    # Main request downloads the export file
    request:
      url: "{state.download_url}"

    response:
      format: csv
      records:
        jmespath: "[*]"

    teardown:
      # Clean up the export job
      - request:
          url: "{state.base_url}/exports/{state.export_id}"
          method: DELETE
        response:
          processors:
            - expression: log("Deleted export job: " + state.export_id)
              output: ""

Sequence vs. Main Request

Feature	Sequences (Setup/Teardown)	Main Request
When Executes	Before/after main requests	During endpoint execution
Supports Iteration	No	Yes (via iterate)
Supports Pagination	Yes (per call)	Yes
State Scope	Shared with endpoint	Per-iteration (if iterating)
Output Records	No (state only)	Yes (to destination)
Conditional Execution	Yes (via if per call)	Yes (via iterate.if)

Best Practices for Sequences

1. Use Logging for Visibility

setup:
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: "response.json.config"
          output: "state.config"
          aggregation: last
        # Log what we loaded
        - expression: log("Loaded config: " + string(state.config))
          output: ""

2. Handle Errors Gracefully

setup:
  - request:
      url: "{state.base_url}/optional-config"
    response:
      rules:
        # Don't fail if optional config is missing
        - action: continue
          condition: "response.status == 404"
          message: "Optional config not found, using defaults"

3. Keep Sequences Focused

# Good: Each step has a clear purpose
setup:
  - request: # Get auth token
  - request: # Get user info
  - request: # Get feature flags

# Avoid: Mixing unrelated concerns
setup:
  - request: # Get auth token
  - request: # Process data (should be main request)
  - request: # Generate report (should be teardown)

4. Use Conditional Steps Wisely

setup:
  # Always get config
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: 'jmespath(response.json, "features")'
          output: "state.features"
          aggregation: last

  # Only load beta features if enabled
  - if: contains(state.features, "beta")
    request:
      url: "{state.base_url}/beta/features"

📝 Note: Setup/Teardown sequences use the same structure as authentication sequences. State changes persist to the main endpoint, making them perfect for initialization and cleanup tasks.

💡 Tip: Use log() function liberally in sequences during development to understand the execution flow and debug issues.