# Requests & Iteration

This page details how to configure HTTP requests and use the iteration feature for looping within Sling API specifications.

## Request Configuration

Each endpoint defines its HTTP request details under the `request` key. These settings merge with and override the `defaults.request` configuration.

### Request Properties

| Property      | Required | Description                                           | Example                                        |
| ------------- | -------- | ----------------------------------------------------- | ---------------------------------------------- |
| `url`         | Yes      | Path (relative to `defaults.request.url`) or full URL | `"users"` or `"https://api.example.com/users"` |
| `method`      | No       | HTTP method (default: `"GET"`)                        | `"POST"`, `"PUT"`, `"PATCH"`, `"DELETE"`       |
| `headers`     | No       | HTTP headers to send                                  | `{"Content-Type": "application/json"}`         |
| `parameters`  | No       | Query parameters (or form fields for POST)            | `{"page": 1, "limit": 100}`                    |
| `payload`     | No       | Request body for POST/PUT/PATCH                       | `{"name": "New User"}`                         |
| `timeout`     | No       | Request timeout in seconds (default: 30)              | `60`                                           |
| `rate`        | No       | Max requests per second (default: 2)                  | `5`                                            |
| `concurrency` | No       | Max concurrent requests (default: 5)                  | `10`                                           |

### Example Request Configuration

```yaml
request:
  state:
    base_url: https://api.example.com/v1
    user_id: 123
  
  url: '{state.base_url}/users/{state.user_id}'
  
  # HTTP Method
  method: "POST"
  
  # Headers (merged with defaults.request.headers)
  headers:
    Content-Type: "application/json"
    Authorization: "Bearer {auth.token}"
    X-Request-ID: "{uuid()}"
    
  # Query Parameters (or form parameters for POST with application/x-www-form-urlencoded)
  parameters:
    active: true
    department: "{state.department}"
    
  # Request Body (for POST/PUT/PATCH methods)
  payload:
    user:
      name: "New User"
      email: "{state.user_email}"
      
  # Request timeout (in seconds)
  timeout: 60
  
  # Rate limiting (max requests per second)
  rate: 5
  
  # Concurrency (max in-flight requests)
  concurrency: 3
```

> 📝 **Note:** When `method` is `GET` or `DELETE`, the `parameters` are added as URL query parameters. When `method` is `POST`, `PUT`, or `PATCH` and `Content-Type` is `application/x-www-form-urlencoded`, the `parameters` are sent as form fields.

> 💡 **Tip:** Use `{...}` expressions to make request values dynamic based on state variables, authentication details, or environment variables.

## Iteration (Looping Requests)

The `iterate` section allows an endpoint to make multiple requests based on a list of items (e.g., IDs from a queue, date ranges).

### How Iteration Works

{% @mermaid/diagram content="graph TD
A\[Start Endpoint] --> B{Has iterate section?}
B -->|Yes| C\[Evaluate 'over' expression]
B -->|No| D\[Run single request with pagination]

```
C --> E[Create item list]
E --> F[For each item in parallel]
F --> G[Create iteration with own state]
G --> H[Set 'into' variable to current item]
H --> I[Run requests with pagination]

D --> J[Process response]
I --> J

style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style F fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style J fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Iteration Properties

| Property      | Required | Description                                     | Example                             |
| ------------- | -------- | ----------------------------------------------- | ----------------------------------- |
| `over`        | Yes      | Expression to get items to iterate over         | `"queue.user_ids"` or `"[1, 2, 3]"` |
| `into`        | Yes      | State variable to store current item            | `"state.current_id"`                |
| `concurrency` | No       | Max iterations to run in parallel (default: 10) | `5`                                 |
| `if`          | No       | Condition to evaluate before starting iteration | `"state.should_process == true"`    |

### Example: Basic Iteration

```yaml
iterate:
  # Get items from a queue filled by another endpoint
  over: "queue.product_ids"
  
  # Store current item in state variable
  into: "state.current_product_id"
  
  # Run up to 5 iterations concurrently
  concurrency: 5
```

> ⚠️ **Important:** Each iteration gets its own state. Changes to state variables in one iteration don't affect other iterations.

### Example: Date Range Iteration

This pattern is great for splitting large data extractions into daily chunks:

```yaml
endpoints:
  daily_reports:
    state:
      # Configure date range (last 7 days by default)
      start_date: '{coalesce(env.START_DATE, date_format(date_add(now(), -7, "day"), "%Y-%m-%d"))}'
      end_date: '{coalesce(env.END_DATE, date_format(now(), "%Y-%m-%d"))}'
    
    iterate:
      # Generate an array of dates between start_date and end_date, one day at a time
      over: >
        range(
          date_parse(state.start_date, "%Y-%m-%d"), 
          date_parse(state.end_date, "%Y-%m-%d"),
          "1d"
        )
      # Store the current date in state.current_day
      into: state.current_day
      concurrency: 10
    
    request:
      url: '{state.base_url}/reports'
      parameters:
        # Format the date for the API
        date: '{date_format(state.current_day, "%Y-%m-%d")}'
```

### Example: Batch Processing with `chunk()`

When an API accepts multiple IDs in a single request, use `chunk()` to process them in batches:

```yaml
endpoints:
  lookup_variants:
    iterate:
      # Split queue into batches of 50 IDs each
      over: "chunk(queue.variant_ids, 50)"
      # state.variant_id_batch will be an array of up to 50 IDs
      into: "state.variant_id_batch"
      concurrency: 5

    request:
      url: '{state.base_url}/variants/lookup'
      parameters:
        # Join IDs into comma-separated string
        ids: '{join(state.variant_id_batch, ",")}'
```

### Using Context Variables for Backfill Ranges

**Context variables** are runtime values passed from the replication configuration to the API spec. They're particularly useful for supporting both backfill and incremental modes with the same endpoint.

**Key context variables:**

* `context.range_start` - Start of backfill range (from `source_options.range`)
* `context.range_end` - End of backfill range (from `source_options.range`)

**Example: Date Range with Context Support**

```yaml
endpoints:
  events:
    # Persist last processed date for incremental runs
    sync: [last_date]

    iterate:
      # Backfill mode: Use context.range_start/range_end from config
      # Incremental mode: Use sync.last_date
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events"
      parameters:
        date: "{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]

      processors:
        # Track last processed date
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"
```

**Replication Config for Backfill:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    source_options:
      # Backfill specific date range
      range: '2024-01-01,2024-01-31'
```

**Replication Config for Incremental:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    # No range - uses sync.last_date for incremental
```

This pattern allows a single endpoint to handle both historical backfills and ongoing incremental updates seamlessly. See [Context Variables](/concepts/api-specs/structure.md#context-variables) for complete details.

## Request Flow Control

You can control the request flow using these features:

| Feature       | Location  | Purpose                   | Example                |
| ------------- | --------- | ------------------------- | ---------------------- |
| `rate`        | `request` | Limit requests per second | `rate: 5`              |
| `concurrency` | `request` | Limit parallel requests   | `concurrency: 3`       |
| `concurrency` | `iterate` | Limit parallel iterations | `concurrency: 10`      |
| `if`          | `iterate` | Conditional iteration     | `if: "state.has_data"` |
| `timeout`     | `request` | Set request timeout       | `timeout: 30`          |

> 💡 **Tip:** Balance performance and API limits:
>
> * Too low `concurrency`: slower extraction but gentler on the API
> * Too high `concurrency`: faster extraction but may trigger rate limits
> * `iterate.concurrency` controls parallelism across different IDs/items
> * `request.concurrency` controls parallelism across paginated requests

## Sequences: Setup and Teardown

Endpoints can define optional `setup` and `teardown` sequences that run before and after the main requests. These are multi-step workflows perfect for initialization, cleanup, or complex processes.

### What are Sequences?

A sequence is an ordered array of API calls. Each call in a sequence can:

* Make HTTP requests
* Process responses
* Update state variables
* Use pagination
* Execute conditionally

Sequences are used in three places:

1. **Authentication** - Custom authentication workflows
2. **Setup** - Pre-execution initialization
3. **Teardown** - Post-execution cleanup

### Sequence Call Structure

Each call in a sequence has the same structure:

```yaml
- if: <conditional expression>      # Optional: Execute only if true
  request: <request configuration>   # Same as endpoint request
  pagination: <pagination config>    # Optional: Paginate this call
  response: <response configuration> # Process response
```

### Setup Sequences

The `setup` sequence runs **once before** the endpoint's main requests/iterations begin. Perfect for:

* Fetching configuration data
* Initializing session data
* Validating prerequisites
* Loading dynamic parameters

#### Basic Setup Example

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
      method: GET
    response:
      processors:
        - expression: 'jmespath(response.json, "config.api_version")'
          output: "state.api_version"
          aggregation: last
        - expression: 'jmespath(response.json, "config.base_url")'
          output: "state.base_url"
          aggregation: last
        - expression: 'jmespath(response.json, "config.rate_limit")'
          output: "state.rate_limit"
          aggregation: last
```

#### Multi-Step Setup Example

```yaml
setup:
  # Step 1: Get session token
  - request:
      url: "{state.base_url}/session"
      method: POST
      payload:
        app_id: "{secrets.app_id}"
    response:
      processors:
        - expression: "response.json.session_token"
          output: "state.session_token"
          aggregation: last

  # Step 2: Use session token to get user info
  - request:
      url: "{state.base_url}/user/me"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: 'jmespath(response.json, "user.id")'
          output: "state.user_id"
          aggregation: last
        - expression: 'jmespath(response.json, "user.permissions")'
          output: "state.user_permissions"
          aggregation: last

  # Step 3: Conditionally load admin data
  - if: contains(state.user_permissions, "admin")
    request:
      url: "{state.base_url}/admin/settings"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.admin_config"
          output: "state.admin_config"
          aggregation: last
```

### Teardown Sequences

The `teardown` sequence runs **once after** all main requests complete. Perfect for:

* Closing sessions
* Cleaning up temporary resources
* Logging completion status
* Archiving processed data

#### Basic Teardown Example

```yaml
teardown:
  - request:
      url: "{state.base_url}/session/close"
      method: POST
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.status"
          output: "state.session_close_status"
          aggregation: last
```

#### Conditional Teardown Example

```yaml
teardown:
  # Only cleanup if we created temporary data
  - if: state.created_temp_data == true
    request:
      url: "{state.base_url}/cleanup"
      method: POST
      payload:
        session_id: "{state.session_id}"
        temp_ids: "{state.temp_resource_ids}"
    response:
      processors:
        - expression: 'jmespath(response.json, "cleanup.status")'
          output: "state.cleanup_status"
          aggregation: last

  # Always log final statistics
  - request:
      url: "{state.base_url}/analytics/log"
      method: POST
      payload:
        endpoint_name: "{env.ENDPOINT_NAME}"
        records_processed: "{state.total_records}"
        duration_seconds: "{state.duration}"
```

### Conditional Execution with `if`

Each call in a sequence can execute conditionally using the `if` field:

```yaml
setup:
  # Always runs
  - request:
      url: "{state.base_url}/status"
    response:
      processors:
        - expression: "response.json.api_status"
          output: "state.api_status"
          aggregation: last

  # Only runs if API is in maintenance mode
  - if: state.api_status == "maintenance"
    request:
      url: "{state.base_url}/maintenance/info"
    response:
      processors:
        - expression: "response.json.maintenance_until"
          output: "state.maintenance_until"
          aggregation: last
        - expression: log("API in maintenance until: " + state.maintenance_until)
          output: ""
```

**Common `if` Patterns:**

```yaml
# Check state variable existence
if: '!is_null(state.session_token)'

# Check for specific value
if: state.environment == "production"

# Check array membership
if: contains(state.enabled_features, "advanced_mode")

# Multiple conditions
if: state.user_type == "admin" && !is_null(state.admin_token)

# Check environment variable
if: env.ENABLE_FEATURE == "true"
```

### Pagination in Sequences

Sequence calls support pagination, useful for multi-page setup data:

```yaml
setup:
  # Load all available categories (paginated)
  - request:
      url: "{state.base_url}/categories"
      parameters:
        limit: 100

    pagination:
      next_state:
        offset: "{state.offset + 100}"
      stop_condition: "length(response.records) < 100"

    response:
      records:
        jmespath: "categories[]"
      processors:
        # Collect all category IDs
        - expression: "record.id"
          output: "queue.category_ids"
```

### State Management in Sequences

**Important behaviors:**

1. **State Isolation**: Setup/teardown sequences have their own state copy
2. **State Merging**: Changes to state persist back to the main endpoint
3. **Headers Inherited**: Request headers from the main endpoint are copied
4. **No Iteration State**: Sequences don't have iteration-specific state

```yaml
endpoints:
  my_endpoint:
    state:
      base_url: "https://api.example.com"
      initial_value: 100

    setup:
      - request:
          url: "{state.base_url}/init"
        response:
          processors:
            # This updates the endpoint's state
            - expression: "response.json.config_value"
              output: "state.config_value"
              aggregation: last

    request:
      # Can now use state.config_value from setup
      url: "{state.base_url}/data"
      parameters:
        config: "{state.config_value}"
```

### Complete Sequence Example

Here's a real-world example showing setup, main execution, and teardown:

```yaml
endpoints:
  export_data:
    state:
      base_url: "https://api.example.com/v2"

    setup:
      # Step 1: Request export job creation
      - request:
          url: "{state.base_url}/exports"
          method: POST
          payload:
            format: "csv"
            filters:
              date_from: "{state.start_date}"
              date_to: "{state.end_date}"
        response:
          processors:
            - expression: "response.json.export_id"
              output: "state.export_id"
              aggregation: last
            - expression: log("Created export job: " + response.json.export_id)
              output: ""

      # Step 2: Poll until export is ready
      - request:
          url: "{state.base_url}/exports/{state.export_id}/status"

        pagination:
          next_state:
            poll_count: "{coalesce(state.poll_count, 0) + 1}"
          stop_condition: >
            jmespath(response.json, "status") == "completed" ||
            jmespath(response.json, "status") == "failed" ||
            state.poll_count >= 60

        response:
          processors:
            - expression: 'jmespath(response.json, "status")'
              output: "state.export_status"
              aggregation: last
            - if: 'jmespath(response.json, "status") == "processing"'
              expression: log("Export still processing, poll " + string(state.poll_count))
              output: ""

      # Step 3: Get download URL (only if completed)
      - if: state.export_status == "completed"
        request:
          url: "{state.base_url}/exports/{state.export_id}"
        response:
          processors:
            - expression: 'jmespath(response.json, "download_url")'
              output: "state.download_url"
              aggregation: last

    # Main request downloads the export file
    request:
      url: "{state.download_url}"

    response:
      format: csv
      records:
        jmespath: "[*]"

    teardown:
      # Clean up the export job
      - request:
          url: "{state.base_url}/exports/{state.export_id}"
          method: DELETE
        response:
          processors:
            - expression: log("Deleted export job: " + state.export_id)
              output: ""
```

### Sequence vs. Main Request

| Feature                   | Sequences (Setup/Teardown) | Main Request                 |
| ------------------------- | -------------------------- | ---------------------------- |
| **When Executes**         | Before/after main requests | During endpoint execution    |
| **Supports Iteration**    | No                         | Yes (via `iterate`)          |
| **Supports Pagination**   | Yes (per call)             | Yes                          |
| **State Scope**           | Shared with endpoint       | Per-iteration (if iterating) |
| **Output Records**        | No (state only)            | Yes (to destination)         |
| **Conditional Execution** | Yes (via `if` per call)    | Yes (via `iterate.if`)       |

### Best Practices for Sequences

#### 1. Use Logging for Visibility

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: "response.json.config"
          output: "state.config"
          aggregation: last
        # Log what we loaded
        - expression: log("Loaded config: " + string(state.config))
          output: ""
```

#### 2. Handle Errors Gracefully

```yaml
setup:
  - request:
      url: "{state.base_url}/optional-config"
    response:
      rules:
        # Don't fail if optional config is missing
        - action: continue
          condition: "response.status == 404"
          message: "Optional config not found, using defaults"
```

#### 3. Keep Sequences Focused

```yaml
# Good: Each step has a clear purpose
setup:
  - request: # Get auth token
  - request: # Get user info
  - request: # Get feature flags

# Avoid: Mixing unrelated concerns
setup:
  - request: # Get auth token
  - request: # Process data (should be main request)
  - request: # Generate report (should be teardown)
```

#### 4. Use Conditional Steps Wisely

```yaml
setup:
  # Always get config
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: 'jmespath(response.json, "features")'
          output: "state.features"
          aggregation: last

  # Only load beta features if enabled
  - if: contains(state.features, "beta")
    request:
      url: "{state.base_url}/beta/features"
```

> 📝 **Note:** Setup/Teardown sequences use the same structure as authentication sequences. State changes persist to the main endpoint, making them perfect for initialization and cleanup tasks.

> 💡 **Tip:** Use `log()` function liberally in sequences during development to understand the execution flow and debug issues.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.slingdata.io/concepts/api-specs/request.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.