# Requests & Iteration

This page details how to configure HTTP requests and use the iteration feature for looping within Sling API specifications.

## Request Configuration

Each endpoint defines its HTTP request details under the `request` key. These settings merge with and override the `defaults.request` configuration.

### Request Properties

| Property      | Required | Description                                           | Example                                        |
| ------------- | -------- | ----------------------------------------------------- | ---------------------------------------------- |
| `url`         | Yes      | Path (relative to `defaults.request.url`) or full URL | `"users"` or `"https://api.example.com/users"` |
| `method`      | No       | HTTP method (default: `"GET"`)                        | `"POST"`, `"PUT"`, `"PATCH"`, `"DELETE"`       |
| `headers`     | No       | HTTP headers to send                                  | `{"Content-Type": "application/json"}`         |
| `parameters`  | No       | Query parameters (or form fields for POST)            | `{"page": 1, "limit": 100}`                    |
| `payload`     | No       | Request body for POST/PUT/PATCH                       | `{"name": "New User"}`                         |
| `timeout`     | No       | Request timeout in seconds (default: 30)              | `60`                                           |
| `rate`        | No       | Max requests per second (default: 2)                  | `5`                                            |
| `concurrency` | No       | Max concurrent requests (default: 5)                  | `10`                                           |

### Example Request Configuration

```yaml
request:
  state:
    base_url: https://api.example.com/v1
    user_id: 123
  
  url: '{state.base_url}/users/{state.user_id}'
  
  # HTTP Method
  method: "POST"
  
  # Headers (merged with defaults.request.headers)
  headers:
    Content-Type: "application/json"
    Authorization: "Bearer {auth.token}"
    X-Request-ID: "{uuid()}"
    
  # Query Parameters (or form parameters for POST with application/x-www-form-urlencoded)
  parameters:
    active: true
    department: "{state.department}"
    
  # Request Body (for POST/PUT/PATCH methods)
  payload:
    user:
      name: "New User"
      email: "{state.user_email}"
      
  # Request timeout (in seconds)
  timeout: 60
  
  # Rate limiting (max requests per second)
  rate: 5
  
  # Concurrency (max in-flight requests)
  concurrency: 3
```

> 📝 **Note:** When `method` is `GET` or `DELETE`, the `parameters` are added as URL query parameters. When `method` is `POST`, `PUT`, or `PATCH` and `Content-Type` is `application/x-www-form-urlencoded`, the `parameters` are sent as form fields.

> 💡 **Tip:** Use `{...}` expressions to make request values dynamic based on state variables, authentication details, or environment variables.

## Iteration (Looping Requests)

The `iterate` section allows an endpoint to make multiple requests based on a list of items (e.g., IDs from a queue, date ranges).

### How Iteration Works

{% @mermaid/diagram content="graph TD
A\[Start Endpoint] --> B{Has iterate section?}
B -->|Yes| C\[Evaluate 'over' expression]
B -->|No| D\[Run single request with pagination]

```
C --> E[Create item list]
E --> F[For each item in parallel]
F --> G[Create iteration with own state]
G --> H[Set 'into' variable to current item]
H --> I[Run requests with pagination]

D --> J[Process response]
I --> J

style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style F fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#ffd54f,stroke:#ffffff,stroke-width:2px,color:#000000
style J fill:#ab47bc,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

### Iteration Properties

| Property      | Required | Description                                     | Example                             |
| ------------- | -------- | ----------------------------------------------- | ----------------------------------- |
| `over`        | Yes      | Expression to get items to iterate over         | `"queue.user_ids"` or `"[1, 2, 3]"` |
| `into`        | Yes      | State variable to store current item            | `"state.current_id"`                |
| `concurrency` | No       | Max iterations to run in parallel (default: 10) | `5`                                 |
| `if`          | No       | Condition to evaluate before starting iteration | `"state.should_process == true"`    |

### Example: Basic Iteration

```yaml
iterate:
  # Get items from a queue filled by another endpoint
  over: "queue.product_ids"
  
  # Store current item in state variable
  into: "state.current_product_id"
  
  # Run up to 5 iterations concurrently
  concurrency: 5
```

> ⚠️ **Important:** Each iteration gets its own state. Changes to state variables in one iteration don't affect other iterations.

### Example: Date Range Iteration

This pattern is great for splitting large data extractions into daily chunks:

```yaml
endpoints:
  daily_reports:
    state:
      # Configure date range (last 7 days by default)
      start_date: '{coalesce(env.START_DATE, date_format(date_add(now(), -7, "day"), "%Y-%m-%d"))}'
      end_date: '{coalesce(env.END_DATE, date_format(now(), "%Y-%m-%d"))}'
    
    iterate:
      # Generate an array of dates between start_date and end_date, one day at a time
      over: >
        range(
          date_parse(state.start_date, "%Y-%m-%d"), 
          date_parse(state.end_date, "%Y-%m-%d"),
          "1d"
        )
      # Store the current date in state.current_day
      into: state.current_day
      concurrency: 10
    
    request:
      url: '{state.base_url}/reports'
      parameters:
        # Format the date for the API
        date: '{date_format(state.current_day, "%Y-%m-%d")}'
```

### Example: Batch Processing with `chunk()`

When an API accepts multiple IDs in a single request, use `chunk()` to process them in batches:

```yaml
endpoints:
  lookup_variants:
    iterate:
      # Split queue into batches of 50 IDs each
      over: "chunk(queue.variant_ids, 50)"
      # state.variant_id_batch will be an array of up to 50 IDs
      into: "state.variant_id_batch"
      concurrency: 5

    request:
      url: '{state.base_url}/variants/lookup'
      parameters:
        # Join IDs into comma-separated string
        ids: '{join(state.variant_id_batch, ",")}'
```

### Using Context Variables for Backfill Ranges

**Context variables** are runtime values passed from the replication configuration to the API spec. They're particularly useful for supporting both backfill and incremental modes with the same endpoint.

**Key context variables:**

* `context.range_start` - Start of backfill range (from `source_options.range`)
* `context.range_end` - End of backfill range (from `source_options.range`)

**Example: Date Range with Context Support**

```yaml
endpoints:
  events:
    # Persist last processed date for incremental runs
    sync: [last_date]

    iterate:
      # Backfill mode: Use context.range_start/range_end from config
      # Incremental mode: Use sync.last_date
      over: >
        range(
          coalesce(context.range_start, sync.last_date, date_format(date_add(now(), -7, "day"), "%Y-%m-%d")),
          coalesce(context.range_end, date_format(now(), "%Y-%m-%d")),
          "1d"
        )
      into: "state.current_date"

    request:
      url: "{state.base_url}/events"
      parameters:
        date: "{state.current_date}"

    response:
      records:
        jmespath: "events[]"
        primary_key: ["event_id"]

      processors:
        # Track last processed date
        - expression: "state.current_date"
          output: "state.last_date"
          aggregation: "maximum"
```

**Replication Config for Backfill:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    source_options:
      # Backfill specific date range
      range: '2024-01-01,2024-01-31'
```

**Replication Config for Incremental:**

```yaml
source: MY_API
target: MY_TARGET_DB

streams:
  events:
    object: analytics.events
    # No range - uses sync.last_date for incremental
```

This pattern allows a single endpoint to handle both historical backfills and ongoing incremental updates seamlessly. See [Context Variables](https://docs.slingdata.io/concepts/structure#context-variables) for complete details.

## Request Flow Control

You can control the request flow using these features:

| Feature       | Location  | Purpose                   | Example                |
| ------------- | --------- | ------------------------- | ---------------------- |
| `rate`        | `request` | Limit requests per second | `rate: 5`              |
| `concurrency` | `request` | Limit parallel requests   | `concurrency: 3`       |
| `concurrency` | `iterate` | Limit parallel iterations | `concurrency: 10`      |
| `if`          | `iterate` | Conditional iteration     | `if: "state.has_data"` |
| `timeout`     | `request` | Set request timeout       | `timeout: 30`          |

> 💡 **Tip:** Balance performance and API limits:
>
> * Too low `concurrency`: slower extraction but gentler on the API
> * Too high `concurrency`: faster extraction but may trigger rate limits
> * `iterate.concurrency` controls parallelism across different IDs/items
> * `request.concurrency` controls parallelism across paginated requests

## Sequences: Setup and Teardown

Endpoints can define optional `setup` and `teardown` sequences that run before and after the main requests. These are multi-step workflows perfect for initialization, cleanup, or complex processes.

### What are Sequences?

A sequence is an ordered array of API calls. Each call in a sequence can:

* Make HTTP requests
* Process responses
* Update state variables
* Use pagination
* Execute conditionally

Sequences are used in three places:

1. **Authentication** - Custom authentication workflows
2. **Setup** - Pre-execution initialization
3. **Teardown** - Post-execution cleanup

### Sequence Call Structure

Each call in a sequence has the same structure:

```yaml
- if: <conditional expression>      # Optional: Execute only if true
  request: <request configuration>   # Same as endpoint request
  pagination: <pagination config>    # Optional: Paginate this call
  response: <response configuration> # Process response
```

### Setup Sequences

The `setup` sequence runs **once before** the endpoint's main requests/iterations begin. Perfect for:

* Fetching configuration data
* Initializing session data
* Validating prerequisites
* Loading dynamic parameters

#### Basic Setup Example

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
      method: GET
    response:
      processors:
        - expression: 'jmespath(response.json, "config.api_version")'
          output: "state.api_version"
          aggregation: last
        - expression: 'jmespath(response.json, "config.base_url")'
          output: "state.base_url"
          aggregation: last
        - expression: 'jmespath(response.json, "config.rate_limit")'
          output: "state.rate_limit"
          aggregation: last
```

#### Multi-Step Setup Example

```yaml
setup:
  # Step 1: Get session token
  - request:
      url: "{state.base_url}/session"
      method: POST
      payload:
        app_id: "{secrets.app_id}"
    response:
      processors:
        - expression: "response.json.session_token"
          output: "state.session_token"
          aggregation: last

  # Step 2: Use session token to get user info
  - request:
      url: "{state.base_url}/user/me"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: 'jmespath(response.json, "user.id")'
          output: "state.user_id"
          aggregation: last
        - expression: 'jmespath(response.json, "user.permissions")'
          output: "state.user_permissions"
          aggregation: last

  # Step 3: Conditionally load admin data
  - if: contains(state.user_permissions, "admin")
    request:
      url: "{state.base_url}/admin/settings"
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.admin_config"
          output: "state.admin_config"
          aggregation: last
```

### Teardown Sequences

The `teardown` sequence runs **once after** all main requests complete. Perfect for:

* Closing sessions
* Cleaning up temporary resources
* Logging completion status
* Archiving processed data

#### Basic Teardown Example

```yaml
teardown:
  - request:
      url: "{state.base_url}/session/close"
      method: POST
      headers:
        X-Session-Token: "{state.session_token}"
    response:
      processors:
        - expression: "response.json.status"
          output: "state.session_close_status"
          aggregation: last
```

#### Conditional Teardown Example

```yaml
teardown:
  # Only cleanup if we created temporary data
  - if: state.created_temp_data == true
    request:
      url: "{state.base_url}/cleanup"
      method: POST
      payload:
        session_id: "{state.session_id}"
        temp_ids: "{state.temp_resource_ids}"
    response:
      processors:
        - expression: 'jmespath(response.json, "cleanup.status")'
          output: "state.cleanup_status"
          aggregation: last

  # Always log final statistics
  - request:
      url: "{state.base_url}/analytics/log"
      method: POST
      payload:
        endpoint_name: "{env.ENDPOINT_NAME}"
        records_processed: "{state.total_records}"
        duration_seconds: "{state.duration}"
```

### Conditional Execution with `if`

Each call in a sequence can execute conditionally using the `if` field:

```yaml
setup:
  # Always runs
  - request:
      url: "{state.base_url}/status"
    response:
      processors:
        - expression: "response.json.api_status"
          output: "state.api_status"
          aggregation: last

  # Only runs if API is in maintenance mode
  - if: state.api_status == "maintenance"
    request:
      url: "{state.base_url}/maintenance/info"
    response:
      processors:
        - expression: "response.json.maintenance_until"
          output: "state.maintenance_until"
          aggregation: last
        - expression: log("API in maintenance until: " + state.maintenance_until)
          output: ""
```

**Common `if` Patterns:**

```yaml
# Check state variable existence
if: '!is_null(state.session_token)'

# Check for specific value
if: state.environment == "production"

# Check array membership
if: contains(state.enabled_features, "advanced_mode")

# Multiple conditions
if: state.user_type == "admin" && !is_null(state.admin_token)

# Check environment variable
if: env.ENABLE_FEATURE == "true"
```

### Pagination in Sequences

Sequence calls support pagination, useful for multi-page setup data:

```yaml
setup:
  # Load all available categories (paginated)
  - request:
      url: "{state.base_url}/categories"
      parameters:
        limit: 100

    pagination:
      next_state:
        offset: "{state.offset + 100}"
      stop_condition: "length(response.records) < 100"

    response:
      records:
        jmespath: "categories[]"
      processors:
        # Collect all category IDs
        - expression: "record.id"
          output: "queue.category_ids"
```

### State Management in Sequences

**Important behaviors:**

1. **State Isolation**: Setup/teardown sequences have their own state copy
2. **State Merging**: Changes to state persist back to the main endpoint
3. **Headers Inherited**: Request headers from the main endpoint are copied
4. **No Iteration State**: Sequences don't have iteration-specific state

```yaml
endpoints:
  my_endpoint:
    state:
      base_url: "https://api.example.com"
      initial_value: 100

    setup:
      - request:
          url: "{state.base_url}/init"
        response:
          processors:
            # This updates the endpoint's state
            - expression: "response.json.config_value"
              output: "state.config_value"
              aggregation: last

    request:
      # Can now use state.config_value from setup
      url: "{state.base_url}/data"
      parameters:
        config: "{state.config_value}"
```

### Complete Sequence Example

Here's a real-world example showing setup, main execution, and teardown:

```yaml
endpoints:
  export_data:
    state:
      base_url: "https://api.example.com/v2"

    setup:
      # Step 1: Request export job creation
      - request:
          url: "{state.base_url}/exports"
          method: POST
          payload:
            format: "csv"
            filters:
              date_from: "{state.start_date}"
              date_to: "{state.end_date}"
        response:
          processors:
            - expression: "response.json.export_id"
              output: "state.export_id"
              aggregation: last
            - expression: log("Created export job: " + response.json.export_id)
              output: ""

      # Step 2: Poll until export is ready
      - request:
          url: "{state.base_url}/exports/{state.export_id}/status"

        pagination:
          next_state:
            poll_count: "{coalesce(state.poll_count, 0) + 1}"
          stop_condition: >
            jmespath(response.json, "status") == "completed" ||
            jmespath(response.json, "status") == "failed" ||
            state.poll_count >= 60

        response:
          processors:
            - expression: 'jmespath(response.json, "status")'
              output: "state.export_status"
              aggregation: last
            - if: 'jmespath(response.json, "status") == "processing"'
              expression: log("Export still processing, poll " + string(state.poll_count))
              output: ""

      # Step 3: Get download URL (only if completed)
      - if: state.export_status == "completed"
        request:
          url: "{state.base_url}/exports/{state.export_id}"
        response:
          processors:
            - expression: 'jmespath(response.json, "download_url")'
              output: "state.download_url"
              aggregation: last

    # Main request downloads the export file
    request:
      url: "{state.download_url}"

    response:
      format: csv
      records:
        jmespath: "[*]"

    teardown:
      # Clean up the export job
      - request:
          url: "{state.base_url}/exports/{state.export_id}"
          method: DELETE
        response:
          processors:
            - expression: log("Deleted export job: " + state.export_id)
              output: ""
```

### Sequence vs. Main Request

| Feature                   | Sequences (Setup/Teardown) | Main Request                 |
| ------------------------- | -------------------------- | ---------------------------- |
| **When Executes**         | Before/after main requests | During endpoint execution    |
| **Supports Iteration**    | No                         | Yes (via `iterate`)          |
| **Supports Pagination**   | Yes (per call)             | Yes                          |
| **State Scope**           | Shared with endpoint       | Per-iteration (if iterating) |
| **Output Records**        | No (state only)            | Yes (to destination)         |
| **Conditional Execution** | Yes (via `if` per call)    | Yes (via `iterate.if`)       |

### Best Practices for Sequences

#### 1. Use Logging for Visibility

```yaml
setup:
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: "response.json.config"
          output: "state.config"
          aggregation: last
        # Log what we loaded
        - expression: log("Loaded config: " + string(state.config))
          output: ""
```

#### 2. Handle Errors Gracefully

```yaml
setup:
  - request:
      url: "{state.base_url}/optional-config"
    response:
      rules:
        # Don't fail if optional config is missing
        - action: continue
          condition: "response.status == 404"
          message: "Optional config not found, using defaults"
```

#### 3. Keep Sequences Focused

```yaml
# Good: Each step has a clear purpose
setup:
  - request: # Get auth token
  - request: # Get user info
  - request: # Get feature flags

# Avoid: Mixing unrelated concerns
setup:
  - request: # Get auth token
  - request: # Process data (should be main request)
  - request: # Generate report (should be teardown)
```

#### 4. Use Conditional Steps Wisely

```yaml
setup:
  # Always get config
  - request:
      url: "{state.base_url}/config"
    response:
      processors:
        - expression: 'jmespath(response.json, "features")'
          output: "state.features"
          aggregation: last

  # Only load beta features if enabled
  - if: contains(state.features, "beta")
    request:
      url: "{state.base_url}/beta/features"
```

> 📝 **Note:** Setup/Teardown sequences use the same structure as authentication sequences. State changes persist to the main endpoint, making them perfect for initialization and cleanup tasks.

> 💡 **Tip:** Use `log()` function liberally in sequences during development to understand the execution flow and debug issues.