# Testing & Debugging

Testing your Sling API specifications is a crucial step before deploying them in production. This document covers tools and techniques to verify your specs work correctly, debug issues, and optimize performance.

## Content Overview

* [Testing Workflow](#testing-workflow)
* [Creating a Connection](#creating-a-connection-with-an-api-spec)
* [Using the Test Command](#using-the-test-command)
* [Debugging Tools](#debugging-tools)
* [Common Issues](#common-issues)
* [Testing Best Practices](#testing-best-practices)

## Testing Workflow

The general workflow for developing and testing API specs is:

{% @mermaid/diagram content="graph TD
A\[Create Initial API Spec] --> B\[Create & Test Connection]
B -->|Success| C\[Discover Available Endpoints]
B -->|Fails| D\[Fix Authentication]
D --> B
C --> E\[Test Individual Endpoints]
E -->|Works| F\[Add More Endpoints]
E -->|Issues| G\[Debug with --trace]
G --> H\[Fix Issues]
H --> E
F --> I\[Create Replication YAML]
I --> J\[Run Full Replication Test]

```
style A fill:#4a9eff,stroke:#ffffff,stroke-width:2px,color:#ffffff
style B fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style C fill:#ff8c42,stroke:#ffffff,stroke-width:2px,color:#ffffff
style E fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style J fill:#7cb342,stroke:#ffffff,stroke-width:2px,color:#ffffff
style G fill:#ef5350,stroke:#ffffff,stroke-width:2px,color:#ffffff
style I fill:#26c6da,stroke:#ffffff,stroke-width:2px,color:#ffffff" %}
```

## Creating a Connection with an API Spec

The first step is create and save your Spec YAML file somewhere accessible. This can be your local drive, or any other be any [storage connection](/connections/file-connections.md) you have setup, such a S3/GCP bucket, or FTP/SFTP. Furthermore, sling supports reading API Specs from a HTTP URL (such as Github URLs).

Once you have an API Spec file to use, you can then create a connection in your [env.yaml](/sling-cli/environment.md#sling-env-file-envyaml) file, like this:

```yaml
connections:
  stripe:
    type: api
    # fetch from github repo
    spec: https://github.com/my-org/my-repo/blob/main/api/specs/stripe.yaml
    secrets:
      api_key: xxxxxxxxxxxxxxx

  my_api:
    type: api
    spec: file:///path/to/my_api.yaml  # read from local file
    secrets:
      account_id: xxxxxxxxxxx
      token: xxxxxxxxxxxxxxx

  my_other_api:
    type: api
    spec: aws_s3/path/to/my_other_api.yaml  # fetches from your s3 connection
```

You can also use an environment variable (YAML or JSON format):

```shell
# Windows Powershell
$env:DBT_CLOUD_API='{ type: api, spec: https://github.com/slingdata-io/sling-cli/blob/main/api/specs/dbt_cloud.yaml, secrets: { account_id: xxxxx, api_token: xxxxxxx } }'

# Linux or Mac
export DBT_CLOUD_API='{ type: api, spec: https://github.com/slingdata-io/sling-cli/blob/main/api/specs/dbt_cloud.yaml, secrets: { account_id: xxxxx, api_token: xxxxxxx } }'
```

The connection should show up like the others:

```shell
# List all your connections
$ sling conns list
+----------------+-----------------+----------------+
| CONN NAME      | CONN TYPE       | SOURCE         |
+----------------+-----------------+----------------+
| POSTGRES       | DB - PostgreSQL | sling env yaml |
| SNOWFLAKE      | DB - Snowflake  | sling env yaml |
| STRIPE         | API - Spec      | sling env yaml |
| MY_API         | API - Spec      | sling env yaml |
| MY_OTHER_API   | API - Spec      | sling env yaml |
| DBT_CLOUD_API  | API - Spec      | env variable   |
+----------------+-----------------+----------------+
```

## Using the Test Command

Sling provides the `conns test` command to verify your API connection and test individual endpoints.

### Testing the Connection

```bash
# Test that your connection is properly configured
sling conns test API_CONNECTION_NAME

# Example testing a Stripe connection
sling conns test STRIPE
```

### Testing Specific Endpoints

```bash
# Test specific endpoints
sling conns test API_CONNECTION_NAME --endpoints endpoint1,endpoint2

# Example testing specific Stripe endpoints
sling conns test STRIPE --endpoints customer,charge
```

### Discovering Available Endpoints

To see the available endpoints in your API spec:

```bash
# List all endpoints in your API spec
sling conns discover API_CONNECTION_NAME

# Example for Shopify
sling conns discover STRIPE
```

This command is particularly useful for verifying that all your endpoints are correctly defined and visible to Sling.

## Debugging Tools

Sling offers two levels of debug output to help diagnose issues with your API specs.

### Debug Flag

The `--debug` flag provides basic information about request flows, pagination, and data processing:

```bash
# Basic debugging information
sling conns test STRIPE --endpoints customer --debug
```

Debug output includes:

* API requests being made
* Response status codes
* Record counts
* Pagination details
* State variable changes

### Trace Flag

For more detailed debugging, use the `--trace` flag:

```bash
# Detailed trace information
sling conns test STRIPE --endpoints customer --trace
```

The trace output includes everything from debug plus:

* Full request headers and parameters
* Response headers
* JSON response bodies (truncated for large responses)
* Detailed expression evaluation
* Auth token refresh events
* Queue operations

> ⚠️ **Warning:** The `--trace` flag may expose sensitive information in logs, such as authorization tokens. Use carefully and don't share unredacted logs.

### Examining Request and Response Flow

With trace enabled, you can see the complete flow of HTTP requests and responses. This is invaluable for debugging pagination, authentication, or data extraction issues:

```
2025-04-27 07:45:39 DBG iteration 1 initial state: {"base_url":"https://api.stripe.com/v1","created_gte":"1743158739","limit":100,"starting_after":null}
2025-04-27 07:45:39 DBG r.0001.jlw GET @ https://api.stripe.com/v1/customers?limit=100
2025-04-27 07:45:39 TRC r.0001.jlw request: {"method":"GET","url":"https://api.stripe.com/v1/customers?limit=100","headers":{"accept":"application/json","authorization":"Bearer xxxxxxxx","content-type":"application/json","stripe-version":"2023-10-16"},"payload":null,"attempts":1}
2025-04-27 07:45:40 DBG r.0001.jlw   response code=200 content-type=application/json duration=839ms size=36kB
2025-04-27 07:45:40 TRC r.0001.jlw response: {"headers":{"access-control-allow-credentials":"true","access-control-allow-methods":"GET, HEAD, PUT, PATCH, POST, DELETE","access-control-allow-origin":"*","access-control-expose-headers":"Request-Id, Stripe-Manage-Version, Stripe-Should-Retry, X-Stripe-External-Auth-Required, X-Stripe-Privileged-Session-Required","access-control-max-age":"300","cache-control":"no-cache, no-store","content-length":"35925","content-security-policy":"base-uri 'none'; default-src 'none'; form-action 'none'; frame-ancestors 'none'; img-src 'self'; script-src 'self' 'report-sample'; style-src 'self'; worker-src 'none'; upgrade-insecure-requests},"records":39,"size":35925,"status":200}
2025-04-27 07:45:40 DBG iteration 1 ending state: {"base_url":"https://api.stripe.com/v1","created_gte":"1743158739","last_id":"cus_xxxxxxxxxx","limit":100,"starting_after":null}
```

## Common Issues

### Authentication Problems

If you're getting `401 Unauthorized` or `403 Forbidden` responses:

```bash
# Check your authentication configuration
sling conns test API_NAME --debug
```

Common fixes:

* Verify the secrets such as API keys or tokens in your environment file
* Check for correct authentication type (bearer, basic, oauth2)
* Ensure required scopes are included for OAuth2

### Pagination Issues

If your endpoint doesn't retrieve all expected data:

```bash
# Trace pagination behavior
sling conns test API_NAME --endpoints ENDPOINT_NAME --trace
```

Look for:

* `stop_condition` evaluation results
* `next_state` changes between requests
* Response headers for Link-based pagination
* `has_more` flags in response bodies

### JMESPath Extraction Problems

If your records aren't being properly extracted:

```bash
# Trace with attention to record extraction
sling conns test API_NAME --endpoints ENDPOINT_NAME --trace
```

Look for:

* Complete response JSON to verify the correct path
* JMESPath extraction results
* Record counts in your output

## Testing Best Practices

### 1. Test Incrementally

When building complex API specs:

1. Start by testing basic authentication
2. Test a simple endpoint without pagination
3. Add and test pagination
4. Test one endpoint that uses iteration
5. Test queue-based workflows with multiple endpoints
6. Finally, test complex transformations and processors

### 2. Limit Data During Testing

Use these techniques to limit data volume during testing:

Set the environment variable `SLING_TEST_ENDPOINT_LIMIT`:

```bash
# Windows Powershell
$env:SLING_TEST_ENDPOINT_LIMIT='30'

# Linux or Mac
export SLING_TEST_ENDPOINT_LIMIT=30
```

```yaml
# In your endpoint definition
response:
  records:
    # Limit total records processed during testing
    limit: 10
```

Or use date filtering if the API supports it:

```yaml
# Example from Shopify endpoint
request:
  parameters:
    # Limit to recent data only
    updated_at_min: '{date_format(date_add(now(), -2, "day"), "%Y-%m-%dT%H:%M:%S%z")}'
```

### 3. Use Replication Files for Full Testing

Create a replication YAML file to test the flow from API to your database:

```yaml
# Example stripe_test.yaml
source: stripe_sling
target: postgres

defaults:
  mode: full-refresh  # Use for testing instead of incremental
  object: apis.stripe_test_{stream_name}  # Use test schema

streams:
  # Test specific streams
  customer:
  charge:

env:
  SLING_LOADED_AT_COLUMN: timestamp
```

Run with:

```bash
sling run -r stripe_test.yaml --debug
```

### 4. Create Environment Variables for Testing

For testing different scenarios:

```bash
# Test with different time ranges
CREATED_GTE=1659312000 sling conns test STRIPE --endpoints charge --debug

# Test with specific user
GITHUB_USERNAME=test-user sling conns test GITHUB --endpoints repos --debug
```

## Real-World Examples

### Testing Stripe Endpoints

This example tests the Stripe customer and balance transaction endpoints:

```bash
# Test specific Stripe endpoints
sling conns test STRIPE --endpoints customer,customer_balance_transaction --debug
```

The corresponding replication file would look like:

```yaml
# r.61.stripe.yaml
source: stripe_sling
target: postgres

defaults:
  mode: incremental
  object: apis.{source_name}_{stream_name}

  source_options:
    flatten: 1 # flatten records 1 level only

streams:
  '*':

env:
  SLING_STATE: postgres/sling_state.stripe # one state table per replication
  SLING_LOADED_AT_COLUMN: timestamp
```

> 💡 **Tip:** When developing a complex API spec, maintain a test script with your commonly used test commands for quick iteration.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.slingdata.io/concepts/api-specs/testing-debug.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.