API Specs

Sling API specs are YAML files that define how to interact with REST APIs. They provide a structured way to specify authentication methods, define endpoints, configure request parameters, handle pagination, process responses, manage state for incremental loads, and more.

CLI Pro Required: APIs require a CLI Pro token or Platform Plan.

Quick Start to build a Spec

Here's a complete, working example of a Sling API spec that fetches user data from a REST API:

name: "Example API"
description: "Simple API to get user data"

defaults:
  state:
    base_url: https://api.example.com/v1

  request:
    headers:
      Accept: "application/json"
      Authorization: "Bearer {secrets.api_token}"  # read from env.yaml secrets section

endpoints:
  users:
    description: "Retrieve list of users"
    docs: https://docs.example.com/users # docs url for endpoint
    disabled: false                      # Set to true to temporarily disable
    
    # State variables control request parameters and track values between runs
    state:
      page: 1          # Start at page 1
      limit: 100       # Fetch 100 records per request
    
    request:
      # Will resolve to https://api.example.com/v1/users
      url: '{state.base_url}/users'
      parameters:
        page: '{state.page}'
        limit: '{state.limit}'
    
    # Control how to fetch next pages
    pagination:
      next_state:
        page: '{state.page + 1}'  # Increment page number for next request
      stop_condition: "length(response.records) < state.limit"  # Stop when page isn't full
    
    # Define how to extract and process response data
    response:
      records:
        jmespath: "data.users[]"  # Extract array of users from response
        primary_key: ["id"]       # Use 'id' field to deduplicate records

To run this spec with Sling:

1. Save your spec somewhere (local disk, S3, SFTP, http). You can access your spec by the location string convention.

2. Create a connection in your env.yaml file, like this:

connections:
  my_api:
    type: api
    spec: file:///path/to/my_api.spec.yaml  # or Github repo file, HTTP URL
    secrets:
      api_key: xxxxxxxxxxxxxxxxxx
  
  my_postgres:
    url: postgres://....

1. Create a replication

source: my_api
target: my_postgres

streams:
   # '*' will read from all endpoints (endpoint name = stream name)
   # or you can simply input the specific endpoint names
  '*':
    object: my_schema.{stream_name}
    mode: full-refresh

1. Run Replication.

sling run -r replication.api_postgres.yaml

That's it!

Official Specs

We are actively building the number of "official" Sling API Specs offered, such as:

• Airtable

• HubSpot

• Stripe

• Github

Go here to see the full list.

Official specs don't require a full URL or Path. They are maintained by the Sling team and can be fetched by specifying the corresponding ID, such as stripe, salesforce or github.

connections:
  stripe_api:
    type: api
    spec: stripe  # Use official stripe spec
    secrets:
      api_key: sk_live_xxxxxx

If you'd like us to build a new spec, please submit a request by filling out this Google Form, or by submitting a new Github issue (choosing the API Spec option).

Forking Official Specs

When you run a connection test or use an official spec, Sling automatically downloads and caches the spec file locally at:

$SLING_HOME_DIR/api/specs/{spec_id}.yaml

For example, if you're using the stripe spec, after running sling conns test my_stripe_conn, you'll find the spec at ~/.sling/api/specs/stripe.yaml.

To fork and customize an official spec:

1. Run a connection test to download the spec:

   sling conns test my_stripe_conn

2. Copy the spec from the cache folder:

   cp ~/.sling/api/specs/stripe.yaml ./my_custom_stripe.spec.yaml

3. Modify the spec as needed for your use case

4. Update your connection to use your custom spec:

   connections:
     my_stripe:
       type: api
       spec: file://./my_custom_stripe.spec.yaml
       secrets:
         api_key: sk_live_xxxxxx

API Spec Documentation

This section covers the details of building Sling API specifications:

• Structure: Understand the fundamental YAML structure of API specs, including endpoints, state management, sync variables, stream overrides, and lifecycle sequences (setup/teardown).

• Authentication: Configure authentication methods including Bearer tokens, Basic Auth, OAuth2, AWS Signature V4, and custom sequence-based authentication workflows.

• Requests & Iteration: Define HTTP requests with URLs, methods, headers, parameters, and payloads. Configure iteration to loop requests over data sets or queues, and use setup/teardown sequences for multi-step workflows.

• Response Processing: Process API responses in multiple formats (JSON, CSV, XML, JSON Lines), extract records with JMESPath, configure deduplication strategies, and access response state for pagination and rules.

• Queues: Pass data between endpoints using queues for multi-step extraction workflows, such as collecting IDs from one endpoint to use in detail requests from another endpoint.

• Dynamic Endpoints: Programmatically generate multiple endpoint configurations based on runtime data, perfect for APIs where available resources aren't known until queried.

• Advanced Features: Master pagination strategies (cursor, offset, page-based), response processors for data transformation, sync state for incremental loads, and rules for error handling with intelligent retry and backoff strategies.

• Expression Functions: Leverage built-in functions within expressions ({...}) for data manipulation, date operations, type casting, string operations, and control flow throughout your API spec configuration.

• Testing & Debugging: Test your API specs using the sling conns test command with --debug and --trace flags to inspect request/response details, troubleshoot issues, and optimize your configuration.

• Troubleshooting: Common error messages, debugging techniques, and solutions for authentication, pagination, and JMESPath issues.

API Workflow Overview

For practical examples, see the API Examples section.

Common Use Cases

Use Case	Key Features
Simple data extraction	Basic endpoint definition, JMESPath extraction
Paginated data	Pagination configuration with next_state and stop_condition
Incremental updates	sync state variables, timestamp filtering
Dependent requests	Queues, iteration over IDs
Authentication	Bearer tokens, Basic auth, OAuth2
Rate limiting	Response rules, backoff strategies

💡 Tip: Start with a minimal working spec and gradually add more advanced features as needed.