# CLI Pro

## CLI Pro Features

Sling CLI Pro extends the core functionality with advanced features designed for production environments and complex data operations.

* ✅ [API Sources](#api-sources) (extract data from any REST API by using Specs)
* ✅ [Parallel Stream Processing](#stream-chunking-and-parallel-processing) (run streams in parallel)
* ✅ [Stream Chunking](#stream-chunking-and-parallel-processing) (split large streams into smaller ones)
* ✅ [Pipelines & Hooks](#pipelines-and-hooks) (such as `http`, `query`, `check` and more)
* ✅ [OpenTelemetry Logging](#opentelemetry-logging) (export structured logs to any OTLP endpoint)
* ✅ [Capture Deletes](#capture-deletes) (similar to CDC)
* ✅ [Staged Transforms](#staged-transforms) (advanced multi-stage transformations with expressions and functions)
* ✅ [File Target Incremental Mode](#state-based-incremental-loading)
* ✅ [State Based Incremental](#state-based-incremental-loading)
* ✅ [ODBC Connections](#odbc-connections) (connect to any database via ODBC drivers)
* ✅ Support Sling and its continuous development

**CLI Pro Max Features**

* ✅ [Change Capture (CDC)](#change-capture-cdc) (continuously replicate row-level changes from transaction logs)
* ✅ [Schema Migration](#schema-migration) (migrate primary keys, foreign keys, indexes, defaults, and more)
* ✅ Priority Support (direct access to the Sling team for faster issue resolution)

{% hint style="success" %}
You can obtain a token for free at <https://dash.slingdata.io>. There is 7-day trial (no credit card needed).

Once you have a token, just put the value into the `SLING_CLI_TOKEN` environment variable before running sling (make sure the version is 1.4+).

For Pricing details see [here](https://slingdata.io/cli-pro).
{% endhint %}

### API Sources

Extract data from any REST API with powerful YAML-based specifications called `API Specs`:

* Define authentication methods (Bearer, Basic, OAuth2)
* Configure endpoints with pagination strategies
* Process responses with JMESPath extraction
* Manage state for incremental synchronization
* Support for queues and dependent requests
* Built-in retry and error handling

In your `env.yaml`:

```yaml
connections:
  stripe_api:
    type: api
    spec: stripe  # Use official spec or custom YAML (e.g. file://path/to/stripe.spec.yaml)
    secrets:
      api_key: sk_live_xxxxxx
```

In your replication:

```yaml
source: stripe_api
target: ducklake

defaults:
  object: stripe.{stream_name}

streams:
  customers:
    mode: incremental
```

See [API Specs](https://docs.slingdata.io/concepts/api-specs) for complete documentation and examples.

### Stream Chunking & Parallel Processing

Process large datasets efficiently with automatic chunking and parallel execution:

* Break down data into manageable chunks for various modes (`full-refresh`, `truncate`,`incremental`, `backfill`)
* Support for time-based (hours, days, months), numeric, count-based, and expression-based chunks
* Run multiple streams concurrently with automatic retry mechanisms
* Configurable concurrency and retry settings

```yaml
streams:
  my_schema.events:
    mode: full-refresh  # works with various modes
    primary_key: [id]
    update_key: event_date
    source_options:
      chunk_count: 8  # Process in 8 equal sized chunks

  my_schema.orders:
    mode: incremental  # works with various modes
    update_key: order_date
    source_options:
      chunk_size: 7d  # Process in 7-day chunks

env:
  SLING_THREADS: 3   # maximum of 3 streams concurrently
  SLING_RETRIES: 1   # maximum of 1 retry per failed stream
```

Environment variables:

* `SLING_THREADS` sets the maximum number of concurrent stream runs. Accepts an integer value, default is `1`.
* `SLING_RETRIES` sets the maximum number of retries for a failed stream run. Accepts an integer value, default is `0`.

See [Chunking](https://docs.slingdata.io/examples/database-to-database/chunking) for detailed examples.

### Pipelines & Hooks

Extend functionality with hooks and pipelines to create complex workflows. Hooks are used within replications to execute custom logic before/after operations, while Pipelines are standalone workflows that execute multiple steps in sequence.

Available action types:

| Step Type   | Description                                                | Documentation                                                            |
| ----------- | ---------------------------------------------------------- | ------------------------------------------------------------------------ |
| Check       | Validate conditions and control flow                       | [Check Hook](https://docs.slingdata.io/concepts/hooks/check)             |
| Command     | Run any command/process                                    | [Command Hook](https://docs.slingdata.io/concepts/hooks/command)         |
| Copy        | Transfer files between local or remote storage connections | [Copy Hook](https://docs.slingdata.io/concepts/hooks/copy)               |
| Delete      | Remove files from local or remote storage connections      | [Delete Hook](https://docs.slingdata.io/concepts/hooks/delete)           |
| Group       | Run sequences of steps or loop over values                 | [Group Hook](https://docs.slingdata.io/concepts/hooks/group)             |
| HTTP        | Make HTTP requests to external services                    | [HTTP Hook](https://docs.slingdata.io/concepts/hooks/http)               |
| Inspect     | Inspect a file or folder                                   | [Inspect Hook](https://docs.slingdata.io/concepts/hooks/inspect)         |
| List        | List files in folder                                       | [List Hook](https://docs.slingdata.io/concepts/hooks/list)               |
| Log         | Output custom messages and create audit trails             | [Log Hook](https://docs.slingdata.io/concepts/hooks/log)                 |
| Query       | Execute SQL queries against any defined connection         | [Query Hook](https://docs.slingdata.io/concepts/hooks/query)             |
| Read        | Read contents of files from storage connections            | [Read Hook](https://docs.slingdata.io/concepts/hooks/read)               |
| Replication | Run a Replication                                          | [Replication Hook](https://docs.slingdata.io/concepts/hooks/replication) |
| Routine     | Execute reusable step sequences from external files        | [Routine Hook](https://docs.slingdata.io/concepts/hooks/routine)         |
| Store       | Store values for later in-process access                   | [Store Hook](https://docs.slingdata.io/concepts/hooks/store)             |
| Write       | Write content to files in storage connections              | [Write Hook](https://docs.slingdata.io/concepts/hooks/write)             |

See [Hooks](https://docs.slingdata.io/concepts/hooks) and [Pipelines](https://docs.slingdata.io/concepts/pipeline) for usage examples and patterns.

### Staged Transforms

Transform data with advanced multi-stage processing using expressions and functions:

* Apply transformations in sequential stages with cross-column references
* Create new columns dynamically without modifying source schemas
* Use 50+ built-in functions for string, numeric, date, and conditional operations
* Build complex logic with `if/then/else` conditions and record references

```yaml
streams:
  customers:
    transforms:
      # Stage 1: Clean and normalize data
      - first_name: "trim_space(value)"
        last_name: "trim_space(value)" 
        email: "lower(value)"
      
      # Stage 2: Create computed columns
      - full_name: 'record.first_name + " " + record.last_name'
        email_hash: 'hash(record.email, "md5")'
      
      # Stage 3: Add business logic
      - customer_type: 'record.total_orders >= 50 ? "vip" : "regular"'
        discount_rate: 'record.customer_type == "vip" ? 0.15 : 0.05'
```

See [Transforms](https://docs.slingdata.io/concepts/replication/transforms) for detailed examples and [Available Functions](https://docs.slingdata.io/concepts/functions) for all available functions.

### State Based Incremental Loading

Maintain state across file & database loads with intelligent incremental processing:

* Track and resume file processing from last successful position
* Support for incremental writes to databases and files
* Automatic file partitioning and truncation management

See [Database to Database Incremental Loading](https://docs.slingdata.io/examples/database-to-database/incremental#using-sling_state), [Database to File Incremental Loading](https://docs.slingdata.io/examples/database-to-file/incremental) and [File to Database Incremental Loading](https://docs.slingdata.io/examples/file-to-database/incremental) for detailed examples.

### Capture Deletes

Track deleted records using a `_sling_deleted_at` column:

* Automatically detect and mark deleted records
* Maintain historical record states
* Support for soft deletes in target systems

See [Delete Missing Records](https://docs.slingdata.io/examples/database-to-database/capture_deletes) for implementation details.

### OpenTelemetry Logging

Export structured logs to any OpenTelemetry-compatible endpoint for centralized logging and observability:

* Send logs to any OTLP HTTP endpoint (Grafana, Datadog, Honeycomb, etc.)
* Automatic enrichment with execution attributes (such as `exec_id`, `stream_name`, `object_name`, `row_count`, `status`, `duration`, etc.)
* Structured log records with severity levels and timestamps
* Seamless integration with existing observability infrastructure

Set the `SLING_OTEL_ENDPOINT` environment variable to enable:

```bash
export SLING_OTEL_ENDPOINT='http://otel-collector:4318/v1/logs'
```

### Schema Migration

Migrate database schema attributes along with your data to preserve structure and relationships:

* Primary keys, foreign keys, and indexes
* Auto-increment/identity columns with seed and increment values
* NOT NULL constraints and default values
* Column and table descriptions/comments
* Automatic topological sorting for foreign key dependencies

```yaml
source: mssql
target: postgres

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  dbo.categories:
  dbo.customers:
  dbo.products:    # FK to categories
  dbo.orders:      # FK to customers
  dbo.order_items: # FK to orders and products

env:
  # Enable all schema attributes
  SLING_SCHEMA_MIGRATION: all

  # Or enable specific attributes
  # SLING_SCHEMA_MIGRATION: description, primary_key, foreign_key, indexes
```

Available options: `all`, `primary_key`, `foreign_key`, `indexes`, `auto_increment`, `nullable`, `default_value`, `description`

See [Schema Migration](https://docs.slingdata.io/examples/database-to-database/schema-migration) for detailed examples and supported databases.

### Change Capture (CDC)

Continuously replicate row-level changes (inserts, updates, deletes) from a source database's transaction log:

* Automatic initial snapshot with chunked, resumable loading
* Incremental change capture from transaction logs (binlog, WAL, etc.)
* Soft delete support to preserve deleted rows with timestamps
* Bounded runs with configurable event limits and duration
* Replay/backfill from earlier positions when needed

```yaml
source: MY_MYSQL
target: MY_POSTGRES

defaults:
  mode: change-capture
  primary_key: [id]
  object: public.{stream_table}
  change_capture_options:
    run_max_events: 10000
    run_max_duration: 10m

streams:
  my_database.customers:
  my_database.orders:
    change_capture_options:
      soft_delete: true
```

See [Change Capture (CDC)](https://docs.slingdata.io/concepts/change-capture) for complete documentation, supported sources, and examples.

### ODBC Connections

Connect to any database using Open Database Connectivity (ODBC) drivers:

* Access databases that may not have a dedicated Sling connector
* Use standardized ODBC interface for maximum compatibility
* Support for SQL Server, PostgreSQL, MySQL, DB2, SAP HANA, Teradata, and more
* Create custom SQL templates for unsupported database dialects

```yaml
connections:
  my_odbc:
    type: odbc
    conn_string: "Driver={ODBC Driver 18 for SQL Server};Server=myserver;Database=mydb;Uid=myuser;Pwd=mypassword"
```

See [ODBC Connections](https://docs.slingdata.io/connections/database-connections/odbc) for complete documentation, driver installation, and custom template examples.

## Frequently Asked Questions

**How are tokens validated?**

Tokens are validated through CloudFlare's global network, ensuring high reliability and fast response times worldwide. This validation occurs when the Sling CLI process initializes. If you'd like to confirm validation, run sling in debug mode (with flag `-d`), and you should see a log message: `CLI Pro token validated`.

**Can I get an offline/air-gapped token?**

For air-gapped or high-security environments, we offer offline license tokens. These require yearly renewal by default, but perpetual licenses are also available. Please contact <support@slingdata.io> to request a quote.

**How many subscriptions do I need?**

Each CLI Pro subscription includes 2 tokens:

* 1 Production token: For use in production environments
* 1 Development token: For development and testing

Each subscription is designed for **a single team** within your organization. A "team" refers to a cohesive group managing separate data pipelines, configurations, or business objectives that benefit from isolation for security, governance, or operational independence.

This per-team structure enables us to maintain predictable flat-rate pricing that's sustainable and fair for everyone, allowing us to deliver high-performance features and priority support without usage-based metering.

**What this means for you:**

* Each team or distinct project needs its own subscription
* Subscriptions are for internal use within your organization only
* You can use the production token across all your production environments (servers, containers, etc.)
* Team members can share the development token for testing and collaboration

**Company-wide licensing:** If you prefer a single license for your entire organization, perpetual licenses are available that cover company-wide usage. Contact <support@slingdata.io> to request a quote.

**Examples:**

* A data engineering team handling customer data → 1 subscription
* A separate analytics team working on reporting → 1 subscription
* Multiple independent teams in your organization → 1 subscription per team
* Consultants or freelancers serving multiple clients → 1 subscription per client

**Important Licensing Restrictions:**

{% hint style="warning" %}
**Reselling and Commercial Redistribution Prohibited**

CLI Pro subscriptions are licensed for use by the subscribing organization only. You are prohibited from:

* Reselling or redistributing access to CLI Pro features
* Acting as a service provider offering CLI Pro to third parties
* White-labeling or rebranding CLI Pro as your own service
* Providing commercial access to CLI Pro without proper licensing

**For Consultants and Service Providers:** If you wish to use CLI Pro in a consulting capacity or provide it to your clients, each client organization should have their own CLI Pro subscription. Contact us at <support@slingdata.io> to discuss partner licensing arrangements.

**For System Integrators:** We offer specific partner licensing programs for system integrators and technology partners. Contact us to discuss appropriate licensing for your use case.

Unauthorized reselling or redistribution will result in immediate termination of your subscription and may subject you to legal action.
{% endhint %}

Please use tokens responsibly and in accordance with our [Terms of Service](https://slingdata.io/terms). Each subscription is intended for use within a single organization or team, not for redistribution to external parties.