# Cloudflare Analytics Engine

Cloudflare Analytics Engine is a time-series analytics database that allows you to store and query custom metrics from your Cloudflare Workers. The Sling Cloudflare Analytics Engine connector extracts data via the SQL API, automatically discovering all tables (datasets) in your account.

{% hint style="success" %}
**CLI Pro Required**: APIs require a [CLI Pro token](https://docs.slingdata.io/sling-cli/cli-pro) or [Platform Plan](https://docs.slingdata.io/sling-platform/platform).
{% endhint %}

## Setup

The following credentials and inputs are accepted:

**Secrets:**

* `api_token` **(required)** -> Your Cloudflare API Token with Analytics Engine read permissions
* `account_id` **(required)** -> Your Cloudflare Account ID

**Inputs:**

* `anchor_date` (optional) -> The starting date for historical data extraction (default: 90 days ago). Format: `YYYY-MM-DD`

### Getting Your Credentials

#### Account ID

1. Log in to your [Cloudflare Dashboard](https://dash.cloudflare.com/)
2. Select any domain in your account
3. Your **Account ID** is displayed in the right sidebar under "API"

#### API Token

1. Go to [Cloudflare API Tokens](https://dash.cloudflare.com/profile/api-tokens)
2. Click **Create Token**
3. Select **Create Custom Token**
4. Configure the token:
   * **Token name:** Sling Analytics Engine (or your preferred name)
   * **Permissions:** Account > Account Analytics > Read
   * **Account Resources:** Include your account
5. Click **Continue to summary** and then **Create Token**
6. Copy the token immediately (it won't be shown again)

{% hint style="warning" %}
**Important:** Make sure your API token has the `Account Analytics: Read` permission. Standard API keys won't work—you need a scoped API token.
{% endhint %}

### Using `sling conns`

{% code overflow="wrap" %}

```bash
sling conns set CF_ANALYTICS type=api spec=cloudflare_analytics_engine secrets='{ api_token: your_token_here, account_id: your_account_id }'
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/sling-cli/environment#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export CF_ANALYTICS='{ type: api, spec: cloudflare_analytics_engine, secrets: { api_token: "your_token_here", account_id: "your_account_id" } }'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/sling-cli/environment#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  CF_ANALYTICS:
    type: api
    spec: cloudflare_analytics_engine
    secrets:
      api_token: "your_token_here"
      account_id: "your_account_id"
```

**With custom anchor date:**

```yaml
connections:
  CF_ANALYTICS:
    type: api
    spec: cloudflare_analytics_engine
    secrets:
      api_token: "your_token_here"
      account_id: "your_account_id"
    inputs:
      anchor_date: "2024-01-01"
```

## Dynamic Endpoints

The Cloudflare Analytics Engine connector uses **dynamic endpoints** to automatically discover all tables (datasets) in your account. When you run `sling conns discover`, the connector:

1. Queries `SHOW TABLES` via the SQL API
2. Creates an endpoint for each discovered table
3. Names each endpoint using snake\_case (e.g., `my_dataset` for a table named "MyDataset")

To see available endpoints:

```bash
sling conns discover CF_ANALYTICS
```

## Replication

Here's an example replication configuration to sync Analytics Engine data to a PostgreSQL database:

```yaml
source: CF_ANALYTICS
target: MY_POSTGRES

defaults:
  mode: incremental
  object: analytics.{stream_name}

streams:
  # sync all discovered tables
  '*':
```

**Full refresh example:**

```yaml
source: CF_ANALYTICS
target: MY_POSTGRES

defaults:
  mode: full-refresh
  object: analytics.{stream_name}

streams:
  # sync specific tables
  my_metrics:
  page_views:
  api_requests:
```

## Incremental Sync

The connector uses time-based incremental sync with the `timestamp` field:

* **First run:** Fetches all records from `anchor_date` (default: 90 days ago) to present
* **Subsequent runs:** Only fetches records with `timestamp` >= last synced timestamp

The connector automatically:

* Tracks the maximum `timestamp` value from each run
* Uses this as the starting point for the next run
* Paginates through large datasets (10,000 records per page)

{% hint style="info" %}
Analytics Engine data has a **90-day retention** by default. Set `anchor_date` appropriately based on your retention settings.
{% endhint %}

## Data Schema

Analytics Engine tables have a flexible schema. Common columns include:

| Column                 | Description                                 |
| ---------------------- | ------------------------------------------- |
| `timestamp`            | Event timestamp (used for incremental sync) |
| `index1`               | First index dimension                       |
| `blob1`...`blob20`     | String blob columns                         |
| `double1`...`double20` | Numeric double columns                      |

The exact columns depend on how you've configured your Analytics Engine datasets.

## Rate Limiting

The connector is configured with conservative rate limiting:

* **Rate:** 5 requests per second
* **Concurrency:** 3 parallel requests
* **Retry:** Exponential backoff on 429 (rate limit) responses

## Common Use Cases

### Sync All Analytics Data

```yaml
source: CF_ANALYTICS
target: MY_SNOWFLAKE

defaults:
  mode: incremental
  object: cloudflare.{stream_name}

streams:
  '*':
```

### Sync Specific Datasets

```yaml
source: CF_ANALYTICS
target: MY_POSTGRES

defaults:
  mode: incremental
  object: metrics.{stream_name}

streams:
  page_views:
  api_latency:
  error_rates:
```

### Historical Backfill

```yaml
source: CF_ANALYTICS
target: MY_POSTGRES

defaults:
  mode: full-refresh
  object: analytics.{stream_name}

env:
  ANCHOR_DATE: "2024-06-01"

streams:
  '*':
```

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).