> For the complete documentation index, see [llms.txt](https://docs.slingdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.slingdata.io/sling-platform/platform/metrics-logs.md).

# Metrics & Logs

Export metrics and logs from your Sling Platform project. Both are configured under **Settings → Metrics & Logs**.

## Metrics

Export job, stream, and agent metrics for a Sling project in Prometheus format. Two transports, one metric model:

1. **Pull.** `GET /metrics` on the master, authenticated with a Project Token. Works with Prometheus, OpenTelemetry Collector, Grafana Alloy, and the Datadog agent.
2. **Push.** The master remote-writes the same series to a Prometheus remote-write 1.0 endpoint on an interval. Works with Prometheus (`--web.enable-remote-write-receiver`), Grafana Mimir and Grafana Cloud, Thanos, Cortex, VictoriaMetrics, Azure Monitor, and AWS managed Prometheus.

Available on the **Advanced** and **Enterprise** plans. An Admin enables it under **Settings → Metrics & Logs**.

{% hint style="info" %}
Counters live in process memory and reset when the master restarts. `rate()` and `increase()` handle this. A retried stream produces a second terminal row, so both attempts count. The master is a single replica; scrape that instance.
{% endhint %}

### Enable

1. Open **Settings → Metrics & Logs**.
2. Turn on **Enable metrics**.
3. Optionally turn on **Stream-level series** (about 9 series per stream) and **Monitor object series** (about 4 series per object). Leave them off on large projects.
4. Add extra labels if you want the same key/value on every series (`cluster`, `env`, …). Names must match `[a-zA-Z_][a-zA-Z0-9_]*` and cannot be `job`, `instance`, `__name__`, `source`, `project`, `project_id`, or any `sling_*` label.
5. Save.

### Pull (scrape)

The scrape URL is shown in the Metrics tab (also in `/status/get` as `capabilities.metrics_scrape_url`).

Authenticate with a [project token](/sling-platform/platform/api.md):

```
Authorization: Sling-Project-Token <PROJECT_TOKEN>
```

Prometheus and most scrapers also accept `Authorization: Bearer <PROJECT_TOKEN>` or HTTP Basic with the token as the password.

#### Prometheus scrape\_config

```yaml
scrape_configs:
  - job_name: sling
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets: ["your-sling-host"]
    authorization:
      type: Sling-Project-Token
      credentials: <PROJECT_TOKEN>
```

#### Prometheus Operator

```yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: sling
spec:
  scheme: HTTPS
  metricsPath: /metrics
  staticConfigs:
    - targets: ["your-sling-host"]
  authorization:
    type: Sling-Project-Token
    credentials:
      name: sling-project-token
      key: token
```

Responses: `401` without a valid token, `403` if the plan does not allow metrics, `404` if metrics are disabled, `200` with `text/plain; version=0.0.4`.

### Push (remote-write)

Turn on **Push to remote-write endpoint**, set the URL (for example `https://prometheus.example:9090/api/v1/write`), interval (minimum 15 seconds, default 30), and auth (none, basic, or bearer). Optional extra headers cover tenants (`X-Scope-OrgID`). **Test push** sends the live series once without saving.

The body is snappy block-compressed `prometheus.WriteRequest` protobuf (remote-write 1.0). Some gateways strip `Content-Encoding`; the error text from **Test push** includes the first bytes of the receiver body.

### Backfill

Live series start at zero when you enable metrics. **Settings → Metrics & Logs → Backfill history** replays historical job and stream runs from `task_runs` (and related run tables) to the same remote-write target, as **bucketed gauges** stamped at UTC day or hour boundaries.

Backfill is a one-time, per-project action (Advanced and Enterprise). It is not available on the pull scrape endpoint — scrapes have no historical timestamps. Agent heartbeat history is not backfilled.

#### What it writes

| Metric                                | Type  | Labels                                                    |
| ------------------------------------- | ----- | --------------------------------------------------------- |
| `sling_job_rows_backfill`             | gauge | sling\_job, job\_name, job\_id, job\_type, source         |
| `sling_job_bytes_backfill`            | gauge | sling\_job, job\_name, job\_id, job\_type, source         |
| `sling_job_runs_backfill`             | gauge | sling\_job, job\_name, job\_id, job\_type, status, source |
| `sling_job_duration_seconds_backfill` | gauge | sling\_job, job\_name, job\_id, job\_type, source         |
| `sling_stream_rows_backfill`          | gauge | sling\_job, stream, source                                |
| `sling_stream_bytes_backfill`         | gauge | sling\_job, stream, source                                |
| `sling_stream_runs_backfill`          | gauge | sling\_job, stream, status, source                        |

`source` is `"backfill"` on these series. Every **live** series (pull and push) carries `source="live"`. The two never collide, so there is no counter-reset seam where history meets the live process.

Duration is the **sum** of run durations in the bucket, not a histogram.

#### `source` and dashboards

Panels that should show both live and history must union the two metric names, for example:

```promql
sum by (job_id) (sling_job_rows_backfill{source="backfill"})
  or
sum by (job_id) (increase(sling_job_rows_total{source="live"}[1d]))
```

Do not reuse the live counter names for backfill. A cumulative counter's value at day *N* depends on where the running total started, so widening the range would silently rewrite shared points (Prometheus last-write-wins). Gauges for "rows in this UTC day" are the same number regardless of the requested range.

#### Receiver: out-of-order window

Vanilla Prometheus rejects samples older than the head block unless `out_of_order_time_window` covers the range. **Check compatibility** pushes a probe sample at the oldest requested timestamp and reports what the receiver needs before any real work.

Prometheus:

```yaml
storage:
  tsdb:
    out_of_order_time_window: 90d
```

Grafana Mimir / Grafana Cloud: `-ingester.out-of-order-time-window`. The default is often much shorter than 90d.

The UI caps the range at 90 days (hourly step only under 14 days). Preflight may narrow it further if the receiver rejects the probe.

#### Idempotency

Buckets align to UTC midnight (or the UTC hour), never to "now minus N". Re-running the same calendar range computes the same values from the same source rows, so Prometheus deduplicates identical timestamp+value samples. It is safe to re-run after a partial failure. If the range overlaps a previous completed backfill, the UI warns and offers **Overwrite**, **Skip overlap**, or **Cancel**.

#### Purge

**Remove backfilled data** calls the Prometheus admin API `delete_series?match[]={source="backfill",project_id="..."}` when it is reachable (`--web.enable-admin-api`). Otherwise the UI copies the exact `curl` to run. Live series (`source="live"`) are never deleted.

#### Grafana Cloud / Mimir

Use the Grafana Cloud Prometheus remote-write URL, basic auth with the instance ID as username and the Cloud Access Policy token as password, and header `X-Scope-OrgID` when the tenant is required.

### Catalog

Namespace `sling_`. Base units seconds and bytes. `_total` for counters. Every series carries `project` (name), `project_id`, and `source` (`live` for scrape and remote-write; `backfill` for historical gauges), plus any labels you configured. The Prometheus `job` label is reserved, so the Sling job name is `sling_job`.

Descriptive attributes (connection names, connector types, versions) live on `_info` gauges with value 1. Dashboards join with `group_left`.

#### Counting rules

Job-level counters (`sling_job_runs_total`, `sling_job_run_duration_seconds`, `sling_job_rows_total`, `sling_job_bytes_total`) count **one observation per execution**. A replication with 10 streams is one job run; duration is wall time from the first stream start to the last stream end; rows and bytes are the sum. Stream and step counters (`sling_stream_*`, `sling_pipeline_step_runs_total`) count one observation per terminal row.

The first collect after enable (or after changing labels or series switches) sets a watermark and counts nothing. Changing labels or series switches rebuilds the in-memory registry, which Prometheus treats as a counter reset. Runs that ended between the last collect and the rebuild are not counted.

#### Jobs and executions

The `sling_job` label is the job **file path** (for example `replications/banking.yaml`). Two jobs can share one file, so this label is not unique. The job **name** (for example `Banking - Hourly`) is on `sling_job_info` as `job_name`, and falls back to the file path when the job has no name.

`job_name` is on the info metric only. To show names on a panel, join it in by `job_id`:

```promql
sum by (job_id) (sling_job_rows_total) * on (job_id) group_left(job_name) sling_job_info
```

#### Default jobs

Every replication, pipeline, and build file gets an auto-created **default job**. It runs the file ad hoc and cannot be scheduled, so a file shows two jobs when a user also makes a scheduled job for it.

`job_default` on `sling_job_info` is `"true"` for these, `"false"` for user-created jobs. Monitor default jobs stay editable, so they are always `"false"`.

A job counts as default when its ID matches its file's default job, or when its name starts with `Default Job (`. The name is the fallback: the file's pointer to its default job can be lost when a file is deleted and recreated, or through git sync.

To exclude default jobs, join through the info metric:

```promql
sum by (job_id) (sling_job_rows_total)
  * on (job_id) group_left(job_name)
    max by (job_id, job_name) (sling_job_info{job_default="false"})
```

Stream metrics carry `sling_job` (the file path) but not `job_id`, so join on `sling_job`:

```promql
sum by (job_name, stream) (
  sum by (sling_job, stream) (sling_stream_rows_total)
    * on (sling_job) group_left(job_name)
      max by (sling_job, job_name) (sling_job_info{job_default="false"})
)
```

| Metric                                                                             | Type      | Labels                                                                                                                    |
| ---------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------- |
| `sling_job_info`                                                                   | gauge 1   | sling\_job, job\_name, job\_id, job\_type, job\_default, source\_conn, source\_type, target\_conn, target\_type, schedule |
| `sling_job_scheduled`                                                              | gauge 0/1 | sling\_job, job\_id                                                                                                       |
| `sling_job_next_run_timestamp_seconds`                                             | gauge     | sling\_job, job\_id                                                                                                       |
| `sling_job_runs_total`                                                             | counter   | sling\_job, job\_id, job\_type, status                                                                                    |
| `sling_job_run_duration_seconds`                                                   | histogram | sling\_job, job\_id, job\_type                                                                                            |
| `sling_job_queue_wait_seconds`                                                     | histogram | job\_type                                                                                                                 |
| `sling_job_rows_total`, `sling_job_bytes_total`                                    | counter   | sling\_job, job\_id                                                                                                       |
| `sling_job_last_run_timestamp_seconds`, `sling_job_last_success_timestamp_seconds` | gauge     | sling\_job, job\_id                                                                                                       |
| `sling_job_last_run_duration_seconds`                                              | gauge     | sling\_job, job\_id                                                                                                       |
| `sling_job_last_run_status`                                                        | gauge 0/1 | sling\_job, job\_id, status                                                                                               |
| `sling_job_running`                                                                | gauge     | sling\_job, job\_id                                                                                                       |
| `sling_executions_queued`                                                          | gauge     | job\_type, job\_id                                                                                                        |

#### Streams and steps

Exported only when **Stream-level series** is on.

| Metric                                                                                  | Type      | Labels                               |
| --------------------------------------------------------------------------------------- | --------- | ------------------------------------ |
| `sling_stream_runs_total`                                                               | counter   | sling\_job, stream, mode, status     |
| `sling_stream_rows_total`, `sling_stream_bytes_total`                                   | counter   | sling\_job, stream                   |
| `sling_stream_last_run_duration_seconds`, `sling_stream_last_success_timestamp_seconds` | gauge     | sling\_job, stream                   |
| `sling_stream_last_run_status`                                                          | gauge 0/1 | sling\_job, stream, status           |
| `sling_pipeline_step_runs_total`                                                        | counter   | sling\_job, step, step\_type, status |

`sling_stream_last_run_status` is 1 on the label that matches the most recent terminal status of the stream, and the stream's other status labels are removed. It is the stream equivalent of `sling_job_last_run_status`.

#### Connections, monitors, agents, platform

| Metric                                                                                         | Type      | Notes                                                                                                                            |
| ---------------------------------------------------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `sling_connection_rows_total`, `sling_connection_bytes_total`                                  | counter   | `direction` is `source` or `target`                                                                                              |
| `sling_monitor_events_total`                                                                   | counter   | always on; labels `connection`, `event_type`, `job_id`                                                                           |
| `sling_monitor_object_*`                                                                       | gauge     | only with **Monitor object series**                                                                                              |
| `sling_agent_up`                                                                               | gauge     | online and heartbeat < 60s                                                                                                       |
| `sling_agent_status`                                                                           | gauge 0/1 | labels `status`                                                                                                                  |
| `sling_agent_info`                                                                             | gauge 1   | `version`, `os`, `mode`, `hosting`                                                                                               |
| `sling_agent_replicas`                                                                         | gauge     | `state` (shared-mode replica counts)                                                                                             |
| `sling_agent_version_outdated`                                                                 | gauge     | below the required agent version                                                                                                 |
| `sling_agent_active_executions`                                                                | gauge     |                                                                                                                                  |
| `sling_agent_cpu_percent`, `sling_agent_cpu_count`                                             | gauge     |                                                                                                                                  |
| `sling_agent_memory_percent`, `sling_agent_memory_rss_bytes`, `sling_agent_memory_total_bytes` | gauge     |                                                                                                                                  |
| `sling_agent_disk_percent`, `sling_agent_disk_free_bytes`                                      | gauge     |                                                                                                                                  |
| `sling_agent_last_heartbeat_timestamp_seconds`                                                 | gauge     |                                                                                                                                  |
| `sling_agent_network_*_bytes_total`                                                            | counter   | cumulative since host boot. Agents built before the transmit-bytes fix report `tx_bytes` as receive+transmit until they upgrade. |
| `sling_platform_info`                                                                          | gauge 1   | version, mode                                                                                                                    |
| `sling_platform_license_expiry_timestamp_seconds`                                              | gauge     | self-hosted only                                                                                                                 |
| `sling_metrics_collect_duration_seconds`                                                       | gauge     | last collect duration                                                                                                            |
| `sling_metrics_push_total`                                                                     | counter   | `result` is `ok` or `error`                                                                                                      |
| `sling_metrics_last_push_timestamp_seconds`                                                    | gauge     | last successful remote-write                                                                                                     |
| `sling_logs_records_total`                                                                     | counter   | log records shipped. `result` is `ok` or `failed`; `source` is `executions`, `agents`, `server`, or `all`                        |
| `sling_logs_dropped_total`                                                                     | counter   | log records dropped. `reason` is `rate` or `hook`; same `source` labels                                                          |

Monitor events reach their job through the execution that raised the event. An event whose execution is gone gets an empty `job_id`, so exclude it in a join: `sling_monitor_events_total{job_id!=""}`.

### Alert examples

Job overdue (scheduled, last success more than 2 hours ago):

```yaml
- alert: SlingJobStale
  expr: time() - sling_job_last_success_timestamp_seconds > 7200 and sling_job_scheduled == 1
```

Next run overdue:

```yaml
- alert: SlingScheduleOverdue
  expr: time() - sling_job_next_run_timestamp_seconds > 900 and sling_job_scheduled == 1
```

Stream failed on its last run:

```yaml
- alert: SlingStreamFailing
  expr: sling_stream_last_run_status{status="error"} == 1
```

Push silence:

```yaml
- alert: SlingMetricsPushSilent
  expr: time() - sling_metrics_last_push_timestamp_seconds > 300
```

### Series estimate

Roughly: jobs × (statuses 4 + histogram buckets \~24 + 10) + agents × 22 + connections × 4, plus streams × 10 when stream-level is on, plus objects × 4 when monitor objects are on. Keep stream-level and object series off until you need them.

`job_name` and `job_default` add no series. They are labels on `sling_job_info`, which is one series per job either way.

Every file adds a default job, so a project with N replication files has at least N jobs before any user-created job.

### Dashboard

Two Grafana dashboards are provided. Both work with either transport, so import the one that matches the detail you want.

| Dashboard                                                                                   | Panels | Use                                          |
| ------------------------------------------------------------------------------------------- | ------ | -------------------------------------------- |
| [`sling-platform.json`](https://slingdata.io/grafana/sling-platform.json)                   | 14     | Headline totals, throughput, and job status. |
| [`sling-platform-detailed.json`](https://slingdata.io/grafana/sling-platform-detailed.json) | 31     | Adds streams, agents, connections, and logs. |

To import, go to **Dashboards → New → Import** in Grafana and paste the URL, or download the file and upload it:

{% code overflow="wrap" %}

```bash
curl -O https://slingdata.io/grafana/sling-platform.json
```

{% endcode %}

On import, Grafana asks which Prometheus datasource to use. Point it at the one that receives your scrape or remote-write. Only the **Execution logs** panel needs Loki, so the detailed dashboard also asks for a Loki datasource. Skip it if you use Prometheus alone.

The `Connections` row of the detailed dashboard is collapsed, and holds 2 of its 31 panels.

Each dashboard has three filter variables. `Project` and `Job` filter the whole dashboard; `Job` lists job names, not file paths. `Default jobs` is `Hide` by default, set to `Show` to include the auto-created default job of each file. The detailed dashboard adds `Log search`, which filters the log panel only.

If one Prometheus holds more than one Sling instance, add a selector to keep them apart. A scrape stamps `job` from the `scrape_config` job name, and remote-write sends no `job` label, so `job="sling"` and `job=""` separate the two.

Stream panels key on the job **file**, not the job name. Stream metrics carry `sling_job` but no `job_id`, and several jobs can share one file, so a stream cannot be attributed to a single job.

## Logs

Export execution, agent, and server logs for a Sling project over **OTLP/HTTP** (protobuf, gzip). Point the master at Loki, Grafana Cloud, an OpenTelemetry Collector, Elastic, or Datadog.

Available on **self-hosted** Platform (**Advanced** and **Enterprise**). Cloud projects can export metrics on the Advanced plan, but not logs. An Admin enables it under **Settings → Metrics & Logs**.

The same extra labels you set for metrics (`cluster`, `env`, …) are attached to log resources, so a dashboard can jump from a metric series to the matching lines by `project`, `sling_job`, `exec_id`, `stream`, and `agent`.

{% hint style="info" %}
The master is the only exporter. Secrets stay on the master. Execution lines already arrive there over NATS; agent process logs are relayed the same way when the Agents source is on. The CLI `SLING_OTEL_ENDPOINT` path is a different feature (a local `sling` process) and is not set by the platform. If both are on, executions can appear twice with different resources.
{% endhint %}

### Enable

1. Open **Settings → Metrics & Logs**.
2. Turn on **Enable log export**.
3. Set the **Endpoint** to the full OTLP/HTTP logs URL. Presets fill Loki, Grafana Cloud, or Collector paths.
4. Choose auth (None / Basic / Bearer) and optional headers (`X-Scope-OrgID`).
5. Pick a **Minimum level** (default Info).
6. Choose sources: Executions, Agents, Server. Self-hosted installs can also **Include all server lines**.
7. **Send test log**, then Save.

Agent lines are Info and above, regardless of **Minimum level**. Query and monitor jobs export their error line only.

### Endpoint examples

```
# Loki
https://loki.example:3100/otlp/v1/logs

# Grafana Cloud
https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/logs
Authorization: Bearer <instance-id>:<api-token>

# OpenTelemetry Collector
http://otel-collector:4318/v1/logs
```

gRPC is not supported.

### Attributes

Resource attributes (Loki indexes `service.name` and `service.namespace` by default):

| Attribute                                  | Value                                               |
| ------------------------------------------ | --------------------------------------------------- |
| `service.name`                             | `sling` (executions), `sling-agent`, `sling-master` |
| `service.namespace`                        | project name                                        |
| `project`, `project_id`                    | project name and ID                                 |
| user labels from Settings → Metrics & Logs | e.g. `cluster=aks-prod`                             |

Record attributes are Loki **structured metadata**. Filter with `| exec_id="..."`, `| service_instance_id="prod-1"`, or `| agent="prod-1"` — they are not index labels unless you promote them in Loki `otlp_config`.

| Attribute                                    | Present on                                                                                  |
| -------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `service.instance.id`                        | agent name (executions, agents) or master hostname (server). Query as `service_instance_id` |
| `service.version`                            | CLI / agent / master version. Query as `service_version`                                    |
| `exec_id`, `job_id`, `sling_job`, `job_type` | executions                                                                                  |
| `stream`, `mode`                             | replication executions                                                                      |
| `step`, `step_type`                          | pipeline executions                                                                         |
| `object`, `node_type`                        | build executions                                                                            |
| `agent`, `agent_id`                          | executions, agents                                                                          |
| `status`                                     | executions                                                                                  |
| `req_id`, `route`, `user_id`, `service`      | server                                                                                      |
| `level`                                      | all                                                                                         |

Body is the rendered line without ANSI codes. Severity follows zerolog: trace 1, debug 5, info 9, warn 13, error 17, fatal 21.

### Status

The Metrics & Logs tab shows the last export:

| Field          | Meaning                                               |
| -------------- | ----------------------------------------------------- |
| delivered      | Records the OTLP endpoint accepted (`records`)        |
| failed         | Records the endpoint rejected after retries           |
| pending        | Accepted locally, still in the SDK batch queue        |
| dropped (rate) | Dropped by the per-source rate limiter before enqueue |

`OK` means the last batch succeeded. A previous `failed` count can remain until the next successful export clears the last error.

### Correlation with metrics

In Grafana Explore (LogQL):

```
{service_namespace="my-project", service_name="sling"} | exec_id="exec_abc"
{service_namespace="my-project"} | sling_job="sync.yaml"
{service_name="sling-agent"} | agent="prod-1"
{service_name="sling-master"} | route="JobSave"
```

The [detailed dashboard](#dashboard) includes an **Execution logs** panel that runs these queries, filtered by its `Log search` variable. It is the only panel that needs the Loki datasource.

### Loki `otlp_config`

Unknown resource attributes land in structured metadata by default. To promote `agent` or a user label to an index label:

```yaml
otlp_config:
  resource_attributes:
    attributes:
      - action: index_label
        attributes: [agent, cluster]
```

### Limits

* Default minimum level is Info (`debug` is allowed). Agent lines stay Info and above.
* Per source: executions 200/s (burst 2000), agents 50/s, server 50/s. Over the limit: drop and count `sling_logs_dropped_total{reason="rate", source="..."}`.
* SDK export queue 4096 records; pending shows what has not been delivered yet.
* Record body cap 16 KiB (UTF-8 safe).
* Agent relay: 50 lines/s per agent, flush every 2 s, max 200 lines per message.

Log export runs only on self-hosted masters. `Include all server lines` is also self-hosted only; otherwise the master exports server lines whose `project_id` matches the project.

### Troubleshooting

| Symptom                         | Cause                                                                                        |
| ------------------------------- | -------------------------------------------------------------------------------------------- |
| 401                             | Wrong Basic/Bearer token or Grafana Cloud instance id.                                       |
| 415                             | Path is not the OTLP logs URL (`/otlp/v1/logs` on Loki, `/v1/logs` on a Collector).          |
| 429                             | Back off; the exporter retries 429/5xx then drops. Test send retries up to 6 s.              |
| Test succeeds, no lines in Loki | Query `{service_namespace="<project name>"}`. Check Min level and Sources.                   |
| Duplicate execution lines       | `SLING_OTEL_ENDPOINT` is also set on the agent host. Turn one off.                           |
| Status says error, `failed > 0` | The endpoint is down or rejecting. Fix URL/auth; the next successful batch clears the error. |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.slingdata.io/sling-platform/platform/metrics-logs.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.