Metrics & Logs
Export metrics and logs from your Sling Platform project. Both are configured under Settings → Metrics & Logs.
Metrics
Export job, stream, and agent metrics for a Sling project in Prometheus format. Two transports, one metric model:
Pull.
GET /metricson the master, authenticated with a Project Token. Works with Prometheus, OpenTelemetry Collector, Grafana Alloy, and the Datadog agent.Push. The master remote-writes the same series to a Prometheus remote-write 1.0 endpoint on an interval. Works with Prometheus (
--web.enable-remote-write-receiver), Grafana Mimir and Grafana Cloud, Thanos, Cortex, VictoriaMetrics, Azure Monitor, and AWS managed Prometheus.
Available on the Advanced and Enterprise plans. An Admin enables it under Settings → Metrics & Logs.
Enable
Open Settings → Metrics & Logs.
Turn on Enable metrics.
Optionally turn on Stream-level series (about 9 series per stream) and Monitor object series (about 4 series per object). Leave them off on large projects.
Add extra labels if you want the same key/value on every series (
cluster,env, …). Names must match[a-zA-Z_][a-zA-Z0-9_]*and cannot bejob,instance,__name__,source,project,project_id, or anysling_*label.Save.
Pull (scrape)
The scrape URL is shown in the Metrics tab (also in /status/get as capabilities.metrics_scrape_url).
Authenticate with a project token:
Prometheus and most scrapers also accept Authorization: Bearer <PROJECT_TOKEN> or HTTP Basic with the token as the password.
Prometheus scrape_config
Prometheus Operator
Responses: 401 without a valid token, 403 if the plan does not allow metrics, 404 if metrics are disabled, 200 with text/plain; version=0.0.4.
Push (remote-write)
Turn on Push to remote-write endpoint, set the URL (for example https://prometheus.example:9090/api/v1/write), interval (minimum 15 seconds, default 30), and auth (none, basic, or bearer). Optional extra headers cover tenants (X-Scope-OrgID). Test push sends the live series once without saving.
The body is snappy block-compressed prometheus.WriteRequest protobuf (remote-write 1.0). Some gateways strip Content-Encoding; the error text from Test push includes the first bytes of the receiver body.
Backfill
Live series start at zero when you enable metrics. Settings → Metrics & Logs → Backfill history replays historical job and stream runs from task_runs (and related run tables) to the same remote-write target, as bucketed gauges stamped at UTC day or hour boundaries.
Backfill is a one-time, per-project action (Advanced and Enterprise). It is not available on the pull scrape endpoint — scrapes have no historical timestamps. Agent heartbeat history is not backfilled.
What it writes
sling_job_rows_backfill
gauge
sling_job, job_name, job_id, job_type, source
sling_job_bytes_backfill
gauge
sling_job, job_name, job_id, job_type, source
sling_job_runs_backfill
gauge
sling_job, job_name, job_id, job_type, status, source
sling_job_duration_seconds_backfill
gauge
sling_job, job_name, job_id, job_type, source
sling_stream_rows_backfill
gauge
sling_job, stream, source
sling_stream_bytes_backfill
gauge
sling_job, stream, source
sling_stream_runs_backfill
gauge
sling_job, stream, status, source
source is "backfill" on these series. Every live series (pull and push) carries source="live". The two never collide, so there is no counter-reset seam where history meets the live process.
Duration is the sum of run durations in the bucket, not a histogram.
source and dashboards
Panels that should show both live and history must union the two metric names, for example:
Do not reuse the live counter names for backfill. A cumulative counter's value at day N depends on where the running total started, so widening the range would silently rewrite shared points (Prometheus last-write-wins). Gauges for "rows in this UTC day" are the same number regardless of the requested range.
Receiver: out-of-order window
Vanilla Prometheus rejects samples older than the head block unless out_of_order_time_window covers the range. Check compatibility pushes a probe sample at the oldest requested timestamp and reports what the receiver needs before any real work.
Prometheus:
Grafana Mimir / Grafana Cloud: -ingester.out-of-order-time-window. The default is often much shorter than 90d.
The UI caps the range at 90 days (hourly step only under 14 days). Preflight may narrow it further if the receiver rejects the probe.
Idempotency
Buckets align to UTC midnight (or the UTC hour), never to "now minus N". Re-running the same calendar range computes the same values from the same source rows, so Prometheus deduplicates identical timestamp+value samples. It is safe to re-run after a partial failure. If the range overlaps a previous completed backfill, the UI warns and offers Overwrite, Skip overlap, or Cancel.
Purge
Remove backfilled data calls the Prometheus admin API delete_series?match[]={source="backfill",project_id="..."} when it is reachable (--web.enable-admin-api). Otherwise the UI copies the exact curl to run. Live series (source="live") are never deleted.
Grafana Cloud / Mimir
Use the Grafana Cloud Prometheus remote-write URL, basic auth with the instance ID as username and the Cloud Access Policy token as password, and header X-Scope-OrgID when the tenant is required.
Catalog
Namespace sling_. Base units seconds and bytes. _total for counters. Every series carries project (name), project_id, and source (live for scrape and remote-write; backfill for historical gauges), plus any labels you configured. The Prometheus job label is reserved, so the Sling job name is sling_job.
Descriptive attributes (connection names, connector types, versions) live on _info gauges with value 1. Dashboards join with group_left.
Counting rules
Job-level counters (sling_job_runs_total, sling_job_run_duration_seconds, sling_job_rows_total, sling_job_bytes_total) count one observation per execution. A replication with 10 streams is one job run; duration is wall time from the first stream start to the last stream end; rows and bytes are the sum. Stream and step counters (sling_stream_*, sling_pipeline_step_runs_total) count one observation per terminal row.
The first collect after enable (or after changing labels or series switches) sets a watermark and counts nothing. Changing labels or series switches rebuilds the in-memory registry, which Prometheus treats as a counter reset. Runs that ended between the last collect and the rebuild are not counted.
Jobs and executions
The sling_job label is the job file path (for example replications/banking.yaml). Two jobs can share one file, so this label is not unique. The job name (for example Banking - Hourly) is on sling_job_info as job_name, and falls back to the file path when the job has no name.
job_name is on the info metric only. To show names on a panel, join it in by job_id:
Default jobs
Every replication, pipeline, and build file gets an auto-created default job. It runs the file ad hoc and cannot be scheduled, so a file shows two jobs when a user also makes a scheduled job for it.
job_default on sling_job_info is "true" for these, "false" for user-created jobs. Monitor default jobs stay editable, so they are always "false".
A job counts as default when its ID matches its file's default job, or when its name starts with Default Job (. The name is the fallback: the file's pointer to its default job can be lost when a file is deleted and recreated, or through git sync.
To exclude default jobs, join through the info metric:
Stream metrics carry sling_job (the file path) but not job_id, so join on sling_job:
sling_job_info
gauge 1
sling_job, job_name, job_id, job_type, job_default, source_conn, source_type, target_conn, target_type, schedule
sling_job_scheduled
gauge 0/1
sling_job, job_id
sling_job_next_run_timestamp_seconds
gauge
sling_job, job_id
sling_job_runs_total
counter
sling_job, job_id, job_type, status
sling_job_run_duration_seconds
histogram
sling_job, job_id, job_type
sling_job_queue_wait_seconds
histogram
job_type
sling_job_rows_total, sling_job_bytes_total
counter
sling_job, job_id
sling_job_last_run_timestamp_seconds, sling_job_last_success_timestamp_seconds
gauge
sling_job, job_id
sling_job_last_run_duration_seconds
gauge
sling_job, job_id
sling_job_last_run_status
gauge 0/1
sling_job, job_id, status
sling_job_running
gauge
sling_job, job_id
sling_executions_queued
gauge
job_type, job_id
Streams and steps
Exported only when Stream-level series is on.
sling_stream_runs_total
counter
sling_job, stream, mode, status
sling_stream_rows_total, sling_stream_bytes_total
counter
sling_job, stream
sling_stream_last_run_duration_seconds, sling_stream_last_success_timestamp_seconds
gauge
sling_job, stream
sling_stream_last_run_status
gauge 0/1
sling_job, stream, status
sling_pipeline_step_runs_total
counter
sling_job, step, step_type, status
sling_stream_last_run_status is 1 on the label that matches the most recent terminal status of the stream, and the stream's other status labels are removed. It is the stream equivalent of sling_job_last_run_status.
Connections, monitors, agents, platform
sling_connection_rows_total, sling_connection_bytes_total
counter
direction is source or target
sling_monitor_events_total
counter
always on; labels connection, event_type, job_id
sling_monitor_object_*
gauge
only with Monitor object series
sling_agent_up
gauge
online and heartbeat < 60s
sling_agent_status
gauge 0/1
labels status
sling_agent_info
gauge 1
version, os, mode, hosting
sling_agent_replicas
gauge
state (shared-mode replica counts)
sling_agent_version_outdated
gauge
below the required agent version
sling_agent_active_executions
gauge
sling_agent_cpu_percent, sling_agent_cpu_count
gauge
sling_agent_memory_percent, sling_agent_memory_rss_bytes, sling_agent_memory_total_bytes
gauge
sling_agent_disk_percent, sling_agent_disk_free_bytes
gauge
sling_agent_last_heartbeat_timestamp_seconds
gauge
sling_agent_network_*_bytes_total
counter
cumulative since host boot. Agents built before the transmit-bytes fix report tx_bytes as receive+transmit until they upgrade.
sling_platform_info
gauge 1
version, mode
sling_platform_license_expiry_timestamp_seconds
gauge
self-hosted only
sling_metrics_collect_duration_seconds
gauge
last collect duration
sling_metrics_push_total
counter
result is ok or error
sling_metrics_last_push_timestamp_seconds
gauge
last successful remote-write
sling_logs_records_total
counter
log records shipped. result is ok or failed; source is executions, agents, server, or all
sling_logs_dropped_total
counter
log records dropped. reason is rate or hook; same source labels
Monitor events reach their job through the execution that raised the event. An event whose execution is gone gets an empty job_id, so exclude it in a join: sling_monitor_events_total{job_id!=""}.
Alert examples
Job overdue (scheduled, last success more than 2 hours ago):
Next run overdue:
Stream failed on its last run:
Push silence:
Series estimate
Roughly: jobs × (statuses 4 + histogram buckets ~24 + 10) + agents × 22 + connections × 4, plus streams × 10 when stream-level is on, plus objects × 4 when monitor objects are on. Keep stream-level and object series off until you need them.
job_name and job_default add no series. They are labels on sling_job_info, which is one series per job either way.
Every file adds a default job, so a project with N replication files has at least N jobs before any user-created job.
Dashboard
Two Grafana dashboards are provided. Both work with either transport, so import the one that matches the detail you want.
To import, go to Dashboards → New → Import in Grafana and paste the URL, or download the file and upload it:
On import, Grafana asks which Prometheus datasource to use. Point it at the one that receives your scrape or remote-write. Only the Execution logs panel needs Loki, so the detailed dashboard also asks for a Loki datasource. Skip it if you use Prometheus alone.
The Connections row of the detailed dashboard is collapsed, and holds 2 of its 31 panels.
Each dashboard has three filter variables. Project and Job filter the whole dashboard; Job lists job names, not file paths. Default jobs is Hide by default, set to Show to include the auto-created default job of each file. The detailed dashboard adds Log search, which filters the log panel only.
If one Prometheus holds more than one Sling instance, add a selector to keep them apart. A scrape stamps job from the scrape_config job name, and remote-write sends no job label, so job="sling" and job="" separate the two.
Stream panels key on the job file, not the job name. Stream metrics carry sling_job but no job_id, and several jobs can share one file, so a stream cannot be attributed to a single job.
Logs
Export execution, agent, and server logs for a Sling project over OTLP/HTTP (protobuf, gzip). Point the master at Loki, Grafana Cloud, an OpenTelemetry Collector, Elastic, or Datadog.
Available on self-hosted Platform (Advanced and Enterprise). Cloud projects can export metrics on the Advanced plan, but not logs. An Admin enables it under Settings → Metrics & Logs.
The same extra labels you set for metrics (cluster, env, …) are attached to log resources, so a dashboard can jump from a metric series to the matching lines by project, sling_job, exec_id, stream, and agent.
Enable
Open Settings → Metrics & Logs.
Turn on Enable log export.
Set the Endpoint to the full OTLP/HTTP logs URL. Presets fill Loki, Grafana Cloud, or Collector paths.
Choose auth (None / Basic / Bearer) and optional headers (
X-Scope-OrgID).Pick a Minimum level (default Info).
Choose sources: Executions, Agents, Server. Self-hosted installs can also Include all server lines.
Send test log, then Save.
Agent lines are Info and above, regardless of Minimum level. Query and monitor jobs export their error line only.
Endpoint examples
gRPC is not supported.
Attributes
Resource attributes (Loki indexes service.name and service.namespace by default):
service.name
sling (executions), sling-agent, sling-master
service.namespace
project name
project, project_id
project name and ID
user labels from Settings → Metrics & Logs
e.g. cluster=aks-prod
Record attributes are Loki structured metadata. Filter with | exec_id="...", | service_instance_id="prod-1", or | agent="prod-1" — they are not index labels unless you promote them in Loki otlp_config.
service.instance.id
agent name (executions, agents) or master hostname (server). Query as service_instance_id
service.version
CLI / agent / master version. Query as service_version
exec_id, job_id, sling_job, job_type
executions
stream, mode
replication executions
step, step_type
pipeline executions
object, node_type
build executions
agent, agent_id
executions, agents
status
executions
req_id, route, user_id, service
server
level
all
Body is the rendered line without ANSI codes. Severity follows zerolog: trace 1, debug 5, info 9, warn 13, error 17, fatal 21.
Status
The Metrics & Logs tab shows the last export:
delivered
Records the OTLP endpoint accepted (records)
failed
Records the endpoint rejected after retries
pending
Accepted locally, still in the SDK batch queue
dropped (rate)
Dropped by the per-source rate limiter before enqueue
OK means the last batch succeeded. A previous failed count can remain until the next successful export clears the last error.
Correlation with metrics
In Grafana Explore (LogQL):
The detailed dashboard includes an Execution logs panel that runs these queries, filtered by its Log search variable. It is the only panel that needs the Loki datasource.
Loki otlp_config
Unknown resource attributes land in structured metadata by default. To promote agent or a user label to an index label:
Limits
Default minimum level is Info (
debugis allowed). Agent lines stay Info and above.Per source: executions 200/s (burst 2000), agents 50/s, server 50/s. Over the limit: drop and count
sling_logs_dropped_total{reason="rate", source="..."}.SDK export queue 4096 records; pending shows what has not been delivered yet.
Record body cap 16 KiB (UTF-8 safe).
Agent relay: 50 lines/s per agent, flush every 2 s, max 200 lines per message.
Log export runs only on self-hosted masters. Include all server lines is also self-hosted only; otherwise the master exports server lines whose project_id matches the project.
Troubleshooting
401
Wrong Basic/Bearer token or Grafana Cloud instance id.
415
Path is not the OTLP logs URL (/otlp/v1/logs on Loki, /v1/logs on a Collector).
429
Back off; the exporter retries 429/5xx then drops. Test send retries up to 6 s.
Test succeeds, no lines in Loki
Query {service_namespace="<project name>"}. Check Min level and Sources.
Duplicate execution lines
SLING_OTEL_ENDPOINT is also set on the agent host. Turn one off.
Status says error, failed > 0
The endpoint is down or rejecting. Fix URL/auth; the next successful batch clears the error.
Last updated
Was this helpful?