High-Availability Agents

Run many agent replicas on one key, with failover and platform-managed scaling

By default, every agent process needs its own unique key (single mode). Agents can instead run in shared mode, where many replica processes connect with the same key and act as one logical agent:

• The platform load-balances job executions across replicas (least-loaded first).

• Heartbeats and metrics are aggregated across replicas in the UI.

• If a replica dies mid-execution, the platform automatically requeues the stalled execution on a surviving replica (failover).

There are two ways to run shared-mode agents: manually (you scale the replicas) or platform-managed (the platform provisions and scales them for you on Docker or Kubernetes).

Enabling Shared Mode

Shared mode is enabled on the platform server, not on the agent:

services:
  sling-platform:
    environment:
      SLING_RUNNER_MODE: shared

This is a server-level setting that applies to every agent created afterwards — existing agents keep their current mode. The platform is authoritative at connect time, so replicas need only the agent key; the agent's own SLING_RUNNER_MODE does not decide the mode.

Manual Shared-Mode Replicas

Run any number of agent processes with the same key. Setting SLING_RUNNER_MODE=shared on the agent is optional (the platform decides), but harmless:

agent-deployment.yaml (shared-mode replicas)

# kubernetes: a Deployment with replicas > 1 and one shared key
env:
- name: SLING_PLATFORM_HOST
  value: "http://sling-platform:7878"
- name: SLING_AGENT_KEY
  valueFrom:
    secretKeyRef:
      name: sling-platform-secrets
      key: SLING_AGENT_KEY
- name: SLING_RUNNER_MODE
  value: "shared"
- name: POD_NAME       # lets the platform display the pod name per replica
  valueFrom:
    fieldRef:
      fieldPath: metadata.name

The same works with docker compose up --scale sling-agent=3 or several docker run invocations sharing one key.

The agent must exist as shared on the platform side, which means it was created while SLING_RUNNER_MODE=shared was set on the platform server (agents created through the Docker (Local) or Kubernetes hosting options are always shared). Each connecting replica then registers its own session against the shared key.

All replicas of a shared key must run a recent agent binary. Older binaries predate HA and subscribe to the broadcast subject without a queue group, so mixing them with new-style replicas on one key can deliver — and run — the same job twice. The platform rejects a session-less (old) binary that tries to join a key already held by a live session, and logs a warning when it sees one. Upgrade every replica before scaling out.

Each replica needs its own cache directory. The agent keeps a local cache under its data dir; point it at per-pod ephemeral storage (an emptyDir volume in Kubernetes, the container filesystem in Docker) — never a shared PersistentVolumeClaim across replicas. This applies to replicas you run yourself; platform-managed agents (Docker (Local) and Kubernetes) already give each replica its own ephemeral cache.

Concurrency Limits in Shared Mode

An agent's max executions is a pool-wide cap in shared mode: the platform compares it against the aggregate number of active executions across all replicas, not per replica. The default heuristic (cpu_count × 2) scales automatically, because the reported CPU count is summed across live replicas — adding a replica raises the pool's capacity without any configuration change.

Docker-Local Managed Agents

For single-host installs (the typical docker compose up deployment), the platform can provision agent replica containers directly on the local Docker daemon. Enable it on the platform container:

docker-compose.yaml (Docker-local runners)

services:
  sling-platform:
    image: slingdata/sling-platform
    environment:
      # ... required vars ...
      SLING_DOCKER_RUNNERS: '{"mode": "unix-socket", "network": "sling-net"}'
    volumes:
      - sling-platform-data:/home/sling
      - /var/run/docker.sock:/var/run/docker.sock # gives the platform access to the daemon
    networks: [sling-net]

networks:
  sling-net:

Once enabled, a Docker (Local) hosting option appears in the agent creation dialog. The platform creates the containers, reconciles the replica count every 30 seconds, scales down oldest-first, and removes the containers when the agent is deleted.

SLING_DOCKER_RUNNERS accepts a bare mode (unix-socket or tcp) or a JSON object. The network field is required when the master runs in a container, so replicas can reach it by container name. All fields except mode:

SLING_DOCKER_RUNNERS fields

{
  "mode": "unix-socket",
  "host": "unix:///var/run/docker.sock",
  "tls_verify": false,
  "cert_path": "",
  "image": "slingdata/sling:latest",
  "network": "sling-net",
  "labels": { "team": "data" },
  "restart_policy": "unless-stopped",
  "cpu": 2,
  "memory_mb": 3072,
  "master_url": "",
  "nats_url": ""
}

Field	Default	Description
mode	(required)	unix-socket or tcp
host	unix:///var/run/docker.sock	Daemon address (tcp://host:2376 for tcp mode)
tls_verify / cert_path	(off)	TLS client certs for tcp mode
network	(empty)	Docker network the replicas join; required when the master is containerized
image	slingdata/sling:<platform version>	Agent image override
cpu	2	CPU limit per replica container
memory_mb	3072	Memory limit (MB) per replica container
master_url	(derived)	URL replicas use to reach the platform
nats_url	(derived)	ws:// NATS URL override
restart_policy	unless-stopped	Restart policy for replica containers
labels	(empty)	Extra container labels (JSON object)

Docker socket security: mounting /var/run/docker.sock into the platform container gives it root-equivalent access to the host. For security-sensitive deployments, use rootless Docker or put a proxy like tecnativa/docker-socket-proxy in front of the socket (allow only container/image endpoints) and point the platform at it with SLING_DOCKER_RUNNERS='{"mode": "tcp", "host": "tcp://docker-socket-proxy:2375"}'.

Kubernetes-Managed Agents (HA)

When the platform runs in (or can reach) a Kubernetes cluster, it can manage agent replicas as a Deployment (plus a Secret for the key, and an optional HorizontalPodAutoscaler).

Set up in this order. The platform tests the Kubernetes API at startup, so apply the RBAC before you restart it.

1. Apply the RBAC

The platform needs permissions in the runner namespace (deployments, secrets, pods, horizontalpodautoscalers). Save the manifest below as runner-rbac.yaml and apply it:

kubectl apply -f runner-rbac.yaml

It creates the runner namespace, the service account, a Role and a RoleBinding.

The manifest puts the service account in namespace sling-platform. If your platform runs in a different namespace, change it in two places: the ServiceAccount metadata and the RoleBinding subject.

runner-rbac.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: sling-runners
---
# Service account used by the sling-platform master pod
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sling-platform
  namespace: sling-platform   # the namespace of your platform deployment
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sling-runner-manager
  namespace: sling-runners
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "update", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sling-runner-manager
  namespace: sling-runners
subjects:
  - kind: ServiceAccount
    name: sling-platform
    namespace: sling-platform   # the namespace of your platform deployment
roleRef:
  kind: Role
  name: sling-runner-manager
  apiGroup: rbac.authorization.k8s.io

To confirm the permissions are correct:

kubectl auth can-i create deployments --namespace sling-runners \
  --as system:serviceaccount:sling-platform:sling-platform

2. Configure the platform

Set the environment variable and the service account on the platform pod. Both are necessary — with mode: incluster the pod authenticates as its service account, and the default one has no permissions.

spec:
  serviceAccountName: sling-platform   # must match the ServiceAccount above
  containers:
  - name: sling-platform
    env:
    - name: SLING_K8S_RUNNERS
      value: "incluster"        # or "kubeconfig" when the platform runs outside the cluster

Replica pods must be able to reach the platform. The platform gives them SLING_PLATFORM_HOST when it is not localhost. An external name such as https://sling.example.com often does not resolve inside the cluster. When that applies, set master_url and nats_url to the internal service DNS:

{
  "mode": "incluster",
  "master_url": "http://sling-platform.<namespace>.svc.cluster.local:7878",
  "nats_url": "ws://sling-platform.<namespace>.svc.cluster.local:4443"
}

3. Restart the platform

kubectl rollout restart deployment/sling-platform -n sling-platform
kubectl rollout status  deployment/sling-platform -n sling-platform

The platform tests the Kubernetes API at startup. If it starts and stays up, the connection is good. If the API is unreachable or the RBAC is wrong, the pod stops with an error in its logs.

4. Create the agent

The configuration alone starts no pods. A Kubernetes hosting option now shows in the agent creation dialog, with:

• Desired replicas — the platform reconciles the Deployment to this count every 30 seconds.

• Autoscale (HPA) — optional; the platform creates an autoscaling/v2 HorizontalPodAutoscaler with your min/max replicas and target CPU %, and stops managing the replica count itself. Requires metrics-server in the cluster. Install it before you use this option.

Create an agent with Kubernetes hosting. The platform then creates the Deployment, the key Secret and the pods in the runner namespace. Each pod is named sling-agent-<short-agent-id>:

kubectl get pods -n sling-runners

Agents created this way are always shared-mode, so SLING_RUNNER_MODE is not necessary for them. Each pod gets its own emptyDir cache at /home/sling — you do not configure storage.

When you delete the agent, the platform removes the Deployment, the HorizontalPodAutoscaler and the Secret.

SLING_K8S_RUNNERS accepts a bare mode (incluster or kubeconfig) or a JSON object. All fields except mode:

SLING_K8S_RUNNERS fields

{
  "mode": "incluster",
  "kubeconfig": "/etc/sling/kubeconfig",
  "namespace": "sling-runners",
  "image": "slingdata/sling:latest",
  "service_account": "",
  "node_selector": { "pool": "workers" },
  "tolerations": [{ "key": "dedicated", "operator": "Equal", "value": "sling", "effect": "NoSchedule" }],
  "image_pull_secrets": ["regcred"],
  "resources": {
    "cpu_request": "500m", "cpu_limit": "2000m",
    "mem_request": "1Gi", "mem_limit": "3Gi"
  },
  "master_url": "",
  "nats_url": ""
}

Field	Default	Description
mode	(required)	incluster or kubeconfig
kubeconfig	$KUBECONFIG or ~/.kube/config	Kubeconfig path (kubeconfig mode)
namespace	sling-runners	Namespace for runner Deployments
image	slingdata/sling:<platform version>	Agent image override
resources.cpu_request / resources.cpu_limit	500m / 2000m	CPU per replica pod
resources.mem_request / resources.mem_limit	1Gi / 3Gi	Memory per replica pod
master_url	(derived)	URL replica pods use to reach the platform (defaults to the platform host or http://sling-platform.<ns>.svc.cluster.local:<port>)
nats_url	(derived)	ws:// NATS URL override
image_pull_secrets	(empty)	Image pull secret names (JSON array)
node_selector	(empty)	Pod nodeSelector (JSON object)
tolerations	(empty)	Pod tolerations (JSON array)
service_account	(empty)	Service account for the runner pods

With SLING_DOCKER_RUNNERS or SLING_K8S_RUNNERS set, the platform verifies connectivity to the Docker daemon / Kubernetes API at startup and fails fast with a clear error if unreachable.