Agents

What is an agent?

To put it simply, an agent is a container for your data pipelines. It runs in the background and waits for data jobs (such as replications). You can run an agent using Docker or by using the Sling CLI. An agent can run on any machine (Windows, Mac, Linux) and connects to the control server via a secure WebSocket connection (TLS). There is no need to open ports or configure firewalls.

Sling Platform Agent

How does the agent work?

Upon starting, the agent will connect to the control server, using a provided API Key, and will wait for jobs to be assigned to it. It will then execute the job (data will flow only through it) and then send the results back to the control server (such as logs, errors, number of rows processed, etc). The agent will continue to run and wait for new jobs to be assigned to it.

Running an Agent

You can simply use the sling agent command from the CLI or Docker.

$ sling agent
agent - Manage the local sling agent

See more details at https://docs.slingdata.io/sling-cli/

  Usage:
    agent [run]

  Subcommands: 
    run   run the sling agent

  Flags: 
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.

Before running it, you need to obtain the SLING_AGENT_KEY from the Sling Platform UI.

Once you have the key, you can start the agent by running sling agent run.

Linux

export SLING_AGENT_KEY='....'

sling agent run

Mac

export SLING_AGENT_KEY='....'

sling agent run

Windows

# using windows Powershell
$env:SLING_AGENT_KEY='....'

sling agent run

Docker CLI

export SLING_AGENT_KEY='....'

docker run -d -e SLING_AGENT_KEY slingdata/sling agent run

Docker Compose

services:
  sling-agent:
    image: slingdata/sling
    command: agent run
    container_name: sling-agent
    restart: unless-stopped
    environment:
      SLING_AGENT_KEY: '....'
    volumes:
      - agent_data:/home/sling

volumes:
  agent_data:

How do development and production agents differ?

A development agent is a special agent, with defined limits, that is used for development purposes. It is not billed, nor intended for production use, and therefore will not be selected for scheduled jobs. All plans include one development agent. On the other hand, a production agent will be selected for scheduled jobs, and requires a paid plan.

What about my Connection Credentials?

If you choose to self-host your Sling Agent, you can define a local env.yaml file to store your connection credentials (on the machine). Using this method, the sensitive credentials will never leave the agent. You can also store your credentials in the Sling Control Server, per project, which is securely transmitted over TLS and encrypted at rest.

Custom Agent Docker Image

If you need to build your own custom Sling agent Docker image (for example, to include additional dependencies or configurations), you can use the official Dockerfiles as a starting point:

• AMD64 Architecture: Dockerfile

• ARM64 Architecture: Dockerfile.arm64

Building a Custom Image

Here's an example of how to build a custom agent image with additional Python packages:

# Start from the official Sling image
FROM slingdata/sling:latest
# FROM slingdata/sling:latest-arm64 # for ARM64

# Switch to root to install additional packages
USER root

# Install additional system dependencies if needed
RUN apt-get update && apt-get install -y \
    your-package-here \
    && rm -rf /var/lib/apt/lists/*

# Install additional Python packages
RUN pip install --no-cache-dir \
    pandas \
    sqlalchemy \
    your-custom-package

# Switch back to sling user
USER sling

# The default command is already set to run the agent