> For the complete documentation index, see [llms.txt](https://docs.slingdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.slingdata.io/concepts/change-capture/continuous-mode.md). # Continuous Mode Run a CDC replication as a long-lived process that applies changes in cycles Continuous mode runs a [Change Capture (CDC)](/concepts/change-capture.md) replication as one long-lived process. The process starts one time. Then it runs cycles until you stop it. You do not need a scheduler. A cycle reads the new changes from the transaction log, writes them to the target, and saves the position. A cycle is the same work as one CDC run, without the start-up cost. {% hint style="info" %} Continuous mode is available in v1.6.8+. It needs the same plan as CDC: a [CLI Pro Max](/sling-cli/cli-pro.md) token or an [Advanced Platform Plan](/sling-platform/platform.md). Platform jobs and UI for continuous mode come in a later release. {% endhint %} ## When to Use It Use continuous mode when you need low lag and you can keep one process alive. Use a scheduled one-shot run when a batch each hour or each day is enough. | | One-shot run | Continuous mode | | ------------- | ----------------------------- | ------------------------------------------ | | Process | Starts and exits for each run | Starts one time, runs until stopped | | Lag | Depends on the schedule | About the `interval` (1 minute by default) | | Source reader | Opens in each run | Opens in each cycle | | Scheduler | Needed | Not needed | ## Quick Start ```yaml # replication.yaml source: MY_MYSQL target: MY_POSTGRES defaults: mode: change-capture object: public.{stream_table} continuous: interval: 30s streams: my_database.customers: my_database.orders: env: SLING_STATE: MY_POSTGRES/sling_state ``` ```bash sling run -r replication.yaml --continuous ``` The same file also works as a one-shot run. Run it without the flag. ```bash sling run -r replication.yaml ``` ## Start and Stop ### Start Use one of these methods: ```bash sling run -r replication.yaml --continuous # or with the environment variable SLING_CONTINUOUS=true sling run -r replication.yaml ``` Sling reads `SLING_CONTINUOUS` one time at start. Then it removes the variable from the environment. Child processes of hooks do not inherit it, so they run one time. Continuous mode needs a replication file. If you use `--continuous` without `-r`, Sling gives this error: ``` --continuous needs a replication file (-r) ``` Sling reads the replication file one time at start. To apply a change to the file, stop the process and start it again. `SLING_TIMEOUT` and `SLING_RETRIES` do not apply to continuous mode. Sling prints a warning if you set them. Use `retry_attempts` and `retry_delay` in `change_capture_options` for retries. ### Stop | Signal | Effect | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | First `Ctrl-C` (SIGINT) or SIGTERM | Sling finishes the cycle in progress, saves the state, and exits with code `0`. | | Second `Ctrl-C` or SIGTERM | Sling stops at once. The exit code is `11` for SIGINT and `111` for SIGTERM. A lease can stay until its time to live ends (see [Lease](#lease)). | On the first signal, Sling prints this line: ``` stop received: finishing running cycles, then exit (Ctrl-C again to stop now) ``` An error that stops the run gives exit code `1`. ### Print the Status Send `SIGUSR1` to the process to print the status table. On macOS, press `Ctrl-T` (SIGINFO) in the terminal. Windows has no status signal. ```bash kill -USR1 ``` Sling also prints the status table each `status_interval`. ## The `continuous` Block The `continuous` block is a root-level key of the replication file. All keys are optional. ```yaml continuous: interval: 1m max_rows: 100000 max_bytes: 0 min_interval: auto status_interval: 60s behind_after: 5m max_session: 24h on_stream_error: stop ``` | Key | Default | Description | | ----------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `interval` | `1m` | Time between the starts of two cycles. Must be greater than `0`. | | `max_rows` | `run_max_events` | Maximum rows for one cycle. When a cycle reaches it, the next cycle starts at once. `0` means not set. | | `max_bytes` | `0` | Maximum bytes for one cycle. `0` means no limit. Sling counts the bytes it writes to the cache files. | | `min_interval` | `auto` | Minimum time between two writes to the target. `0s` means no minimum. See [min\_interval](#min_interval-and-warehouse-cost). | | `status_interval` | `60s` | Time between two status tables in the log. Must be greater than `0`. | | `behind_after` | `5m` | A stream is `behind` when its lag reaches this value. `0` turns off the lag check. | | `max_session` | `24h` | Maximum age of a session. Then Sling opens the session again inside the same process. Must be greater than `0`. | | `on_stream_error` | `stop` | What to do when a stream fails after all retries. Only `stop` is available. | Durations use the Go format: `500ms`, `30s`, `5m`, `1h30m`. The format has no day unit. Use `24h` and not `1d`. A negative value gives an error at start. ### Overlap with `change_capture_options` Continuous mode replaces two [CDC options](/concepts/change-capture.md#options-reference): | CDC option | In continuous mode | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `run_max_events` | Sets the default of `max_rows`. If streams have different values, Sling uses the highest. If `max_rows` and `run_max_events` are both not set, the default is `100000`. | | `run_max_duration` | Sling ignores it. `interval` replaces it. Sling prints a warning if you set both. | | `retry_attempts`, `retry_delay` | Apply as before. See [Errors and Retries](#errors-and-retries). | All other CDC options, such as `soft_delete` and the `snapshot_*` options, work as before. ### `max_bytes` `max_bytes` applies only to MySQL, MariaDB, and PostgreSQL groups that use a shared reader. For other sources, Sling prints a warning and ignores it. A cycle can write a little more than the limit before it stops. ### `behind_after` If `behind_after` is not greater than `interval`, streams show `behind` between cycles. Sling prints a warning at start. ### `min_interval` and Warehouse Cost `min_interval` is the shortest time between two writes that contain rows. Use it to limit the cost on targets that charge for each write. With `auto`, Sling picks the value from the target type: | Target | `auto` value | | ---------- | ------------ | | Snowflake | `1m` | | BigQuery | `5m` | | All others | `0s` | On Snowflake and BigQuery, each write starts compute or a load job. A short `min_interval` keeps the warehouse active and raises the cost. To override `auto`, set a duration, for example `min_interval: 10m`. ## How a Cycle Works A cycle starts each `interval`. The cycle reads until it reaches the head of the source log that it saw at the start of the cycle, or until it reaches `max_rows` or `max_bytes`. It does not wait for new events. The reader always reads whole transactions. Then Sling writes the changes to the target and saves the position. * **Normal cycle.** The next cycle starts one `interval` after the start of this cycle. * **Catch-up.** If a cycle stops at `max_rows` or `max_bytes`, the next cycle starts at once. This lets a stream with a large backlog catch up. * **Minimum time.** The next cycle never starts before `min_interval` has passed since the last write that contained rows. The reader opens again in each cycle. Sling runs the cycles of one replication one after the other. ### Initial Load Continuous mode uses the same [snapshot options](/concepts/change-capture.md#options-reference) as a one-shot run. A new stream makes its snapshot first. Then it joins the group and receives changes. The health of the stream is `loading` until the snapshot ends. ## Log Output When the process starts, Sling prints one line: ``` INF continuous run started: 1 cdc group (2 streams), interval 1m / 100k rows ``` ### Cycle Lines A cycle that writes rows prints one line. A cycle with no rows prints nothing. ``` INF [cdc] cycle #4 | 2 streams | +3 ~1 -1 | 1.1s | lag 1.1s | mysql-bin.000034:70630477 INF [cdc] cycle #5 | 2 streams | +600,000 ~0 -0 | 6.6s | lag 9.6s | mysql-bin.000034:76666406 (max_rows, next cycle now) ``` | Field | Meaning | | ---------------------------- | ------------------------------------------------------------- | | `cycle #N` | The number of the cycle in this process. | | `+3 ~1 -1` | Inserts, updates, and deletes. | | `1.1s` | Time to read and write. | | `lag` | The lag of the streams (see [Health States](#health-states)). | | Last field | The source position after the cycle. | | `(max_rows, next cycle now)` | The cycle stopped at a limit. The next cycle starts at once. | Streams that use their own reader (SQL Server, Oracle, MongoDB, or `slot_level: stream`) print `N rows` in place of `+I ~U -D`. They print no position. The per-stream banners and task progress lines print at debug level. Use `--debug` to see them. ### Status Table ``` INF ── continuous status · up 11s · 3 cycles · 0 errors ── STREAM MODE HEALTH LAG LAST CYCLE ROWS POSITION test_db.cont_a cdc idle 5.0s 4.9s ago 0 mysql-bin.000034:70629216 ``` `ROWS` is the number of rows that the stream wrote in this process. ### Health States | State | Meaning | | --------- | ------------------------------------------------------------------------------ | | `loading` | The stream makes its initial snapshot. | | `failing` | The stream used all retries and still fails. | | `behind` | A retry is in progress, or the lag is equal to or greater than `behind_after`. | | `idle` | The last cycle reached the head of the log and had 0 rows. | | `healthy` | None of the above. | The lag is the time since the start of the last cycle that read up to the head of the log. It is about one `interval` plus the cycle time. Sling prints a line when a stream moves to or from `behind` or `failing`: ``` WRN [cdc] test_db.cont_a idle -> behind (retry in progress) WRN [cdc] test_db.cont_a behind -> failing (retries used up) ``` ### Session and Stop Lines ``` INF session restart (max_session 24h) INF continuous run stopped · 8 cycles · +3 ~1 -1 · state saved WRN continuous run stopped by an error · 5 cycles · +2 ~0 -1: ERROR: ... ``` ## Errors and Retries A read error or a write error retries the cycle. Sling uses `retry_attempts` (default `3`) and `retry_delay` (default `5s`) from `change_capture_options`. The delay doubles with each retry. If streams have different values, Sling uses the highest. ``` WRN [cdc] cycle #5 write failed for test_db.cont_a (retry 1/3 in 5s): ERROR: ... ``` During a retry: * The position does not advance. * Streams that succeeded in the cycle are not written again. * The stream health is `behind`. After the last retry, `on_stream_error: stop` applies. Sling stops the process and exits with code `1`. The end hooks still run. If you stop the process during the wait for a retry of a failed write, Sling exits with an error. The position of the failed streams does not advance. The next session reads their changes again. `on_stream_error: quarantine` is not available. Sling gives this error at start: ``` on_stream_error: quarantine is not supported yet ``` ## Lease A continuous session holds one lease for each stream. The lease is a record in the state store (`SLING_STATE`), next to the CDC state. It has a time to live (TTL) of 60 seconds. The session renews it every 20 seconds. * **Second session.** A second continuous session for the same streams fails at start: ``` stream X is held by another continuous session (host=... pid=... exec_id=..., expires in Ns) ``` * **One-shot run.** A one-shot `sling run` of a CDC stream that a live session holds fails: ``` stream X is held by a continuous session (host=... pid=... exec_id=...). Stop that session first ``` * **Lost lease.** If a session loses its lease, it exits with code `1` and does not commit. ### After a Crash If a session ends without a clean stop, its lease stays until the TTL ends. * If the new session runs on the same host and in the same PID namespace, Sling sees that the old process is gone. The new session takes the lease at once. * In all other cases, for example a new container, the new session waits up to 60 seconds. To clear a stale lease before the TTL ends, make sure that the old process is gone. Then delete the lease record from the state store. For a database state store, run this statement. Replace the schema and the stream name: ```sql DELETE FROM sling_state._sling_state WHERE column_type = 'lease' AND source_stream = '_sling_lease/my_database.orders'; ``` ## Hooks * Replication-level `start` and `end` hooks run one time for each process. They do not run again when a `max_session` restart opens a new session. * The `end` hooks run after the stop, also when an error stops the run. * Stream-level `pre` and `post` hooks are not available. If a stream has them, Sling gives an error at start. * A [`replication` hook](/concepts/hooks/replication.md) runs the nested replication as a one-shot run. ## Limits in 1.6.7 * Only `change-capture` streams. For a stream with a different mode, Sling gives this error: ``` continuous mode supports only change-capture streams in this version (stream X has mode Y) ``` * The target must be a database. * No quarantine (`on_stream_error: quarantine`). * No stream-level `pre` and `post` hooks. * The source reader opens again in each cycle. * `max_bytes` applies only to MySQL, MariaDB, and PostgreSQL groups. ## Run as a Service ### systemd `/etc/systemd/system/sling-continuous.service`: ```ini [Unit] Description=Sling continuous replication After=network-online.target Wants=network-online.target [Service] Type=simple ExecStart=/usr/local/bin/sling run -r /etc/sling/replication.yaml --continuous Restart=on-failure RestartSec=10 TimeoutStopSec=300 [Install] WantedBy=multi-user.target ``` * systemd sends SIGTERM to stop the service. Sling finishes the cycle in progress and exits with code `0`. * Set `TimeoutStopSec` to a value greater than the longest cycle. If the value is too small, systemd sends SIGKILL and the lease stays until its TTL ends. * `Restart=on-failure` restarts the process after an error (exit code `1`). A restart on the same host takes the old lease at once. ```bash sudo systemctl daemon-reload sudo systemctl enable --now sling-continuous.service journalctl -u sling-continuous.service -f ``` ### Docker Docker sends SIGTERM when you stop a container. Set `stop_grace_period` to a value greater than the longest cycle. The default is 10 seconds. ```yaml services: sling: image: slingdata/sling command: run -r /config/replication.yaml --continuous restart: on-failure stop_grace_period: 5m volumes: - ./replication.yaml:/config/replication.yaml:ro ``` A new container has a new PID namespace. After a crash, the new container can wait up to 60 seconds for the old lease to expire. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.slingdata.io/concepts/change-capture/continuous-mode.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.