# Incremental

In order to write to files incrementally, we need to provide an update\_key (which will be used as the partition key) to partition the files into a time resolution. Therefore, specific runtime variables are required to be part of the target object path.

Here are the list of [runtime variables](/concepts/replication/runtime-variables.md#partition-patterns) for partitioning the file chunks: `part_year`, `part_month`, `part_year_month`, `part_day`, `part_week`, `part_hour`, `part_minute`.

Additionally, we need to provide a environment variable called `SLING_STATE`, which is a location where sling will store the respective incremental values. See [Global Variables](/sling-cli/variables.md#global-environment-variables) for more details.

Here is an example of incrementally writing to an S3 bucket.

```yaml
source: postgres
target: aws_s3

# applies to all streams
defaults:
  object: tables/{stream_schema}/{stream_table}/{part_year}/{part_month}
  mode: incremental   # mode applies to all streams
  target_options:     # target_options applies to all streams
    format: parquet

streams:
  # all tables in schema main
  main.*:
    primary_key: id
    update_key: created_dt
    target_options:     # overwrites default target_options (write as csv)
      format: csv

  public.transactions:
    primary_key: tx_id
    update_key: created_dt

  public.orders:
    primary_key: order_id
    update_key: timestamp
    sql: |
      select *
      from public.orders
      where status not in ('voided')
        and {incremental_where_cond}

env:
  # uses the `path/to/folder` in the same AWS_S3 connection
  SLING_STATE: AWS_S3/path/to/folder
```

This will write data from tables `public.transactions` and `public.orders` (custom SQL) into the respective paths (such as `tables/public/transactions/2024/11/data_0.parquet` and `tables/public/orders/2024/11/data_0.parquet`) at the year and month level:

* `tables/public/transactions/created_dt_year=2024/created_dt_month=01/data_0.parquet`
* `tables/public/transactions/created_dt_year=2024/created_dt_month=02/data_0.parquet`
* ...
* `tables/public/orders/timestamp_year=2024/timestamp_month=01/data_0.parquet`
* `tables/public/orders/timestamp_year=2024/timestamp_month=01/data_0.parquet`
* ...

The first time the replication is ran, it will select all the data (unless limited), and write the last incremental value into the `SLING_STATE` location. This incremental value will be truncated to the lowest partition level. For example, if the lowest level is `part_day`, a value of `2024-11-04 04:05:06` will be truncated to `2024-11-04`. If the lowest level was `part_month`, the truncated value would be `2024-11-01`.

Once the replication is ran again, it will read this truncated incremental value from the `SLING_STATE`, and use it to obtain the complete partition so that no data is missed.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.slingdata.io/examples/database-to-file/incremental.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.