> For the complete documentation index, see [llms.txt](https://docs.slingdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.slingdata.io/connections/datalake-connections/ducklake.md). # DuckLake Connect & Ingest data from / to a DuckLake database DuckLake is a data lake format specification that combines the power of DuckDB with flexible catalog backends and scalable data storage. It provides versioned, ACID-compliant tables with support for multiple catalog databases and various storage backends. See for more details. ## Setup The following credentials keys are accepted: * `catalog_type` **(required)** -> The catalog database type: `duckdb`, `sqlite`, `postgres`, or `mysql`. Default is `duckdb`. * `catalog_conn_string` **(required)** -> The connection string for the catalog database. See [here](https://ducklake.select/docs/stable/duckdb/usage/choosing_a_catalog_database) for details. Examples: * postgres: `dbname=ducklake_catalog host=localhost` * sqlite: `metadata.sqlite` * duckdb: `metadata.ducklake` * mysql: `db=ducklake_catalog host=localhost` * `catalog_schema` (optional) -> The schema to use to store catalog tables (e.g. `public`). * `data_path` (optional) -> Path to data files (local, S3, Azure, GCS). e.g. `/local/path`, `s3://bucket/data`, `r2://bucket/data`, `az://container/data`, `gs://bucket/data`. * `data_inlining_limit` (optional) -> The row limit to use for [`DATA_INLINING_ROW_LIMIT`](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining.html). * `copy_format` (optional *v1.6.4+*) -> the data format that Sling uses to move data between Sling and DuckDB, for reads and writes. Acceptable values: `arrow`, `csv`. Default is `arrow`. Arrow needs the DuckDB `arrow` community extension. If DuckDB cannot load this extension (for example, with no internet access), Sling uses `csv`. **If you have issues with arrow, set `csv`**. * `copy_method` (deprecated) -> use `copy_format`. `arrow_http` becomes `arrow`. All other values become `csv`. * `schema` (optional) -> The default schema to use to read/write data. Default is `main`. * `url_style` (optional) -> specify `path` to use path url style (e.g. when using MinIO) * `use_ssl` (optional) -> specify `false` to not use HTTPS (e.g. when using MinIO) * `duckdb_version` (optional) -> The CLI version of DuckDB to use. You can also specify the env. variable `DUCKDB_VERSION`. * `max_buffer_size` (optional *v1.5*) -> the max buffer size to use when piping data from DuckDB CLI. Specify this if you have extremely large one-line text values in your dataset. Default is `10485760` (10MB). * `max_line_size` (optional *v1.5.25+*) -> the max line size (in bytes) to use when piping CSV data into DuckDB. Sling raises this to `268435456` (256MB) when the data has string, text, json or binary columns, and uses `2000000` (2MB) otherwise. Specify this only if a single row is larger than 256MB. Applies when `copy_format` is `csv`. ### Storage Configuration For **S3/S3-compatible storage**: * `s3_access_key_id` (optional) -> AWS access key ID * `s3_secret_access_key` (optional) -> AWS secret access key * `s3_session_token` (optional) -> AWS session token * `s3_region` (optional) -> AWS region * `s3_profile` (optional) -> AWS profile to use * `s3_endpoint` (optional) -> S3-compatible endpoint URL (e.g. `localhost:9000` for MinIO) For **Azure Blob / ADLS Storage**: DuckDB's azure secret only accepts `CONNECTION_STRING`, `ACCOUNT_NAME`, proxy keys, and provider-specific fields — **not** a raw `ACCOUNT_KEY` or `SAS_TOKEN`. Sling accepts the keys below and translates them into a valid DuckDB secret. * `azure_connection_string` (optional, preferred) -> Full Azure connection string (e.g. `DefaultEndpointsProtocol=https;AccountName=…;AccountKey=…;EndpointSuffix=core.windows.net`). Passed through to DuckDB as-is. * `azure_account_name` (optional) -> Azure storage account name. Used with `azure_account_key` / `azure_sas_token`, or alone with credential\_chain / managed identity. * `azure_account_key` (optional) -> Azure storage account key. Sling builds a `CONNECTION_STRING` from `azure_account_name` + this key (DuckDB rejects `ACCOUNT_KEY` as a secret parameter). * `azure_sas_token` (optional) -> Azure SAS token. Sling builds a `CONNECTION_STRING` with `SharedAccessSignature=…` (DuckDB rejects `SAS_TOKEN` as a secret parameter). Prefer `azure_connection_string` when you already have one. * `azure_tenant_id` / `azure_client_id` / `azure_client_secret` (optional) -> Service principal. Sling sets `PROVIDER service_principal` automatically. * `azure_client_id` alone (optional) -> User-assigned managed identity. Sling sets `PROVIDER managed_identity` automatically. * `client_certificate_path` (optional) -> Path to a PEM/PKCS12 certificate for service principal auth (also forces `PROVIDER service_principal`). For **Google Cloud Storage**: * `gcs_key_file` (optional) -> Path to GCS service account key file * `gcs_project_id` (optional) -> GCS project ID * `gcs_access_key_id` (optional) -> GCS HMAC access key ID * `gcs_secret_access_key` (optional) -> GCS HMAC secret access key ### Using `sling conns` Here are examples of setting a connection named `DUCKLAKE`. We must provide the `type=ducklake` property: {% code overflow="wrap" %} ```bash # Default DuckDB catalog with local data $ sling conns set DUCKLAKE type=ducklake catalog_conn_string=metadata.ducklake # DuckDB catalog with S3 data storage $ sling conns set DUCKLAKE type=ducklake catalog_conn_string=metadata.ducklake data_path=data # SQLite catalog for multi-client access $ sling conns set DUCKLAKE type=ducklake catalog_type=sqlite catalog_conn_string=metadata.sqlite # PostgreSQL catalog for multi-user access $ sling conns set DUCKLAKE type=ducklake catalog_type=postgres catalog_conn_string="host=localhost port=5432 user=myuser password=mypass dbname=ducklake_catalog" # MySQL catalog with Azure storage $ sling conns set DUCKLAKE type=ducklake catalog_type=mysql catalog_conn_string="host=localhost user=root password=pass db=ducklake_catalog" data_path=data ``` {% endcode %} ### Environment Variable See [here](https://docs.slingdata.io/connections/datalake-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file. {% code overflow="wrap" %} ```bash # configuration with s3 properties export DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "s3://bucket/path/to/data/folder/", s3_access_key_id: "AKIAIOSFODNN7EXAMPLE", s3_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", s3_region: "us-east-1" }' # configuration with Non-HTTPS MinIO properties export DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "s3://bucket/path/to/data/folder/", s3_endpoint: machine:9000, s3_access_key_id: "AKIAIOSFODNN7EXAMPLE", s3_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", s3_region: "minio", url_style: "path", use_ssl: false }' # configuration with r2 properties export DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "r2://bucket/path/to/data/folder/", s3_access_key_id: "AKIAIOSFODNN7EXAMPLE", s3_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", s3_endpoint: ".r2.cloudflarestorage.com" }' # Windows PowerShell — Azure with account key (Sling builds CONNECTION_STRING for DuckDB) $env:DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "az://my_container/path/to/data/folder/", azure_account_name: "", azure_account_key: "", }' # Preferred: pass a full Azure connection string export DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "abfs://ducklake@myaccount.dfs.core.windows.net/lake/", azure_connection_string: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=;EndpointSuffix=core.windows.net" }' # Service principal (Sling sets PROVIDER service_principal for DuckDB) export DUCKLAKE='{ type: ducklake, catalog_type: postgres, catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog", data_path: "az://my_container/path/to/data/folder/", azure_account_name: "", azure_tenant_id: "", azure_client_id: "", azure_client_secret: "" }' ``` {% endcode %} ### Sling Env File YAML See [here](https://docs.slingdata.io/connections/datalake-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file. ```yaml connections: DUCKLAKE: type: ducklake catalog_type: catalog_conn_string: data_path: database: schema: # S3 Configuration (if using S3 storage) s3_access_key_id: s3_secret_access_key: s3_region: s3_endpoint: http://localhost:9000 # Azure Configuration (if using Azure storage) # Prefer azure_connection_string. azure_account_key / azure_sas_token are # accepted by Sling and converted to a DuckDB CONNECTION_STRING internally. azure_connection_string: azure_account_name: azure_account_key: # Or service principal: # azure_tenant_id: # azure_client_id: # azure_client_secret: # GCS Configuration (if using GCS storage) gcs_key_file: gcs_project_id: ``` ## Common Usage Examples ### Basic Operations ```bash # List tables sling conns discover DUCKLAKE # Query data sling run --src-conn DUCKLAKE --src-stream "SELECT * FROM sales.orders LIMIT 10" --stdout # Export to CSV sling run --src-conn DUCKLAKE --src-stream sales.orders --tgt-object file://./orders.csv ``` ### Data Import/Export ```bash # Import CSV to DuckLake sling run --src-stream file://./data.csv --tgt-conn DUCKLAKE --tgt-object sales.new_data # Import from PostgreSQL sling run --src-conn POSTGRES_DB --src-stream public.customers --tgt-conn DUCKLAKE --tgt-object sales.customers # Export to Parquet files sling run --src-conn DUCKLAKE --src-stream sales.orders --tgt-conn AWS_S3 --tgt-object s3://bucket/exports/orders.parquet ``` ## Geometry Data Sling types PostGIS `geometry`, MySQL and MariaDB spatial columns, and BigQuery `geography` as `geometry`. When you load into a DuckLake table, Sling creates the column with the native `GEOMETRY` type. The `spatial` extension loads automatically. Available from *v1.5.27*. If the target table already exists and the column is `VARCHAR`, Sling keeps that type. The value then stays WKB hex text. Convert it after the load: ```sql select st_geomfromwkb(unhex(location)) as location from my_table; ``` Export to Parquet files writes these columns as native `GEOMETRY` with GeoParquet `geo` metadata as well. See the [DuckDB Geometry Data](/connections/database-connections/duckdb.md#geometry-data) section for details. If you are facing issues connecting, give [`sling assist`](/sling-cli/ai/assist.md) a try, or reach out to us at , on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.slingdata.io/connections/datalake-connections/ducklake.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `build a script that syncs our docs to a CMS` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.