DuckDB

Connect & Ingest data from / to a DuckDB database

Setup

The following credentials keys are accepted:

• instance (required) -> The local file path of the database file

• schema (optional) -> The default schema to use.

• duckdb_version (optional) -> The CLI version of DuckDB to use. You can also specify the env. variable DUCKDB_VERSION.

• read_only (optional) -> Whether to open the connection in readonly mode. Accepts true or false. Default is false.

• interactive (optional) -> Whether to communicate to the DuckDB CLI via interactive mode instead of reopening the connection each time. Accepts true or false. Default is false.

• copy_method (optional) -> the method data is copied from sling into DuckDB. Acceptable values: csv_files, csv_http, arrow_http, named_pipes. Default is csv_http.

• max_buffer_size (optional v1.5) -> the max buffer size to use when piping data from DuckDB CLI. Specify this if you have extremely large one-line text values in your dataset. Default is 10485760 (10MB).

• use_adbc (optional) -> Enable Arrow Database Connectivity (ADBC) driver for high-performance data transfer. See ADBC for setup and details. (v1.5.2+)

• adbc_uri (optional) -> Override the automatically constructed ADBC connection URI when using use_adbc=true.

Using sling conns

Here are examples of setting a connection named DUCKDB. We must provide the type=duckdb property:

# for local files 
$ sling conns set DUCKDB type=duckdb instance=/path/to/file.db

# for local files (Windows), don't use backslash (\)
$ sling conns set DUCKDB type=duckdb instance=C:/path/to/file.db

# Or use url (only for local files)
$ sling conns set DUCKDB url="duckdb:///path/to/file.db"

# url for Windows, don't use backslash (\)
$ sling conns set DUCKDB url="duckdb://C:/path/to/file.db"

Environment Variable

export DUCKDB='duckdb:///path/to/file.db'
export DUCKDB='{ type: duckdb, instance: "/path/to/file.db" }'

$env:DUCKDB="duckdb://C:/path/to/file.db" # for Windows PowerShell

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  DUCKDB:
    type: duckdb
    instance: <instance>
    schema: <schema>
    duckdb_version: '<duckdb_version>'

Specifying a DuckDB version

By default, Sling will download the DuckDB binary automatically. If you would like to use a specific DuckDB version, you can specify this way, and sling will download that version:

export DUCKDB_VERSION='0.7.0'

Specifying a DuckDB Binary Path & Extensions

If you already have DuckDB on your machine and would like to use it (instead of having Sling download it), you can specify the binary path this way:

export DUCKDB_PATH='/opt/homebrew/bin/duckdb'
export DUCKDB_USE_INSTALLED_EXTENSIONS='true'  # tells sling not to download extensions

If you are facing issues connecting, please reach out to us at [email protected], on discord or open a Github Issue here.

Potential Issue

If you have a .duckdbrc file, which runs commands whenever the DuckDB CLI is invoked, this may interfere with normal Sling operation. If you are facing weird issues and have this file, try again after deleting it.