ADBC (Arrow)

Connect & Ingest data from / to multiple databases via Arrow Database Connectivity (ADBC)

Arrow Database Connectivity (ADBC) provides a standardized interface for accessing various databases using the Apache Arrow columnar format. ADBC enables efficient, high-performance data movement with zero-copy semantics and native Arrow support (v1.5.2+)

Supported Database Types

Sling supports the following databases via ADBC drivers:

• PostgreSQL - Full support via ADBC PostgreSQL driver

• MySQL - Support via ADBC MySQL driver (v9.4+)

• SQL Server - Full support via ADBC SQL Server driver

• Snowflake - Full support via ADBC Snowflake driver

• SQLite - Support via ADBC SQLite driver

• DuckDB - Full support via ADBC DuckDB driver

• BigQuery - Full support via ADBC BigQuery driver

• Trino - Support via ADBC Trino driver (v4.0+)

Setup

Before using ADBC connections, you need to install the appropriate ADBC driver using the dbc CLI tool (see details here):

# Install drivers for your target databases
dbc install postgresql
dbc install mysql
dbc install trino
dbc install snowflake
# ... etc

Drivers are automatically installed to:

• macOS: ~/Library/Application Support/ADBC/Drivers

• Linux: ~/.local/share/ADBC/Drivers

• Windows: %LOCALAPPDATA%\ADBC\Drivers

Sling will automatically discover installed drivers from these locations.

ADBC Driver Manager Installation

If your system does not have the ADBC Driver Manager library installed, you may need to install it manually and set the ADBC_DRIVER_MANAGER_LIB environment variable to point to the library file.

macOS

Installation

Using Conda (Recommended):

conda install conda-forge::adbc-driver-manager

Using pip:

pip install adbc-driver-manager

Finding the Library

• Conda path: $CONDA_PREFIX/lib/libadbc_driver_manager.dylib

• Pip path: Inside your site-packages/adbc_driver_manager/ directory

• System search: mdfind -name libadbc_driver_manager.dylib

Setting the Environment Variable

export ADBC_DRIVER_MANAGER_LIB="$CONDA_PREFIX/lib/libadbc_driver_manager.dylib"

Linux

Windows

Enabling ADBC on Connections

To use ADBC with your existing database connections, simply add use_adbc: true to your connection configuration. This allows you to keep your original connection properties and format while enabling ADBC's high-performance data transfer.

Connection Properties

The following ADBC-specific properties can be added to any supported database connection:

• use_adbc (optional) -> Enable ADBC driver for this connection (true or false). Default is false.

• adbc_uri (optional) -> Override the automatically constructed ADBC URI. Sling automatically builds the ADBC URI from your connection properties, but you can specify a custom URI if needed.

Database-Specific Examples

PostgreSQL with ADBC

connections:
  POSTGRES:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    port: 5432
    use_adbc: true
    # adbc_uri: "postgresql://myuser:mypass@localhost:5432/mydatabase"  # optional override

ADBC URI format: postgresql://user:password@host:port/database

Official Documentation: Apache ADBC PostgreSQL Driver

MySQL with ADBC

connections:
  MYSQL:
    type: mysql
    host: localhost
    user: root
    password: mypass
    database: mydb
    port: 3306
    use_adbc: true
    # adbc_uri: "root@tcp(localhost:3306)/mydb"  # optional override

ADBC URI format: user@tcp(host:port)/database

Official Documentation: Apache ADBC MySQL Driver

SQL Server with ADBC

connections:
  MSSQL:
    type: sqlserver
    host: localhost
    user: sa
    password: mypass
    database: master
    port: 1433
    use_adbc: true
    # adbc_uri: "mssql://sa:mypass@localhost:1433/master"  # optional override

ADBC URI format: mssql://user:password@host:port/database

Official Documentation: Apache ADBC SQL Server Driver

Snowflake with ADBC

connections:
  SNOWFLAKE:
    type: snowflake
    account: myaccount.us-east-1
    user: myuser
    password: mypass
    database: mydb
    schema: myschema
    use_adbc: true
    # adbc_uri: "snowflake://myuser:mypass@myaccount/mydb/myschema"  # optional override

ADBC URI format: snowflake://user:password@account/database/schema

Official Documentation: Apache ADBC Snowflake Driver

DuckDB with ADBC

connections:
  DUCKDB:
    type: duckdb
    instance: /path/to/file.db
    use_adbc: true
    # adbc_uri: "duckdb:///path/to/file.db"  # optional override

ADBC URI format: duckdb:///path/to/file.db or duckdb://:memory:

Official Documentation: Apache ADBC DuckDB Driver

SQLite with ADBC

connections:
  SQLITE:
    type: sqlite
    instance: /path/to/file.db
    use_adbc: true
    # adbc_uri: "sqlite:///path/to/file.db"  # optional override

ADBC URI format: sqlite:///path/to/file.db

Official Documentation: Apache ADBC SQLite Driver

BigQuery with ADBC

connections:
  BIGQUERY:
    type: bigquery
    project: myproject
    dataset: mydataset
    key_file: /path/to/service.account.json
    use_adbc: true
    # adbc_uri: "bigquery://myproject"  # optional override

ADBC URI format: bigquery://project

Official Documentation: Apache ADBC BigQuery Driver

Trino with ADBC

connections:
  TRINO:
    type: trino
    http_url: "http://myuser@localhost:8080?catalog=hive&schema=default"
    use_adbc: true
    # adbc_uri: "http://myuser@localhost:8080?catalog=hive&schema=default"  # optional override

ADBC URI format: http://user@host:port?catalog=catalog&schema=schema

Official Documentation: Apache ADBC Trino Driver

Using sling conns

Here are examples of enabling ADBC on existing connections:

# PostgreSQL with ADBC
$ sling conns set POSTGRES type=postgres host=localhost user=myuser password=mypass database=mydb use_adbc=true

# MySQL with ADBC
$ sling conns set MYSQL type=mysql host=localhost user=root password=mypass database=mydb use_adbc=true

# SQL Server with ADBC
$ sling conns set MSSQL type=sqlserver host=localhost user=sa password=mypass database=master use_adbc=true

# Snowflake with ADBC
$ sling conns set SNOWFLAKE type=snowflake account=myaccount user=myuser password=mypass database=mydb use_adbc=true

# With custom ADBC URI override
$ sling conns set POSTGRES type=postgres host=localhost user=myuser password=mypass database=mydb use_adbc=true adbc_uri="postgresql://myuser:mypass@localhost:5432/mydb"

Environment Variable

export POSTGRES='{ type: postgres, host: localhost, user: myuser, password: mypass, database: mydb, use_adbc: true }'

export MYSQL='{ type: mysql, host: localhost, user: root, password: mypass, database: mydb, use_adbc: true }'

export SNOWFLAKE='{ type: snowflake, account: myaccount, user: myuser, password: mypass, database: mydb, use_adbc: true }'

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  # PostgreSQL with ADBC
  POSTGRES:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    use_adbc: true

  # MySQL with ADBC
  MYSQL:
    type: mysql
    host: localhost
    user: root
    password: mypass
    database: mydb
    use_adbc: true

  # SQL Server with ADBC
  MSSQL:
    type: sqlserver
    host: localhost
    user: sa
    password: mypass
    database: master
    use_adbc: true

  # Snowflake with ADBC
  SNOWFLAKE:
    type: snowflake
    account: myaccount
    user: myuser
    password: mypass
    database: mydb
    schema: myschema
    use_adbc: true

  # DuckDB with ADBC
  DUCKDB:
    type: duckdb
    instance: /path/to/file.db
    use_adbc: true

  # With custom ADBC URI override
  POSTGRES_CUSTOM:
    type: postgres
    host: localhost
    user: myuser
    password: mypass
    database: mydatabase
    use_adbc: true
    adbc_uri: "postgresql://myuser:mypass@localhost:5432/mydatabase?sslmode=require"

Additional Resources

• Apache ADBC Project

• ADBC Driver Documentation

• Apache Arrow Documentation

If you are facing issues connecting, please reach out to us at [email protected], on discord or open a Github Issue here.