> For the complete documentation index, see [llms.txt](https://docs.slingdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.slingdata.io/connections/database-connections/bigquery.md).

# BigQuery

## Setup

The following credentials keys are accepted:

* `project` **(required)** -> The GCP project ID for the project
* `dataset` **(required)** -> The default dataset (like a schema)
* `gc_bucket` (optional) -> The Google Cloud Storage Bucket to use for loading (Recommended)
* `key_file` (optional) -> The path of the Service Account JSON. If not provided, the Google [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) will be used.
* `key_body` (optional) -> The Service Account JSON key content as a string. You can also provide the JSON content in env var `GC_KEY_BODY`.
* `location` (optional) -> The location of the account, such as `US` or `EU`. Default is `US`.
* `extra_scopes` (optional) -> An array of strings, which represent scopes to use in addition to `https://www.googleapis.com/auth/bigquery`. e.g. `["https://www.googleapis.com/auth/drive", "https://www.googleapis.com/auth/spreadsheets"]`
* `use_adbc` (optional) -> Enable Arrow Database Connectivity (ADBC) driver for high-performance data transfer. See [ADBC](/connections/database-connections/adbc.md) for setup and details. (*v1.5.2+*)
* `adbc_uri` (optional) -> Override the automatically constructed ADBC connection URI when using `use_adbc=true`.

{% hint style="warning" %}
If you'd like to have sling use the machine's Google Cloud Application Default Credentials (usually with `cloud auth application-default login`), don't specify a `key_file` (or the env var `GC_KEY_BODY`).
{% endhint %}

### Using `sling conns`

Here are examples of setting a connection named `BIGQUERY`. We must provide the `type=bigquery` property:

{% code overflow="wrap" %}

```bash
$ sling conns set BIGQUERY type=bigquery project=<project> dataset=<dataset> gc_bucket=<gc_bucket> key_file=/path/to/service.account.json location=<location>
```

{% endcode %}

### Environment Variable

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#dot-env-file-.env.sling) to learn more about the `.env.sling` file.

{% code overflow="wrap" %}

```bash
export BIGQUERY='{type: bigquery, project: my-google-project, gc_bucket: my_gc_bucket, dataset: public, location: US, key_file: /path/to/service.account.json}'
```

{% endcode %}

You can also provide Sling the Service Account JSON in `key_body` as a string, or via environment variable `GC_KEY_BODY`, instead of a `key_file`.

{% code overflow="wrap" %}

```bash
export GC_KEY_BODY='{"type": "service_account","project_id": ...........}'
```

{% endcode %}

### Sling Env File YAML

See [here](https://docs.slingdata.io/connections/database-connections/pages/eAdVs2BHCgdr6RS8GoJC#sling-env-file-env.yaml) to learn more about the sling `env.yaml` file.

```yaml
connections:
  BIGQUERY:
    type: bigquery
    project: <project>
    dataset: <dataset>
    gc_bucket: <gc_bucket>
    key_file: '<key_file>'

  # using with `key_body` instead of `key_file`
  BIGQUERY:
    type: bigquery
    project: <project>
    dataset: <dataset>
    gc_bucket: <gc_bucket>
    key_body: |
      { "type": "service_account", ... } 
```

### BigQuery Table Partitioning

```yaml
streams:
  my_schema.another_table:
    object: my_dataset.{stream_table}
    target_options:
      table_keys:
        partition: [ DATE_TRUNC(transaction_date, MONTH) ]

# OR
streams:
  my_schema.another_table:
    object: my_dataset.{stream_table}
    target_options:
      table_ddl: |
         CREATE TABLE my_dataset.{stream_table} ({col_types}) 
          PARTITION BY
            DATE_TRUNC(transaction_date, MONTH)
            OPTIONS (
              partition_expiration_days = 3,
              require_partition_filter = TRUE)
```

## BigQuery as a source

BigQuery is a standard SQL source: Sling reads it with a `SELECT` statement, so full-refresh, incremental, and backfill all work.

### Partitioned tables

A plain full-refresh reads a partitioned table with `SELECT *` — partitioning by itself does not block the read.

If the source table sets **`require_partition_filter = TRUE`**, BigQuery rejects any query that does not filter on the partition column. Sling does not detect this option or inject a partition predicate automatically, so a plain `SELECT *` fails. Add a `where` source option (or `--where` flag) that filters the partition column to satisfy the requirement:

```yaml
source: BIGQUERY
target: MY_SNOWFLAKE

streams:
  analytics.events:
    mode: full-refresh
    source_options:
      where: "event_date >= '2024-01-01'"
```

```bash
sling run --src-conn BIGQUERY --src-stream analytics.events \
  --tgt-conn MY_SNOWFLAKE --tgt-object public.events \
  --mode full-refresh --where "event_date >= '2024-01-01'"
```

### Incremental from BigQuery

BigQuery works as an incremental source with `primary_key` + `update_key` like any SQL database. Sling pulls rows where `update_key > max(update_key)` in the target:

```yaml
source: BIGQUERY
target: MY_SNOWFLAKE

streams:
  analytics.events:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
```

If the table also has `require_partition_filter = TRUE`, the incremental predicate is on the `update_key`, which is not necessarily the partition column. Add a `where` on the partition column, or make the partition column your `update_key`, so BigQuery's filter requirement is met.

### Multiple datasets to Snowflake (full-refresh, wildcards)

Use a wildcard stream to replicate every table across datasets. Reference the source dataset in the object name with the `{stream_schema}` runtime variable:

```yaml
source: BIGQUERY
target: MY_SNOWFLAKE

defaults:
  mode: full-refresh
  object: raw.{stream_schema}_{stream_table}

streams:
  # all tables in these datasets
  analytics.*:
  marketing.*:
```

Source partitioning does not affect full-refresh ingestion (unless `require_partition_filter` is set — see above).

If you are facing issues connecting, please reach out to us at <support@slingdata.io>, on [discord](https://discord.gg/q5xtaSNDvp) or open a Github Issue [here](https://github.com/slingdata-io/sling-cli/issues).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.slingdata.io/connections/database-connections/bigquery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.