Source Options

Specification

Here we have various keys accepted for source options:

Key	Description
compression	(Only for file source) The type of compression to use when reading files. Valid inputs are none, auto and gzip, zstd, snappy. Default is auto.
datetime_format	The ISO 8601 date format to use when reading date values. Default is auto
delimiter	(Only for file source) The delimiter to use when parsing tabular files. Default is auto.
escape	(Only for file source - since v1.2.4) The escape character to use when parsing tabular files. Default is "
empty_as_null	Whether empty fields should be treated as NULL. Default is true.
flatten	(Only for file source) Whether to flatten a semi-structure file source format (JSON, XML)
format	(Only for file source) The format of the file(s). Options are: csv, parquet, xlsx, avro, json, jsonlines, sas7bdat and xml.
header	(Only for file source) Whether to consider the first line as header. Default is true.
jmespath	(Only for file source) Specify a JMESPath expression to use to filter / extract nested JSON data. See https://jmespath.org/ for more
limit	The maximum number of rows to pull from the source
null_if	Whether this case-sensitive value should be treated as NULL when encountered. Default is NULL.
sheet	(Only for Excel source files) The name of the sheet to use as a data source, for example Sheet1. Default is the first sheet. You can also specify the range (Sheet2!B:H, Sheet3!B1:H70).
range	The range to use for backfill mode, separated by a single comma. Example: 2021-01-01,2021-02-01 or 1,10000
skip_blank_lines	Whether blank lines should be skipped when encountered. Default is false.