Source Options
Last updated
Last updated
Here we have various keys accepted for source options:
compression
(Only for file source)
The type of compression to use when reading files. Valid inputs are none, auto and gzip, zstd, snappy. Default is auto.
chunk_size
(Only for database source)
The chunk size for backfill processing. This tells Sling to split a stream into many. Accepts values such as 12h, 7d or 1m. See for more details.
datetime_format
The date format to use when reading date values. Default is auto
delimiter
(Only for file source)
The delimiter to use when parsing tabular files. Default is auto.
escape
(Only for file source - since v1.2.4)
The escape character to use when parsing tabular files. Default is "
empty_as_null
Whether empty fields should be treated as NULL. Default is false starting in v1.4.5. Prior, default depends on the kind of the source connection: false for , true for .
flatten
Whether to flatten a semi-structure source format (JSON, XML). Accepts true or false boolean values. Since v1.4.5, also accepts an integer, representing the maximum flattening depth. 0 means infinite depth.
format
(Only for file source)
The format of the file(s). Options are: csv, parquet, xlsx, avro, json, jsonlines, sas7bdat and xml.
header
(Only for file source) Whether to consider the first line as header. Default is true.
jmespath
(Only for file and NoSQL database source)
Specify a JMESPath expression to use to filter / extract nested JSON data. See for more
limit
The maximum number of rows to pull from the source
null_if
Whether this case-sensitive string value should be treated as a database null value when encountered. Default is NULL.
sheet
(Only for Excel source files) The name of the sheet to use as a data source, for example Sheet1. Default is the first sheet. You can also specify the range (Sheet2!B:H, Sheet3!B1:H70).
range
The range to use for backfill mode, separated by a single comma. Example: 2021-01-01,2021-02-01 or 1,10000
skip_blank_lines
Whether blank lines should be skipped when encountered. Default is false.