OpenSearch

Extract data from an OpenSearch instance

OpenSearch is the community-driven, Apache-2.0 licensed search and analytics suite (a fork of Elasticsearch). Sling connects to OpenSearch using the official opensearch-go client and extracts documents from one or more indexes.

Sling is read-only for OpenSearch: you can extract data from an OpenSearch instance, but writing to an OpenSearch index as a target is not supported.

Setup

The following credentials keys are accepted:

• http_url (optional) -> Comma-separated list of HTTP URLs to connect to the cluster

• host (optional) -> The hostname / ip of the instance

• port (optional) -> The port of the instance. Default is 9200

• user (optional) -> The username to access the instance

• password (optional) -> The password to access the instance

• tls (optional) -> whether to use TLS for connecting (true/skip-verify/custom)

• cert_file (optional) -> the client certificate for TLS (file path or raw)

• cert_key_file (optional) -> the client key for TLS (file path or raw)

• cert_ca_file (optional) -> the client CA certificate for TLS (file path or raw)

Using sling conns

Here are examples of setting a connection named OPENSEARCH. We must provide the type=opensearch property:

$ sling conns set OPENSEARCH type=opensearch host=<host> user=<user> password=<password> port=9200

Environment Variable

See here to learn more about the .env.sling file.

export OPENSEARCH='opensearch://user:[e-mail]:9200'
export OPENSEARCH='{ type: opensearch, user: "user", password: "password", host: "host.ip", port: 9200 }'

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  OPENSEARCH:
    type: opensearch
    host: host.ip
    port: 9200
    user: <user>
    password: <password>
    tls: true # if TLS connection needed

  OPENSEARCH_URL:
    type: opensearch
    http_url: http://host1:9200,http://host2:9200

Indexes and Tables

Each OpenSearch index is exposed as a stream. The index name is used directly as the stream name:

streams:
  my_index:
  logs-2024.01.01:

Columns are derived from the index mapping: text, keyword and string fields become text columns, long and integer fields become bigint, float and double become float, date becomes timestamp, boolean becomes bool, and binary becomes binary. When discovering columns from the index mapping, nested object fields are expanded into dotted column names (for example meta.city). If an index has no mapping, a single data (json) column is returned.

Replication

Here's an example replication configuration to sync OpenSearch indexes to a PostgreSQL database:

source: OPENSEARCH
target: MY_POSTGRES

defaults:
  mode: full-refresh
  object: public.{stream_table}

streams:
  # extract an entire index
  my_index:

  # extract a limited number of documents
  events:
    source_options:
      limit: 10000

  # incremental extraction based on a date or numeric field
  logs:
    mode: incremental
    primary_key: id
    update_key: created_at

Flattening Documents

By default, each document is extracted as a single data JSON column:

data
"{""amount"":10.5,""created"":""2024-01-01T00:00:00Z"",""id"":1,""name"":""alice""}"

Set the flatten source option to expand the document fields into individual columns. Nested objects are flattened using __ as the separator, and arrays are kept as JSON:

streams:
  my_index:
    source_options:
      flatten: true

amount,created,id,name,meta__city,meta__zip,tags
5.5,,10,dave,paris,75001,"[""a"",""b""]"

Incremental and Backfill

For incremental extraction, set update_key to a date or numeric field in the index. Sling reads documents where that field is greater than the last stored value:

streams:
  logs:
    mode: incremental
    primary_key: id
    update_key: created_at

To backfill a specific range of an index, use the range source option together with update_key:

streams:
  logs:
    source_options:
      range: "2024-01-01,2024-06-01"
    primary_key: id
    update_key: created_at

If you are facing issues connecting, give sling assist a try, or reach out to us at [e-mail], on discord or open a Github Issue here.