MongoDB

Extract data from a MongoDB instance

Setup

The following credentials keys are accepted:

• host (required) -> The hostname / ip of the instance

• user (required) -> The username to access the instance

• password (optional) -> The password to access the instance

• port (optional) -> The port of the instance. Default is 27017.

• tls (optional) -> whether to use TLS for connecting (true/false).

• cert_file (optional) -> the client certificate to use to access the instance via TLS (file path or raw)

• cert_key_file (optional) -> the client key to use to access the instance via TLS (file path or raw)

• cert_ca_file (optional) -> the client CA certificate to use to access the instance via TLS (file path or raw)

Using sling conns

Here are examples of setting a connection named MONGODB. We must provide the type=mongodb property:

$ sling conns set MONGODB type=mongodb host=<host> user=<user> password=<password> port=<port> 

Environment Variable

# see https://www.mongodb.com/docs/manual/reference/connection-string-options/#std-label-connections-connection-options for URL options
export MONGODB='mongodb://root:[email protected]:27017?authSource=<database>'
export MONGODB='{ type: mongodb, user: "root", password: "password", host: "host.ip", port: 27017, authSource: "<database>" }'

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  MONGODB:
    type: mongodb
    host: host.ip
    port: 27017
    user: <user>
    password: <password>
    tls: true # if TLS connection needed

  # see https://www.mongodb.com/docs/manual/reference/connection-string-options/#std-label-connections-connection-options for URL options
  MONGODB_URL:
    type: mongodb
    url: mongodb://root:[email protected]:27017?authSource=<database>

  MONGODB_TLS:
    type: mongodb
    url: mongodb://root:[email protected]:27017?tls=true

Examples

source: mongodb
target: oracle

defaults:
  mode: full-refresh
  object: user1.{stream_schema}_{stream_table}

streams:
  db1.collection1:
  db1.collection2:
    select: [col1, col2]
    where: '{"field": "value", "field2": {"$gt": 100}}'  # use "where" starting v1.3.6
  db2.collection3:
    where: '[{"$or": [{"field1": "value1"}, {"field2": "value2"}]}]'  # use "where" starting v1.3.6
  db2.collection4:
    mode: incremental
    primary_key: [id]
    update_key: last_mod_at
  db2.collection5:
    mode: incremental
    primary_key: [id]
    where: '{"_id": {"$gte": "67859d8ee682ab32317abc6f", "$lte": "67859d8ee682ab32317abcc8"}}' # filter on ID (v1.4.21+)
    update_key: last_mod_at

If you are facing issues connecting, please reach out to us at [email protected], on discord or open a Github Issue here.