AWS S3

Connect & Ingest data from / into a AWS S3 Bucket

Setup

The following credentials keys are accepted:

• bucket (required)

• use_environment (optional) -> whether to use the AWS environment variables to access. Accepts true or false.

• profile (optional) (or provide environment variable AWS_PROFILE)

• access_key_id (optional) (or provide environment variable AWS_ACCESS_KEY_ID)

• secret_access_key (optional) (or provide environment variable AWS_SECRET_ACCESS_KEY)

• session_token (optional) (or provide environment variable AWS_SESSION_TOKEN)

• region (optional) (or provide environment variable AWS_REGION)

• endpoint (optional) (or provide environment variable AWS_ENDPOINT)

• role_arn (optional) (or provide environment variable AWS_ROLE_ARN)

• anonymous (optional) Tells to access the bucket as anonymous (set to true)

• encryption_algorithm (optional since v1.2.15) The server-side encryption algorithm to use. Accepts AES256, aws:kms or aws:kms:arn.

• encryption_kms_key (optional since v1.2.15) If encryption_algorithm is aws:kms or aws:kms:arn, this is the KMS key ID or ARN to use for encryption.

• http_timeout (optional) The HTTP client timeout for S3 requests. Accepts Go duration format (e.g., 30s, 5m, 10m30s). Default is 30s. Increase this value if you're experiencing timeout errors with large files or slow network connections.

Using sling conns

Here are examples of setting a connection named AWS_S3. We must provide the type=s3 property:

$ sling conns set AWS_S3 type=s3 bucket=sling-bucket profile=my-profile

$ sling conns set AWS_S3 type=s3 bucket=sling-bucket access_key_id=ACCESS_KEY_ID secret_access_key="SECRET_ACCESS_KEY"

Environment Variable

In JSON/YAML format:

export AWS_S3='{type: s3, bucket: sling-bucket, access_key_id: ACCESS_KEY_ID, secret_access_key: "SECRET_ACCESS_KEY"}'

export AWS_S3='{type: s3, bucket: sling-bucket, profile: my-profile}'

# or via AWS classic environment variables
export AWS_ACCESS_KEY_ID='<aws_access_key_id>'
export AWS_SECRET_ACCESS_KEY='<aws_secret_access_key>'
export AWS_SESSION_TOKEN='<aws_session_token>'

Sling Env File YAML

See here to learn more about the sling env.yaml file.

connections:
  AWS_S3:
    type: s3
    bucket: <bucket>
    profile: <my-profile>

  AWS_S3_OTHER:
    type: s3
    bucket: <bucket>
    access_key_id: <access_key_id>
    secret_access_key: '<secret_access_key>'
    http_timeout: 5m  # Increase timeout for large files or slow connections

If you are facing issues connecting, please reach out to us at [email protected], on discord or open a Github Issue here.