S3

S3 uses the File artifact model: generators produce {prefix, value} envelopes, then the connection groups them into rolling JSON or JSONL objects.

The scenario below previews:

{
  "prefix": "orders/part-",
  "value": {
    "orderId": "ORD-001",
    "status": "placed"
  }
}

Write order objects

connections:
  objectStore:
    type: s3
    bucket: synthtraffic-demo
    region: us-east-1

generators:
  - name: orders
    connection: objectStore
    config:
      maxEvents: 2
    vars:
      orderId: =seq(format=ORD-%03d)
    prefix: orders/part-
    value:
      orderId: $orderId
      status: placed

bucket and region select the destination. prefix becomes the beginning of each S3 object key; it is not the name of a pre-existing folder.

Preview without S3

sample prints a short preview to your terminal. It does not wait for real-time pacing, and it never opens Kafka, PostgreSQL, or other destinations.

synthtraffic sample s3.yaml --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/s3.yaml:/work/s3.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  sample /work/s3.yaml --events 2 --seed 42

run --stdout prints the destination-shaped envelope locally without opening the configured connection. Remove --stdout when you are ready to send to that destination.

synthtraffic run s3.yaml --stdout --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/s3.yaml:/work/s3.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  run /work/s3.yaml --stdout --events 2 --seed 42

Full flag lists: sample and run. Install and license: Install.

Neither command resolves AWS credentials, checks the bucket, or sends a request.

Connection settings

endpoint cannot contain credentials, query parameters, or a fragment. pathStyle and concurrency must be literals; do not wrap them in env().

concurrency limits simultaneous object commits. maxOpenStreams limits the number of active generator-and-prefix streams before the least recently used stream is committed.

Authentication

A connected run loads the standard AWS configuration and credential chain. Common sources include AWS environment variables, shared config and credential files, and workload roles. Access keys are not valid scenario fields.

For the CLI, export the appropriate AWS values before running. For Docker, pass them with an additional environment file:

docker run --rm \
  --env-file ./license.env \
  --env-file ./aws.env \
  --volume "${PWD}/s3.yaml:/work/s3.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  run /work/s3.yaml --events 2 --seed 42

Keep both files out of source control.

S3-compatible services

Use endpoint and usually path-style addressing for MinIO:

connections:
  objectStore:
    type: s3
    bucket: synthtraffic-demo
    region: us-east-1
    endpoint: http://localhost:9000
    pathStyle: true

Complete example: examples/connectors/s3-minio.yaml.

Object and delivery behavior

• The bucket must already exist.
• Object names use {prefix}{runULID}-{ordinal}.{format}[.gz].
• Synthtraffic refuses to overwrite an existing object.
• Objects at least 8 MiB are uploaded in 8 MiB multipart parts.
• jsonl objects use application/x-ndjson; json uses application/json. Gzip adds Content-Encoding: gzip.
• An event counts as delivered only after its artifact commits. Missing buckets, name collisions, credential failures, and upload failures fail the run.

Prefix validation, rolling thresholds, formats, compression, and stream eviction are identical to File.

Write for real

Set credentials, create the bucket, and remove --stdout:

synthtraffic run s3.yaml --events 2 --seed 42

Complete AWS example: examples/connectors/s3.yaml.