File

A file connection groups generated values into rolling artifacts under one root directory. Each generator supplies a relative prefix that controls subdirectories and the beginning of each artifact name.

The preview for this scenario is an artifact envelope:

{
  "prefix": "orders/part-",
  "value": {
    "orderId": "ORD-001",
    "status": "placed"
  }
}

Write order artifacts

connections:
  localFiles:
    type: file
    directory: ./output

generators:
  - name: orders
    connection: localFiles
    config:
      maxEvents: 2
    vars:
      orderId: =seq(format=ORD-%03d)
    prefix: orders/part-
    value:
      orderId: $orderId
      status: placed

• directory is the root for every artifact written by the connection.
• format, compression, and rolling apply to every generator using it.
• connection: localFiles selects the file connection.
• prefix and value are required storage output fields.

Preview without writing files

sample prints a short preview to your terminal. It does not wait for real-time pacing, and it never opens Kafka, PostgreSQL, or other destinations.

synthtraffic sample file.yaml --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/file.yaml:/work/file.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  sample /work/file.yaml --events 2 --seed 42

run --stdout prints the destination-shaped envelope locally without opening the configured connection. Remove --stdout when you are ready to send to that destination.

synthtraffic run file.yaml --stdout --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/file.yaml:/work/file.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  run /work/file.yaml --stdout --events 2 --seed 42

Full flag lists: sample and run. Install and license: Install.

Both commands print {prefix, value} envelopes. They do not create the output directory or any artifact. A prefix derived from env() or a sensitive field is shown as [REDACTED].

Connection settings

You may override only one rolling value; omitted values keep their defaults. For example, rolling: {maxEvents: 100} keeps the default byte and time limits.

directory is resolved relative to the scenario file, not the shell’s current directory. Synthtraffic creates missing directories. Connection settings may use literals or env() only.

Generator fields

Storage generators use:

Standard generator fields such as config, const, vars, instances, and stateMachine still work. Destination fields from other connectors—such as topic, row, and body—do not.

Prefix rules

prefix may be a literal or a scalar expression:

vars:
  tenantPrefix: =format(orders/tenant=%s/part-, $tenantId)
prefix: $tenantPrefix

A prefix must:

• be non-empty valid UTF-8
• use / separators and not begin with /
• contain no backslash, empty path segment, control character, . segment, or .. segment

The completed artifact name is limited to 1024 UTF-8 bytes, and each local path component is limited to 255 bytes. Reusing the same literal prefix for two generators on one connection produces a warning because the streams are independent and their names can be confusing.

Artifact names and contents

Artifacts are named:

{prefix}{runULID}-{ordinal}.{format}[.gz]

For example: orders/part-01K4...P9-000001.jsonl.gz.

• runULID uniquely identifies the run.
• ordinal starts at 000001 for a prefix and increases as it rolls.
• jsonl writes one JSON value followed by a newline for each event.
• json writes one compact JSON array.
• gzip adds .gz and compresses the complete artifact.

Rolling and open streams

Each generator-and-prefix pair has its own stream. The stream rolls as soon as the first configured threshold is reached:

• event count reaches rolling.maxEvents
• encoded artifact size reaches rolling.maxBytes
• rolling.interval elapses

maxOpenStreams limits active streams on one connection. When the limit is reached, Synthtraffic commits the least recently used stream before opening another.

Safe commits and delivery

File artifacts are written to a temporary file and installed atomically without overwriting an existing final path. Parent directories are created as needed, and resolved paths and symbolic links cannot escape directory.

An event counts as delivered only after its artifact commits. A stage boundary or successful run flushes open artifacts; a write, path, or name collision error fails the run.

Write for real

Remove --stdout to create the directory and artifacts:

synthtraffic run file.yaml --events 2 --seed 42

The same artifact format, rolling, prefix, and delivery model is used by S3, GCS, and Azure Blob.

Complete example: examples/connectors/file.yaml.