Kafka

A Kafka connection holds reusable broker and security settings. Every generator that uses it produces one Kafka record with a required topic and value, plus an optional key and headers.

Suppose an order service expects this record:

{
  "topic": "shop.orders",
  "key": "ORD-001",
  "headers": {
    "eventType": "ORDER_PLACED",
    "source": "synthtraffic"
  },
  "value": {
    "orderId": "ORD-001",
    "status": "placed",
    "total": 49.99
  }
}

Publish an order record

connections:
  eventBus:
    type: kafka
    brokers: [localhost:9092]

generators:
  - name: orders
    connection: eventBus
    config:
      maxEvents: 2
    vars:
      orderId: =seq(format=ORD-%03d)
    topic: shop.orders
    key: $orderId
    headers:
      eventType: ORDER_PLACED
      source: synthtraffic
    value:
      orderId: $orderId
      status: placed
      total: =cycle(49.99, 84.50)

The connection and generator have different jobs:

• connections.eventBus configures the Kafka client once.
• brokers is a non-empty list of bootstrap addresses.
• connection: eventBus selects that client for the generator.
• topic and value are required for every Kafka generator.
• key and headers are optional record metadata.
• vars.orderId calculates the ID once so the key and payload reuse the same value.

Preview without Kafka

sample and run --stdout preserve the Kafka envelope but do not open a broker connection. This is also true when the configured broker is unavailable.

sample prints a short preview to your terminal. It does not wait for real-time pacing, and it never opens Kafka, PostgreSQL, or other destinations.

synthtraffic sample kafka.yaml --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/kafka.yaml:/work/kafka.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  sample /work/kafka.yaml --events 2 --seed 42

run --stdout prints the destination-shaped envelope locally without opening the configured connection. Remove --stdout when you are ready to send to that destination.

synthtraffic run kafka.yaml --stdout --events 2 --seed 42

docker run --rm \
  --env-file ./license.env \
  --volume "${PWD}/kafka.yaml:/work/kafka.yaml:ro" \
  synthtraffic/synthtraffic:latest \
  run /work/kafka.yaml --stdout --events 2 --seed 42

Full flag lists: sample and run. Install and license: Install.

Generator fields

Kafka generators cannot use PostgreSQL, HTTP, or storage output fields such as row, body, or prefix.

Serialization

The default value serializer is json. The key serializer is selected from the key value: objects and lists use json; scalars use string. Override either choice when the consumer requires a specific wire format:

serialization:
  key: string
  value: json

Supported serializers are json, string, and bytes. bytes only accepts a bytes value; it does not convert ordinary strings into bytes. A missing key remains a Kafka null key.

Connection settings

Only literals and env() may appear in connection settings. Event expressions such as uuid() belong on a generator. If one broker entry comes from an environment variable, a comma-separated value is split into multiple broker addresses.

Invalid acks or compression values fail when the Kafka client opens. tls: true uses the operating system CA pool; the DSL does not accept custom CA or client-certificate fields.

TLS and SASL

Managed Kafka commonly requires TLS and SASL:

connections:
  eventBus:
    type: kafka
    brokers:
      - =env(KAFKA_BOOTSTRAP_SERVERS)
    clientId: synthtraffic
    acks: all
    compression: zstd
    tls: true
    sasl:
      mechanism: scram-sha-512
      username: =env(KAFKA_SASL_USERNAME)
      password: =env(KAFKA_SASL_PASSWORD)

sasl requires all three fields. mechanism supports plain, scram-sha-256, and scram-sha-512; username and password must be non-empty literals or env() values. Keep credentials out of scenario files.

Publish for real

Before removing --stdout:

1. Make sure every broker address is reachable from the Synthtraffic process.
2. Set any environment variables used by the connection.
3. Create shop.orders if the cluster disables automatic topic creation.
4. Run:

synthtraffic run kafka.yaml --events 2 --seed 42

Synthtraffic allows broker-side automatic topic creation and checks broker connectivity when the run opens. It flushes accepted records before a successful run exits. A publish error fails the run instead of silently dropping later records.

Complete secured-broker example: examples/connectors/kafka_cloud.yaml.

Related

• Connection mental model: Connections
• Shared IDs between key and value: vars and const
• Delay, discard, and repeat: Delivery behavior
• Environment values: Strings and env