Skip to content
ON CHAIN
Menu

How to Build a Crypto Trading Pipeline With Kafka: Outbox, Dead-Letter Queues and Replay

Trading systems cannot lose an order or process one twice. This guide builds the event pipeline that prevents both, using Postgres, Kafka and a few well-known patterns.

By OnChain Team8 min read

A trading pipeline built on Kafka needs four things to be safe: a transactional outbox, so an order saved in your database is always published; idempotent consumers, so an event delivered twice is processed once; a dead-letter queue, so one bad event does not block everything behind it; and a way to replay events after you fix a bug. None of these are exotic. This guide builds each one in TypeScript with Postgres and Kafka, using an order-placement flow as the example.

Why trading backends end up event-driven

Placing an order touches many systems: risk checks, the exchange or on-chain venue, balances, notifications, analytics. Calling them one after another in a single request makes the slowest or flakiest one decide your latency and your uptime. Publishing an OrderPlaced event and letting each system consume it independently decouples them. The price is that you now have to think about delivery guarantees, which is what the rest of this guide is about.

The flow we will build

How an order moves through the pipeline
USERAPIPlace orderDATAPostgresorders + outbox inone txOutbox relaySKIP LOCKED pollingDATAorders topicKeyed by accountExecutionIdempotent consumerNotificationsIdempotent consumerEXTERNALVenueclientOrderId = eventidFAILURE PATHorders.dlqAfter 5 failedattemptsinsertnew rowspublishplace order5 failuresreplay after fix

Main flowSecondaryFailure pathSwipe to see the whole diagram →

Step 1: Set up

TerminalTerminal
npm install pg @confluentinc/kafka-javascript
npm install -D typescript tsx @types/node @types/pg

KafkaJS was the default Node client for years, but it has had no release since 2023. Confluent's @confluentinc/kafka-javascript offers a KafkaJS-compatible API on top of librdkafka, so that is what this guide uses (checked November 2026). Configuration goes inside a kafkaJS object.

Create the topics with enough partitions for your expected consumer count. You can add partitions later, but doing so changes which partition a given key maps to.

Step 2: Define an event envelope

Every event gets a unique ID, a type, a version and a timestamp. The ID is what makes idempotency possible later.

events.tsTypeScript
import { randomUUID } from "node:crypto";

export type Envelope<T> = {
  id: string;
  type: string;
  version: number;
  occurredAt: string;
  key: string; // decides the partition, and so the ordering
  payload: T;
};

export function envelope<T>(type: string, key: string, payload: T): Envelope<T> {
  return {
    id: randomUUID(),
    type,
    version: 1,
    occurredAt: new Date().toISOString(),
    key,
    payload,
  };
}

Choose the key carefully. Kafka only guarantees order within a partition, and all events with the same key land on the same partition. Keying by account means all of one account's events are processed in order, which is usually what balance and risk logic needs.

Step 3: Write the order and the event in one transaction

The classic bug: save the order, then publish to Kafka, and the process dies in between. The order exists, but nobody downstream knows. The transactional outbox fixes it by writing the event to a table in the same database transaction as the order.

SQLSQL
CREATE TABLE orders (
  id uuid PRIMARY KEY,
  account_id text NOT NULL,
  market text NOT NULL,
  side text NOT NULL,
  size numeric NOT NULL,
  status text NOT NULL,
  created_at timestamptz DEFAULT now()
);
CREATE TABLE outbox (
  id uuid PRIMARY KEY,
  topic text NOT NULL,
  key text NOT NULL,
  payload jsonb NOT NULL,
  created_at timestamptz DEFAULT now(),
  published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;
place-order.tsTypeScript
import pg from "pg";
import { randomUUID } from "node:crypto";
import { envelope } from "./events";

export const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

export async function placeOrder(
  accountId: string,
  market: string,
  side: "buy" | "sell",
  size: string,
) {
  const orderId = randomUUID();
  const event = envelope("OrderPlaced", accountId, {
    orderId,
    accountId,
    market,
    side,
    size,
  });
  const tx = await db.connect();
  try {
    await tx.query("BEGIN");
    await tx.query(
      "INSERT INTO orders (id, account_id, market, side, size, status) " +
        "VALUES ($1, $2, $3, $4, $5, 'pending')",
      [orderId, accountId, market, side, size],
    );
    await tx.query(
      "INSERT INTO outbox (id, topic, key, payload) VALUES ($1, 'orders', $2, $3)",
      [event.id, event.key, event],
    );
    await tx.query("COMMIT");
    return orderId;
  } catch (e) {
    await tx.query("ROLLBACK");
    throw e;
  } finally {
    tx.release();
  }
}

Either both rows exist or neither does. Note that size is a string: never pass monetary amounts through JavaScript floating-point numbers.

Step 4: Relay the outbox to Kafka

A small relay process reads unpublished rows, publishes them and marks them as sent. FOR UPDATE SKIP LOCKED lets you run several relays without two of them sending the same row at the same moment.

relay.tsTypeScript
import { KafkaJS } from "@confluentinc/kafka-javascript";
import { db } from "./place-order";

const kafka = new KafkaJS.Kafka({
  kafkaJS: { clientId: "outbox-relay", brokers: [process.env.KAFKA_BROKER!] },
});
// wait for all in-sync replicas
const producer = kafka.producer({ kafkaJS: { idempotent: true, acks: -1 } });
await producer.connect();

for (;;) {
  const tx = await db.connect();
  try {
    await tx.query("BEGIN");
    const { rows } = await tx.query(
      `SELECT id, topic, key, payload FROM outbox WHERE published_at IS NULL
       ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED`,
    );
    for (const row of rows) {
      await producer.send({
        topic: row.topic,
        messages: [
          {
            key: row.key,
            value: JSON.stringify(row.payload),
            headers: { "event-id": row.id },
          },
        ],
      });
    }
    if (rows.length) {
      await tx.query("UPDATE outbox SET published_at = now() WHERE id = ANY($1)", [
        rows.map((r) => r.id),
      ]);
    }
    await tx.query("COMMIT");
    if (!rows.length) await new Promise((r) => setTimeout(r, 200));
  } catch (e) {
    await tx.query("ROLLBACK");
    console.error("relay error, retrying", e);
    await new Promise((r) => setTimeout(r, 1000));
  } finally {
    tx.release();
  }
}

This gives you at-least-once delivery. If the relay crashes after sending but before committing, those rows are sent again on restart. That is expected, and the next step makes it harmless.

Polling is simple and good enough for most systems. At high volume, change data capture (reading the database's write-ahead log, for example with Debezium) replaces polling with lower latency and less load.

Step 5: Make consumers idempotent

Because events can arrive more than once, every consumer records the IDs it has processed, in the same transaction as its own side effects.

SQLSQL
CREATE TABLE processed_events (
  consumer text,
  event_id uuid,
  processed_at timestamptz DEFAULT now(),
  PRIMARY KEY (consumer, event_id)
);
handle.tsTypeScript
import type pg from "pg";

export async function handleOnce(
  db: pg.Pool,
  consumer: string,
  eventId: string,
  work: (tx: pg.PoolClient) => Promise<void>,
) {
  const tx = await db.connect();
  try {
    await tx.query("BEGIN");
    const inserted = await tx.query(
      "INSERT INTO processed_events (consumer, event_id) VALUES ($1, $2) " +
        "ON CONFLICT DO NOTHING",
      [consumer, eventId],
    );
    if (inserted.rowCount === 0) {
      await tx.query("ROLLBACK"); // already processed: skip silently
      return false;
    }
    await work(tx);
    await tx.query("COMMIT");
    return true;
  } catch (e) {
    await tx.query("ROLLBACK");
    throw e;
  } finally {
    tx.release();
  }
}

For side effects outside your database, such as sending an order to a venue, pass the event ID as the client order ID. Most venues reject a second order with the same client ID, which gives you idempotency on their side too.

Step 6: Retry, then send to a dead-letter queue

A consumer that throws on one malformed event and retries forever blocks its whole partition. Retry a few times with backoff, then move the event to a dead-letter topic and carry on.

consumer.tsTypeScript
import { KafkaJS } from "@confluentinc/kafka-javascript";
import { db } from "./place-order";
import { handleOnce } from "./handle";

const kafka = new KafkaJS.Kafka({
  kafkaJS: { clientId: "execution", brokers: [process.env.KAFKA_BROKER!] },
});
const consumer = kafka.consumer({
  kafkaJS: { groupId: "execution", autoCommit: false },
});
const dlq = kafka.producer({ kafkaJS: { idempotent: true, acks: -1 } });
await Promise.all([consumer.connect(), dlq.connect()]);
await consumer.subscribe({ topics: ["orders"] });

const MAX_ATTEMPTS = 5;

async function execute(event: any, tx: unknown) {
  // risk checks, venue call with clientOrderId = event.id, order status update via tx...
}

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value!.toString());
    for (let attempt = 1; ; attempt++) {
      try {
        await handleOnce(db, "execution", event.id, (tx) => execute(event, tx));
        break;
      } catch (err) {
        if (attempt >= MAX_ATTEMPTS) {
          await dlq.send({
            topic: `${topic}.dlq`,
            messages: [
              {
                key: message.key,
                value: message.value,
                headers: {
                  error: String(err),
                  "source-partition": String(partition),
                  "source-offset": message.offset,
                },
              },
            ],
          });
          break;
        }
        await new Promise((r) => setTimeout(r, 200 * 2 ** attempt)); // 400 ms, 800 ms, 1.6 s...
      }
    }
    // Commit only after the event is handled or parked in the DLQ
    await consumer.commitOffsets([
      { topic, partition, offset: (BigInt(message.offset) + 1n).toString() },
    ]);
  },
});

Alert when anything lands in a .dlq topic. A dead-letter queue nobody watches is just a slower way to lose data.

One caution for trading: skipping an event changes the order of what follows for that key. If later events depend on earlier ones, as balance updates do, stop processing that key rather than skipping, and page someone.

Step 7: Replay after a fix

Two kinds of replay cover most incidents.

Replay dead-lettered events. After fixing the bug, a small script reads orders.dlq and publishes each message back to orders. Idempotent consumers make this safe even if some were partly processed.

Reprocess a time range. To rebuild a consumer's state, stop it, move its offsets back with the Kafka CLI, and start it again:

TerminalTerminal
kafka-consumer-groups.sh --bootstrap-server "$KAFKA_BROKER" \
  --group analytics --topic orders \
  --reset-offsets --to-datetime 2026-11-14T00:00:00.000 --execute

This only works if the topic still holds that data, so set retention on business topics with replay in mind. For data you must keep for years, archive events to cheap storage as well.

Step 8: Watch the right numbers

Metric Healthy Investigate when
Outbox backlog (unpublished rows) Near zero Growing for more than a minute
Consumer lag per group Small and stable Growing steadily
DLQ messages Zero Any arrive
End-to-end latency (order to execution) Your target Above target at p99

What we would do

Start with the outbox, idempotent consumers and a DLQ from day one. They are cheap to add early and painful to retrofit after money has gone missing. Key by account, keep amounts as strings or integers in the smallest unit, and test failure on purpose: kill the relay mid-batch and kill a consumer mid-event, then check that every order was processed exactly once.

We design and build trading backends like this in our trading platform work.

Contact

Need this built for your product?

Tell us what you are building and where you are stuck. Email is the fastest way to reach the team.

contact@onchain.ro