# How to Build a Real-Time Solana Data Pipeline

> Stream Solana transactions over Geyser gRPC, decode them into events, aggregate them in time windows and serve them live. A step-by-step guide with TypeScript.

- Author: OnChain Team
- Published: 2026-10-23
- Updated: 2026-11-15
- Page: https://www.onchain.ro/blog/solana-real-time-data-pipeline

A real-time Solana data pipeline has four stages: ingest transactions from a Geyser gRPC stream, decode them into the events you care about, aggregate those events in time windows, and serve the results over an API. Put a durable queue between every stage, so a slow or crashed stage never makes you lose data or block ingestion. This guide builds each stage in TypeScript with the Yellowstone gRPC client and Redis Streams.

## Why not just poll the RPC?

Polling `getSignaturesForAddress` and `getTransaction` works for a handful of addresses. For a whole program, such as a DEX or a token launchpad, you would be making thousands of calls per minute and still arrive late. Geyser plugins stream data straight from a validator as it processes blocks. [Yellowstone gRPC](https://github.com/rpcpool/yellowstone-grpc) is the most widely used open-source Geyser plugin, and many RPC providers offer it as a paid endpoint.

## The architecture

```diagram
title: Four stages, with a durable queue between each
node grpc at 0,0.5 external "Geyser gRPC" "Validator stream"
node backfill at 0,1.7 external "Backfill job" "Historical transactions"
node ingest at 1,0.5 default "Ingest" "Minimal work, never blocks"
node raw at 2,1 store "raw:tx" "Redis Stream"
node decode at 3,1 default "Decoders × N" "One consumer group"
node agg at 4,1 default "Aggregator" "One-minute windows"
node pg at 5,0.4 store "Postgres" "Closed windows"
node ws at 5,1.6 default "WebSocket" "Live updates to clients"
edge grpc -> ingest
edge ingest -> raw "XADD"
edge backfill -> raw "same stream" dashed
edge raw -> decode "XREADGROUP"
edge decode -> agg "events"
edge decode -> raw "reclaim stuck" dashed
edge agg -> pg "closed"
edge agg -> ws "live"
```

Each arrow into a stream is a Redis Stream. Each stage is a separate process you can restart, scale or replace on its own.

## Step 1: Set up

```bash
mkdir sol-pipeline && cd sol-pipeline
npm init -y
npm install @triton-one/yellowstone-grpc ioredis bs58 ws pg
npm install -D typescript tsx @types/node @types/ws @types/pg
```

You need a Yellowstone endpoint and token from your provider, a Redis instance, and Postgres for the stored aggregates.

```bash
export GRPC_ENDPOINT="https://your-geyser-endpoint.example"
export GRPC_TOKEN="..."
export PROGRAM_ID="the program whose transactions you want"
export REDIS_URL="redis://localhost:6379"
```

The code below was checked against version 10 of `@triton-one/yellowstone-grpc` in November 2026. The client changes between major versions, so compare it with the [official TypeScript example](https://github.com/rpcpool/yellowstone-grpc/tree/master/examples/typescript) if yours differs.

## Step 2: Ingest transactions from gRPC

Subscribe to successful, non-vote transactions that touch your program, and write a compact JSON version of each one to a Redis Stream. Do as little work as possible here: this process has to keep up with the chain.

```ts
// ingest.ts
import Client, {
  CommitmentLevel,
  type SubscribeRequest,
} from "@triton-one/yellowstone-grpc";
import Redis from "ioredis";
import bs58 from "bs58";

const redis = new Redis(process.env.REDIS_URL!);
const client = new Client(
  process.env.GRPC_ENDPOINT!,
  process.env.GRPC_TOKEN,
  undefined,
);
await client.connect();

const request: SubscribeRequest = {
  accounts: {},
  slots: {},
  transactions: {
    program: {
      vote: false,
      failed: false,
      accountInclude: [process.env.PROGRAM_ID!],
      accountExclude: [],
      accountRequired: [],
    },
  },
  transactionsStatus: {},
  entry: {},
  blocks: {},
  blocksMeta: {},
  blockFooter: {}, // required by recent client versions, even when unused
  accountsDataSlice: [],
  commitment: CommitmentLevel.CONFIRMED,
  ping: undefined,
};

const stream = await client.subscribe();

stream.on("data", async (update) => {
  const tx = update.transaction;
  const info = tx?.transaction;
  const message = info?.transaction?.message;
  if (!tx || !info?.meta || !message) return; // not a transaction update
  const raw = {
    slot: String(tx.slot),
    signature: bs58.encode(Buffer.from(info.signature)),
    accountKeys: message.accountKeys.map((k: Uint8Array) =>
      bs58.encode(Buffer.from(k)),
    ),
    pre: info.meta.preTokenBalances,
    post: info.meta.postTokenBalances,
    receivedAt: Date.now(),
  };
  // Cap the stream so a stalled consumer cannot fill Redis memory
  await redis.xadd("raw:tx", "MAXLEN", "~", "200000", "*", "data", JSON.stringify(raw));
});

stream.on("error", (err) => {
  console.error("stream error", err);
  process.exit(1); // let the process manager restart us
});

await new Promise<void>((resolve, reject) =>
  stream.write(request, (err: Error | null | undefined) =>
    err ? reject(err) : resolve(),
  ),
);
console.log("subscribed");
```

Exiting on error is deliberate. A supervisor (systemd, Docker, Kubernetes) restarts the process, and the queue absorbs the gap. The client also has a built-in reconnect option worth enabling once the basics work.

## Step 3: Decode events with a consumer group

Consumer groups let several decoder processes share one stream. Each entry goes to one consumer, and it stays "pending" until that consumer acknowledges it, so a crash mid-batch does not lose data.

```ts
// decode.ts
import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL!);
const STREAM = "raw:tx";
const GROUP = "decoders";
const CONSUMER = `decoder-${process.pid}`;

try {
  await redis.xgroup("CREATE", STREAM, GROUP, "$", "MKSTREAM");
} catch (e) {
  if (!String(e).includes("BUSYGROUP")) throw e; // the group already exists
}

type Balance = { owner: string; mint: string; uiTokenAmount: { uiAmount: number } };

function decode(raw: {
  signature: string;
  slot: string;
  pre: Balance[];
  post: Balance[];
  receivedAt: number;
}) {
  // Net token change per (owner, mint): positive means the owner received tokens.
  const delta = new Map<string, number>();
  for (const b of raw.pre)
    delta.set(`${b.owner}|${b.mint}`, -(b.uiTokenAmount?.uiAmount ?? 0));
  for (const b of raw.post) {
    const key = `${b.owner}|${b.mint}`;
    delta.set(key, (delta.get(key) ?? 0) + (b.uiTokenAmount?.uiAmount ?? 0));
  }
  return [...delta]
    .filter(([, d]) => Math.abs(d) > 1e-12)
    .map(([key, d]) => {
      const [owner, mint] = key.split("|");
      return {
        signature: raw.signature,
        slot: raw.slot,
        owner,
        mint,
        delta: d,
        ts: raw.receivedAt,
      };
    });
}

for (;;) {
  const res = (await redis.xreadgroup(
    "GROUP",
    GROUP,
    CONSUMER,
    "COUNT",
    200,
    "BLOCK",
    5000,
    "STREAMS",
    STREAM,
    ">",
  )) as [string, [string, string[]][]][] | null;
  if (!res) continue;

  for (const [, entries] of res) {
    const pipeline = redis.pipeline();
    for (const [id, fields] of entries) {
      const raw = JSON.parse(fields[1]);
      for (const event of decode(raw)) {
        pipeline.xadd(
          "events",
          "MAXLEN",
          "~",
          "500000",
          "*",
          "data",
          JSON.stringify(event),
        );
      }
      pipeline.xack(STREAM, GROUP, id);
    }
    await pipeline.exec();
  }
}
```

This decoder only computes token balance changes, which works for any program. For richer events, such as pool, price or fee, parse the program's instructions or its logs using its published interface (IDL). Keep that logic in this stage only, so you can change it and replay without touching ingestion.

Entries left pending by a crashed consumer can be taken over with [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim/). Run it on startup so nothing stays stuck.

## Step 4: Aggregate in time windows

Most dashboards need numbers per token per minute, not individual events. Keep the open window in memory, and write it to Postgres when the minute closes.

```ts
// aggregate.ts
import Redis from "ioredis";
import pg from "pg";

const redis = new Redis(process.env.REDIS_URL!);
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const pub = new Redis(process.env.REDIS_URL!);

await db.query(`CREATE TABLE IF NOT EXISTS token_minute (
  mint text, minute timestamptz, events int, inflow double precision, outflow double precision,
  wallets int, PRIMARY KEY (mint, minute))`);

type Window = { events: number; inflow: number; outflow: number; wallets: Set<string> };
const windows = new Map<string, Window>(); // key: mint|minuteStart

async function flush(before: number) {
  for (const [key, w] of windows) {
    const [mint, start] = key.split("|");
    if (Number(start) >= before) continue;
    await db.query(
      `INSERT INTO token_minute VALUES ($1, to_timestamp($2 / 1000.0), $3, $4, $5, $6)
       ON CONFLICT (mint, minute) DO UPDATE SET
         events = EXCLUDED.events, inflow = EXCLUDED.inflow,
       outflow = EXCLUDED.outflow, wallets = EXCLUDED.wallets`,
      [mint, Number(start), w.events, w.inflow, w.outflow, w.wallets.size],
    );
    windows.delete(key);
  }
}

let lastId = "$";
for (;;) {
  const res = (await redis.xread(
    "COUNT",
    500,
    "BLOCK",
    2000,
    "STREAMS",
    "events",
    lastId,
  )) as [string, [string, string[]][]][] | null;
  for (const [, entries] of res ?? []) {
    for (const [id, fields] of entries) {
      lastId = id;
      const e = JSON.parse(fields[1]);
      const minute = Math.floor(e.ts / 60000) * 60000;
      const key = `${e.mint}|${minute}`;
      const w = windows.get(key) ?? {
        events: 0,
        inflow: 0,
        outflow: 0,
        wallets: new Set<string>(),
      };
      w.events += 1;
      if (e.delta > 0) w.inflow += e.delta;
      else w.outflow += -e.delta;
      w.wallets.add(e.owner);
      windows.set(key, w);
      await pub.publish(
        "live",
        JSON.stringify({
          mint: e.mint,
          minute,
          events: w.events,
          wallets: w.wallets.size,
        }),
      );
    }
  }
  // close every window before the current minute
  await flush(Math.floor(Date.now() / 60000) * 60000);
}
```

Two production notes. First, this aggregator tracks its position in memory, so a restart starts from new events only. Store `lastId` in Redis or Postgres to resume where you stopped. Second, if you run several aggregators, split tokens between them (for example by a hash of the mint) so two processes never own the same window.

## Step 5: Serve live updates over WebSocket

The aggregator publishes every window update to a Redis channel. A thin server relays them to browsers, and a REST route can read history from Postgres.

```ts
// serve.ts
import Redis from "ioredis";
import { WebSocketServer, WebSocket } from "ws";

const sub = new Redis(process.env.REDIS_URL!);
const wss = new WebSocketServer({ port: 8080 });

await sub.subscribe("live");
sub.on("message", (_channel, message) => {
  for (const client of wss.clients) {
    if (client.readyState === WebSocket.OPEN) client.send(message);
  }
});
console.log("ws://localhost:8080");
```

## Step 6: Backfill and correctness

Live streaming only gives you data from the moment you connect. Plan for three gaps:

- **Backfill.** For history, page through `getSignaturesForAddress` for the program and fetch each transaction, writing to the same `raw:tx` stream. The decoder cannot tell the difference, which is the point of the design.
- **Duplicates.** Reconnects and backfills overlap with live data. Make writes idempotent with the signature as a unique key, as the `ON CONFLICT` clause above does for windows.
- **Commitment.** `confirmed` data can, rarely, be rolled back. If a figure must be exact, for billing or reporting, recompute it from `finalized` data in a nightly job.

## Step 7: Watch the pipeline itself

A pipeline that falls behind silently is worse than one that crashes. Track three numbers and alert on them:

| Metric | How to measure | Why |
|---|---|---|
| Ingest lag | Time from `receivedAt` to now at the decoder | Shows whether you keep up with the chain |
| Pending entries | `XPENDING raw:tx decoders` | Grows when decoders are stuck or crashing |
| Stream length | `XLEN raw:tx` | Near your `MAXLEN` cap means data is about to be dropped |

## What we would do

Start with Redis Streams, as in this guide. They are simple to run and fast enough for a single program's traffic. Move to Kafka when you need long retention, replay of days of data, or many independent consumers. The stage boundaries stay the same, only the transport changes. Keep raw data for at least as long as you might want to fix a decoding bug and replay.

If you only need a list of specific wallets, the simpler approach in [tracking real-time trader activity on Solana](/blog/solana-real-time-trader-activity) may be enough. We design and run pipelines like this in our [Solana data and analytics work](/services/solana-data-scrapers).

---

OnChain · 15 Bld. Iuliu Maniu, 061075 Bucharest, Romania · contact@onchain.ro · https://www.onchain.ro
