Skip to content
ON CHAIN
Menu

How to Build a Real-Time Solana Data Pipeline

A practical architecture for real-time Solana data: gRPC ingestion, a durable queue between stages, decoding, windowed aggregation and a live API, with working code for each stage.

By OnChain Team8 min read

A real-time Solana data pipeline has four stages: ingest transactions from a Geyser gRPC stream, decode them into the events you care about, aggregate those events in time windows, and serve the results over an API. Put a durable queue between every stage, so a slow or crashed stage never makes you lose data or block ingestion. This guide builds each stage in TypeScript with the Yellowstone gRPC client and Redis Streams.

Why not just poll the RPC?

Polling getSignaturesForAddress and getTransaction works for a handful of addresses. For a whole program, such as a DEX or a token launchpad, you would be making thousands of calls per minute and still arrive late. Geyser plugins stream data straight from a validator as it processes blocks. Yellowstone gRPC is the most widely used open-source Geyser plugin, and many RPC providers offer it as a paid endpoint.

The architecture

Four stages, with a durable queue between each
EXTERNALGeyser gRPCValidator streamEXTERNALBackfill jobHistoricaltransactionsIngestMinimal work, neverblocksDATAraw:txRedis StreamDecoders × NOne consumer groupAggregatorOne-minute windowsDATAPostgresClosed windowsWebSocketLive updates toclientsXADDsame streamXREADGROUPeventsreclaim stuckclosedlive

Main flowSecondarySwipe to see the whole diagram →

Each arrow into a stream is a Redis Stream. Each stage is a separate process you can restart, scale or replace on its own.

Step 1: Set up

TerminalTerminal
mkdir sol-pipeline && cd sol-pipeline
npm init -y
npm install @triton-one/yellowstone-grpc ioredis bs58 ws pg
npm install -D typescript tsx @types/node @types/ws @types/pg

You need a Yellowstone endpoint and token from your provider, a Redis instance, and Postgres for the stored aggregates.

TerminalTerminal
export GRPC_ENDPOINT="https://your-geyser-endpoint.example"
export GRPC_TOKEN="..."
export PROGRAM_ID="the program whose transactions you want"
export REDIS_URL="redis://localhost:6379"

The code below was checked against version 10 of @triton-one/yellowstone-grpc in November 2026. The client changes between major versions, so compare it with the official TypeScript example if yours differs.

Step 2: Ingest transactions from gRPC

Subscribe to successful, non-vote transactions that touch your program, and write a compact JSON version of each one to a Redis Stream. Do as little work as possible here: this process has to keep up with the chain.

ingest.tsTypeScript
import Client, {
  CommitmentLevel,
  type SubscribeRequest,
} from "@triton-one/yellowstone-grpc";
import Redis from "ioredis";
import bs58 from "bs58";

const redis = new Redis(process.env.REDIS_URL!);
const client = new Client(
  process.env.GRPC_ENDPOINT!,
  process.env.GRPC_TOKEN,
  undefined,
);
await client.connect();

const request: SubscribeRequest = {
  accounts: {},
  slots: {},
  transactions: {
    program: {
      vote: false,
      failed: false,
      accountInclude: [process.env.PROGRAM_ID!],
      accountExclude: [],
      accountRequired: [],
    },
  },
  transactionsStatus: {},
  entry: {},
  blocks: {},
  blocksMeta: {},
  blockFooter: {}, // required by recent client versions, even when unused
  accountsDataSlice: [],
  commitment: CommitmentLevel.CONFIRMED,
  ping: undefined,
};

const stream = await client.subscribe();

stream.on("data", async (update) => {
  const tx = update.transaction;
  const info = tx?.transaction;
  const message = info?.transaction?.message;
  if (!tx || !info?.meta || !message) return; // not a transaction update
  const raw = {
    slot: String(tx.slot),
    signature: bs58.encode(Buffer.from(info.signature)),
    accountKeys: message.accountKeys.map((k: Uint8Array) =>
      bs58.encode(Buffer.from(k)),
    ),
    pre: info.meta.preTokenBalances,
    post: info.meta.postTokenBalances,
    receivedAt: Date.now(),
  };
  // Cap the stream so a stalled consumer cannot fill Redis memory
  await redis.xadd("raw:tx", "MAXLEN", "~", "200000", "*", "data", JSON.stringify(raw));
});

stream.on("error", (err) => {
  console.error("stream error", err);
  process.exit(1); // let the process manager restart us
});

await new Promise<void>((resolve, reject) =>
  stream.write(request, (err: Error | null | undefined) =>
    err ? reject(err) : resolve(),
  ),
);
console.log("subscribed");

Exiting on error is deliberate. A supervisor (systemd, Docker, Kubernetes) restarts the process, and the queue absorbs the gap. The client also has a built-in reconnect option worth enabling once the basics work.

Step 3: Decode events with a consumer group

Consumer groups let several decoder processes share one stream. Each entry goes to one consumer, and it stays "pending" until that consumer acknowledges it, so a crash mid-batch does not lose data.

decode.tsTypeScript
import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL!);
const STREAM = "raw:tx";
const GROUP = "decoders";
const CONSUMER = `decoder-${process.pid}`;

try {
  await redis.xgroup("CREATE", STREAM, GROUP, "$", "MKSTREAM");
} catch (e) {
  if (!String(e).includes("BUSYGROUP")) throw e; // the group already exists
}

type Balance = { owner: string; mint: string; uiTokenAmount: { uiAmount: number } };

function decode(raw: {
  signature: string;
  slot: string;
  pre: Balance[];
  post: Balance[];
  receivedAt: number;
}) {
  // Net token change per (owner, mint): positive means the owner received tokens.
  const delta = new Map<string, number>();
  for (const b of raw.pre)
    delta.set(`${b.owner}|${b.mint}`, -(b.uiTokenAmount?.uiAmount ?? 0));
  for (const b of raw.post) {
    const key = `${b.owner}|${b.mint}`;
    delta.set(key, (delta.get(key) ?? 0) + (b.uiTokenAmount?.uiAmount ?? 0));
  }
  return [...delta]
    .filter(([, d]) => Math.abs(d) > 1e-12)
    .map(([key, d]) => {
      const [owner, mint] = key.split("|");
      return {
        signature: raw.signature,
        slot: raw.slot,
        owner,
        mint,
        delta: d,
        ts: raw.receivedAt,
      };
    });
}

for (;;) {
  const res = (await redis.xreadgroup(
    "GROUP",
    GROUP,
    CONSUMER,
    "COUNT",
    200,
    "BLOCK",
    5000,
    "STREAMS",
    STREAM,
    ">",
  )) as [string, [string, string[]][]][] | null;
  if (!res) continue;

  for (const [, entries] of res) {
    const pipeline = redis.pipeline();
    for (const [id, fields] of entries) {
      const raw = JSON.parse(fields[1]);
      for (const event of decode(raw)) {
        pipeline.xadd(
          "events",
          "MAXLEN",
          "~",
          "500000",
          "*",
          "data",
          JSON.stringify(event),
        );
      }
      pipeline.xack(STREAM, GROUP, id);
    }
    await pipeline.exec();
  }
}

This decoder only computes token balance changes, which works for any program. For richer events, such as pool, price or fee, parse the program's instructions or its logs using its published interface (IDL). Keep that logic in this stage only, so you can change it and replay without touching ingestion.

Entries left pending by a crashed consumer can be taken over with XAUTOCLAIM. Run it on startup so nothing stays stuck.

Step 4: Aggregate in time windows

Most dashboards need numbers per token per minute, not individual events. Keep the open window in memory, and write it to Postgres when the minute closes.

aggregate.tsTypeScript
import Redis from "ioredis";
import pg from "pg";

const redis = new Redis(process.env.REDIS_URL!);
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const pub = new Redis(process.env.REDIS_URL!);

await db.query(`CREATE TABLE IF NOT EXISTS token_minute (
  mint text, minute timestamptz, events int, inflow double precision, outflow double precision,
  wallets int, PRIMARY KEY (mint, minute))`);

type Window = { events: number; inflow: number; outflow: number; wallets: Set<string> };
const windows = new Map<string, Window>(); // key: mint|minuteStart

async function flush(before: number) {
  for (const [key, w] of windows) {
    const [mint, start] = key.split("|");
    if (Number(start) >= before) continue;
    await db.query(
      `INSERT INTO token_minute VALUES ($1, to_timestamp($2 / 1000.0), $3, $4, $5, $6)
       ON CONFLICT (mint, minute) DO UPDATE SET
         events = EXCLUDED.events, inflow = EXCLUDED.inflow,
       outflow = EXCLUDED.outflow, wallets = EXCLUDED.wallets`,
      [mint, Number(start), w.events, w.inflow, w.outflow, w.wallets.size],
    );
    windows.delete(key);
  }
}

let lastId = "$";
for (;;) {
  const res = (await redis.xread(
    "COUNT",
    500,
    "BLOCK",
    2000,
    "STREAMS",
    "events",
    lastId,
  )) as [string, [string, string[]][]][] | null;
  for (const [, entries] of res ?? []) {
    for (const [id, fields] of entries) {
      lastId = id;
      const e = JSON.parse(fields[1]);
      const minute = Math.floor(e.ts / 60000) * 60000;
      const key = `${e.mint}|${minute}`;
      const w = windows.get(key) ?? {
        events: 0,
        inflow: 0,
        outflow: 0,
        wallets: new Set<string>(),
      };
      w.events += 1;
      if (e.delta > 0) w.inflow += e.delta;
      else w.outflow += -e.delta;
      w.wallets.add(e.owner);
      windows.set(key, w);
      await pub.publish(
        "live",
        JSON.stringify({
          mint: e.mint,
          minute,
          events: w.events,
          wallets: w.wallets.size,
        }),
      );
    }
  }
  // close every window before the current minute
  await flush(Math.floor(Date.now() / 60000) * 60000);
}

Two production notes. First, this aggregator tracks its position in memory, so a restart starts from new events only. Store lastId in Redis or Postgres to resume where you stopped. Second, if you run several aggregators, split tokens between them (for example by a hash of the mint) so two processes never own the same window.

Step 5: Serve live updates over WebSocket

The aggregator publishes every window update to a Redis channel. A thin server relays them to browsers, and a REST route can read history from Postgres.

serve.tsTypeScript
import Redis from "ioredis";
import { WebSocketServer, WebSocket } from "ws";

const sub = new Redis(process.env.REDIS_URL!);
const wss = new WebSocketServer({ port: 8080 });

await sub.subscribe("live");
sub.on("message", (_channel, message) => {
  for (const client of wss.clients) {
    if (client.readyState === WebSocket.OPEN) client.send(message);
  }
});
console.log("ws://localhost:8080");

Step 6: Backfill and correctness

Live streaming only gives you data from the moment you connect. Plan for three gaps:

  • Backfill. For history, page through getSignaturesForAddress for the program and fetch each transaction, writing to the same raw:tx stream. The decoder cannot tell the difference, which is the point of the design.
  • Duplicates. Reconnects and backfills overlap with live data. Make writes idempotent with the signature as a unique key, as the ON CONFLICT clause above does for windows.
  • Commitment. confirmed data can, rarely, be rolled back. If a figure must be exact, for billing or reporting, recompute it from finalized data in a nightly job.

Step 7: Watch the pipeline itself

A pipeline that falls behind silently is worse than one that crashes. Track three numbers and alert on them:

Metric How to measure Why
Ingest lag Time from receivedAt to now at the decoder Shows whether you keep up with the chain
Pending entries XPENDING raw:tx decoders Grows when decoders are stuck or crashing
Stream length XLEN raw:tx Near your MAXLEN cap means data is about to be dropped

What we would do

Start with Redis Streams, as in this guide. They are simple to run and fast enough for a single program's traffic. Move to Kafka when you need long retention, replay of days of data, or many independent consumers. The stage boundaries stay the same, only the transport changes. Keep raw data for at least as long as you might want to fix a decoding bug and replay.

If you only need a list of specific wallets, the simpler approach in tracking real-time trader activity on Solana may be enough. We design and run pipelines like this in our Solana data and analytics work.

Contact

Need this built for your product?

Tell us what you are building and where you are stuck. Email is the fastest way to reach the team.

contact@onchain.ro