Scaling Fintech Infrastructure: Deterministic Logic in High-Frequency Trading

Most financial platforms don't fail because of bad code. They fail because of non-deterministic execution paths — race conditions in order matching, floating-point drift in risk calculations, and retry storms that cascade through monolithic services during market volatility. When we inherited a legacy trading platform processing 800 orders/second with a 12% dropped-transaction rate, the root cause wasn't performance. It was architecture.

This is a technical breakdown of the rebuild: event-sourced microservices, deterministic execution guarantees, and a data pipeline that reduced p99 latency from 340ms to 18ms.

The Problem: Non-Deterministic Execution Paths

The legacy system was a monolithic Java application backed by Oracle RAC. Order matching happened in-process with optimistic locking. Under normal load, this worked. During market spikes — earnings releases, macro events — the system hit lock contention walls. Orders queued in JVM heap, GC pauses introduced 200ms+ stalls, and the retry logic (exponential backoff with jitter) created thundering-herd effects that amplified the original bottleneck.

Three critical failures:

Architecture: Event-Sourced Microservices

We decomposed the monolith into five bounded contexts: Order Intake, Matching Engine, Risk Gateway, Settlement, and Compliance Ledger. Each context owns its data. Communication is strictly asynchronous via Apache Kafka with exactly-once semantics (idempotent producers + transactional consumers).

Every state transition is an immutable event. Orders don't get "updated" — they emit OrderPlaced, OrderValidated, OrderMatched, OrderSettled events. The current state is a left-fold over the event stream.

// Order aggregate — event-sourced with Zod validation
import { z } from 'zod';

const OrderEventSchema = z.discriminatedUnion('type', [
  z.object({
    type: z.literal('OrderPlaced'),
    orderId: z.string().uuid(),
    symbol: z.string().min(1).max(12),
    side: z.enum(['BUY', 'SELL']),
    quantity: z.number().int().positive(),
    price: z.bigint(), // cents — no floating point
    timestamp: z.number().int(),
  }),
  z.object({
    type: z.literal('OrderMatched'),
    orderId: z.string().uuid(),
    matchId: z.string().uuid(),
    fillPrice: z.bigint(),
    fillQuantity: z.number().int().positive(),
    timestamp: z.number().int(),
  }),
]);

type OrderEvent = z.infer<typeof OrderEventSchema>;

function applyEvent(state: OrderState, event: OrderEvent): OrderState {
  switch (event.type) {
    case 'OrderPlaced':
      return {
        ...state,
        status: 'OPEN',
        remainingQty: event.quantity,
        price: event.price,
      };
    case 'OrderMatched':
      const remaining = state.remainingQty - event.fillQuantity;
      return {
        ...state,
        status: remaining === 0 ? 'FILLED' : 'PARTIAL',
        remainingQty: remaining,
        fills: [...state.fills, {
          matchId: event.matchId,
          price: event.fillPrice,
          quantity: event.fillQuantity,
        }],
      };
  }
}

Two critical decisions here. First, all monetary values use bigint representing cents (or basis points for rates). No IEEE 754 anywhere in the pipeline. Second, every event is validated at the boundary with Zod schemas before entering the event store. Malformed events are dead-lettered, not silently dropped.

The Risk-Limit Engine

The matching engine doesn't execute orders — it proposes matches. Every proposed match passes through a custom risk-limit engine that evaluates position exposure, margin requirements, and circuit-breaker thresholds in under 2ms.

# Risk evaluation — deterministic, pure function
from decimal import Decimal, ROUND_HALF_EVEN
from dataclasses import dataclass

@dataclass(frozen=True)
class RiskCheckResult:
    approved: bool
    exposure_delta: Decimal
    margin_utilization: Decimal
    breach_reason: str | None

def evaluate_risk(
    position: 'PositionSnapshot',
    proposed_fill: 'FillProposal',
    limits: 'AccountLimits',
) -> RiskCheckResult:
    # All arithmetic uses Decimal with explicit rounding
    new_exposure = position.net_exposure + (
        Decimal(str(proposed_fill.quantity))
        * Decimal(str(proposed_fill.price))
        / Decimal('100')  # cents to dollars
    ).quantize(Decimal('0.01'), rounding=ROUND_HALF_EVEN)

    margin_util = (new_exposure / limits.max_exposure).quantize(
        Decimal('0.0001'), rounding=ROUND_HALF_EVEN
    )

    if margin_util > Decimal('0.95'):
        return RiskCheckResult(
            approved=False,
            exposure_delta=new_exposure - position.net_exposure,
            margin_utilization=margin_util,
            breach_reason=f'MARGIN_BREACH: {margin_util} > 0.95 threshold',
        )

    return RiskCheckResult(
        approved=True,
        exposure_delta=new_exposure - position.net_exposure,
        margin_utilization=margin_util,
        breach_reason=None,
    )

The function is pure — no I/O, no side effects, no mutable state. Position snapshots are materialized from the event stream. This makes the risk engine fully testable with property-based testing: we generate thousands of random order sequences and assert that no approved fill ever violates limits.

Data Layer: PostgreSQL WAL + CQRS with Redis Projections

The event store runs on PostgreSQL with logical replication. Events are appended to a partitioned table (partitioned by date, 90-day retention in hot storage). A Change Data Capture pipeline reads the WAL and projects read-optimized views into Redis.

The write path: Kafka → event store (PostgreSQL, append-only). The read path: WAL → CDC consumer → Redis Sorted Sets / Hashes. Trading dashboards, position summaries, and order books all read from Redis. Zero direct queries against the event store during trading hours.

// CDC consumer — WAL to Redis projection
async function projectOrderBook(event: OrderEvent): Promise<void> {
  const pipeline = redis.pipeline();

  if (event.type === 'OrderPlaced') {
    // Sorted set: price as score, orderId as member
    const key = `orderbook:${event.symbol}:${event.side}`;
    pipeline.zadd(key, Number(event.price), event.orderId);
    pipeline.hset(`order:${event.orderId}`, {
      symbol: event.symbol,
      side: event.side,
      price: event.price.toString(),
      remainingQty: event.quantity.toString(),
      status: 'OPEN',
    });
  }

  if (event.type === 'OrderMatched') {
    pipeline.hset(`order:${event.orderId}`, {
      remainingQty: event.fillQuantity.toString(),
      status: event.fillQuantity === 0 ? 'FILLED' : 'PARTIAL',
    });
  }

  await pipeline.exec();
}

This separation collapsed p99 read latency from 340ms to 18ms. The PostgreSQL instance now handles only sequential appends — the workload it's optimized for. Redis handles the fan-out to hundreds of concurrent dashboard connections via pub/sub.

Results

After 90 days in production with staged rollout across three trading desks:

The system has processed over 2.1 billion events without data loss. The event store grows at approximately 4.2 TB/month, managed through tiered storage: hot (NVMe, 90 days), warm (SSD, 1 year), cold (S3-compatible object storage, indefinite).

The core insight isn't about technology choices — Kafka, PostgreSQL, and Redis are well-understood tools. The insight is that financial systems demand determinism as an architectural constraint, not a feature. When every state transition is an immutable, validated, ordered event, the entire class of problems that plagued the legacy system — race conditions, phantom states, unauditable mutations — simply cannot exist.