Tracewave: Real-Time Anomaly Detection on a Live FirehoseTracewave

Most data portfolios show a static notebook. Tracewave is a live distributed system: ingestion, a stream bus, windowed stream processing, online ML, time-series storage and a real-time frontend, plus its own observability. It maintains rolling windows over the Wikimedia EventStreams firehose, runs three anomaly detectors at once, and puts an anomaly card on the dashboard with the evidence and a confidence score as soon as something unusual spikes.

The same Processor core runs as one in-memory process in dev or split across Redis and Timescale containers in prod, so it does not depend on the transport. Events fold into 1-second tumbling windows on a supplied clock (deterministic and unit-tested), then three online detectors score them: a rolling z-score, an EWMA control chart, and river's multivariate Half-Space Trees. The ensemble rewards agreement, so 3 of 3 firing corroborates and 1 of 3 is suppressed unless it is very strong. For the "why," it keeps a decaying per-dimension baseline and diffs the spiking window against it, so a card reads "+312 edits, ~98% from en.wikipedia.org, namespace 0, bot actors" instead of just "anomaly."

Open the dashboard and events are flowing, sparklines updating, and anomaly cards arriving with their evidence. Backpressure sheds oldest-first and counts the drops, the ingestor reconnects with backoff and resumes from the last event id, every service exposes Prometheus metrics, and 22 tests cover the windowing and detector math. A deployed link falls back to an in-browser demo stream, so the page always shows data.

“

It is the full data-platform stack end to end, which is also its weakness: there are a lot of moving parts to keep healthy. I would add a schema registry for the firehose sources and rely more on the source registry, so swapping Wikipedia for another stream takes one entry instead of a refactor.