Overview

A nightly IP regression runs dozens of EDA tools across a dependency graph, each spewing gigabyte logs where a tool can exit 0 and still have failed. GAUNTLET rebuilds that loop end to end — flow execution, log parsing, waivers, and run-to-run diff — refusing to fake the hard parts. It's pure Python 3.10+ with a single dependency (PyYAML): no network, no services, no Docker; real subprocesses, real signals, real exit codes, logs as files on disk.

Tech Stack

Challenges

• A tool can exit 0 and still have failed — exit codes under-report, so the log has to be the source of truth.
• Running a dependency graph of tools in parallel while cascading failures correctly to dependents.
• Killing a timed-out step cleanly when a shell wrapper can die while its child simulator lives on.
• Distinguishing a known, owned failure from a new regression without silently hiding either.

Solution

A ready-set scheduler over a ThreadPoolExecutor runs a step only when every dependency is PASS/WAIVED; failures cascade SKIPPED transitively, and --keep-going lets independent branches finish. Logs are streamed line-by-line against compiled regex rulepacks (never read whole — real EDA logs run to gigabytes), so a clean exit code with an unwaived ERROR still FAILs. Waivers are rule_id + substring matches with an owner and reason — a waived failure is a visible WAIVED status, never a silent pass. Steps launch in their own process group; on timeout the whole group is signalled, graced, then hard-killed, cross-platform. Everything persists to SQLite, so history, report and diff work from the DB alone.

Outcome

The money shot is the diff: introduce a flake, run again, and gauntlet diff buckets NEW FAILURES / FIXED / STILL FAILING and exits non-zero on regressions, so CI can gate on the delta rather than any single run's pass/fail. 43 tests run fully offline in ~3s, cross-platform; a self-contained HTML report ships with zero new dependencies; and flaky steps that only pass on retry are marked as such in every report.

What I'd do differently

The design discipline was refusing to fake the parts that make regression infra hard: WAIVED is a distinct status, not a silent pass; SKIPPED isn't a failure in diffs (the diff blames the upstream step that actually failed); and verifiable facts like status never get left to a heuristic. Small enough to read, honest enough to trust.