The Testing Thesis
AI-powered testing for legacy systems and migration risk
Legacy government systems need a different kind of test platform. The one you build for a greenfield web app — Playwright, snapshot diffs, a CI pipeline — falls apart the moment you aim it at a 30-year-old benefits application with no prior test coverage and a retiring team that holds the business logic in their heads. This page is about the kind of testing that actually works on those systems, and the two platforms in production right now that prove the thesis.
Cossette Consulting LLC · Veteran-Owned Texas LLC · Current engagement: Multnomah County, OR
The pattern
Index. Capture. Generate. Verify. Gate.
Index
Read the legacy codebase into something an AI can reason over.
Capture
Extract the behavioral knowledge the retiring SMEs hold.
Generate
Produce draft specs grounded in the real DOM and the real logic.
Verify
Run adversarial cross-model review. One model writes; another audits.
Gate
Human review at every decision point. No unreviewed test runs.
In production
Two platforms, same thesis
Behavioral test capture
Vantage
Rikki spent 36 years learning Multnomah County's Universal Client Registry. Before she retired, Vantage captured what she knew into 208 behavioral Playwright tests the modernized app must pass before it ships — her institutional knowledge, converted into an acceptance contract no future rebuild can violate. Multi-tenant from day one; UCR is tenant 1, ACHP onboards next.
Migration validation
FDM Snowflake
Multnomah County's SQL Server → Snowflake financial data-mart migration (Workday / Questica sources). A six-agent Claude Code pipeline grounds in the source schema, generates validation scripts, runs adversarial cross-model review via Gemini, verifies against live data three times, and triages defects. Delivered the County's build-vs-buy decision to standardize on vendor-native tooling.
Why this is a specialty
What makes legacy testing hard
The code is thirty years of accumulated decisions
UCR is three decades of stored procedures, trigger chains, and business logic by people who moved on. An index that can serve a greenfield app can't serve this; the AI has to read, cluster, and annotate in a form the test generators can actually use.
The knowledge lives in humans, not documents
The analyst who's run the benefits workflow for 36 years knows which fields matter in which order. No document captures that. Extracting it structured enough to compile into tests requires a session-capture workflow, typed extraction, and SME confirmation — not flat RAG.
Rebuilds break what people quietly depend on
A modernized UCR will replace stored procedures with services, change query shapes, rename columns. The tests are the contract that says: whatever you change underneath, the workflows the humans depend on must still behave the same way. 208 Playwright tests at 98.1% compile-clean is the UCR rebuild's acceptance gate.
Migrations need validation, not just generation
FDM Snowflake isn't a web app test platform; it's a data validation platform. One model generates validation scripts. Another (different model family) audits them adversarially. The live run happens three times, with defect triage before the generated validations are allowed to stand. That multi-model routing is the architecture, not a feature.
Human review is the design, not the exception
Every test enters the locked suite through a human gate. Every self-healed selector proposal is reviewed before it's applied. The AI accelerates the authoring and the healing; the humans remain accountable for what passes. This is the governance pattern that makes the work defensible under audit.
The Vantage fleet
Eight agents, one loop
Five agents generate and curate tests; three operate the platform and keep the suite running as the application evolves. The count is fixed — a ninth requires an Architecture Decision Record. Every one is live and running in production against Multnomah County's Universal Client Registry.
Test-authoring agents · produce a reviewable test
CODEX
LiveCodebase intelligence. Indexes symbols from the application's source, detects drift when code changes, and answers structural queries. Other agents query CODEX rather than parsing the codebase themselves.
typed primitives · find_symbol · impact_of
TESTGEN
LiveTest generation. Produces Playwright specs from code analysis, SME knowledge, and corpus material. Outputs a draft a human curates — never a test that runs unreviewed.
DOM-grounded PageObjects
KNOWLEDGE
LiveSME capture. Extracts typed rows from recordings, documents, and meeting notes. Feeds TESTGEN's generators and CODEX's annotations.
typed extraction, not flat RAG
GUARDIAN
LiveRegression watch. Runs the locked suite against the live application, streams results, and detects regressions. Orphan reaper for runs that never reach a terminal state.
queued runner · heartbeat · run-status machine
INTEGRATION
LiveVendor seam and contract testing — the boundary between the application and external systems. Shipped alongside the first legacy→modernized migration tenant.
contract tests at the legacy seam
Operate / heal / orchestrate · keep the suite running as the app lives
MEDIC
LiveSelf-healing selectors. When GUARDIAN catches a selector failure, MEDIC reads the DOM snapshot at the failure point and proposes a PageObject diff for human review — rather than failing the test and waiting for someone to notice.
DOM-diff proposals · heal-on-review loop
OPERATOR
LiveOrchestration and per-tenant configuration. Watches the other agents, applies deterministic rules, routes decisions to the right human, and manages tenant config and briefing cadence. Each tenant has its own.
one conversational cockpit per application
VANTAGE ENGINEER
LivePlatform-tier roll-up. Summarizes what changed across every tenant, surfaces what needs a decision, and gives the human a cockpit view across the fleet.
cross-tenant briefings · platform-wide state
The loop
The test-authoring and run loop
The agents exist to serve one loop. The locked-green count is the metric it produces: how many tests have been verified against the live application and are currently passing. The loop is human-in-the-loop at both decision points — a draft that has not been reviewed is never run as part of the locked suite, and a heal that has not been approved is never applied.
Multi-tenant boundary
One platform, many applications
Every tenant-partitioned table carries a tenant_id UUID; PostgreSQL row-level security enforces isolation. An unscoped read returns zero rows rather than another tenant's data — verified live across 78 of 80 tenant-partitioned tables under FORCE RLS.
| Tenant | Application |
|---|---|
| UCR Legacy | Universal Client Registry — the county's case-management system |
| ACHP | Adult Care Home Program — licensing and inspections |
Substrate
The stack underneath
| Layer | Technology |
|---|---|
| Agents | Python 3.12 |
| Frontend | React 18 · Vite · TypeScript |
| Database | PostgreSQL · Cloud SQL · RLS |
| Test runner | Playwright |
| Hosting | Google Cloud Run |
| Model | Vertex AI (Claude / Gemini) |
If you're writing an RFP that uses the word “testing”, “validation”, or “migration”
Start with a 30-minute working session. Describe the system, the retiring team, and the deadline. The firm assesses fit against current program commitments and recommends the shape of the work — Architecture & Build Sprint for a scoped platform, flagship Embedded Program Lead for an ongoing one, or a referral elsewhere if the fit isn't right. You receive a written summary within two business days you can forward internally.