Back to Public Sector

The Testing Thesis

AI-powered testing for legacy systems and migration risk

Legacy government systems need a different kind of test platform. The one you build for a greenfield web app — Playwright, snapshot diffs, a CI pipeline — falls apart the moment you aim it at a 30-year-old benefits application with no prior test coverage and a retiring team that holds the business logic in their heads. This page is about the kind of testing that actually works on those systems, and the two platforms in production right now that prove the thesis.

Cossette Consulting LLC · Veteran-Owned Texas LLC · Current engagement: Multnomah County, OR

The pattern

Index. Capture. Generate. Verify. Gate.

Index

Read the legacy codebase into something an AI can reason over.

Capture

Extract the behavioral knowledge the retiring SMEs hold.

Generate

Produce draft specs grounded in the real DOM and the real logic.

Verify

Run adversarial cross-model review. One model writes; another audits.

Gate

Human review at every decision point. No unreviewed test runs.

In production

Two platforms, same thesis

Behavioral test capture

Vantage

Rikki spent 36 years learning Multnomah County's Universal Client Registry. Before she retired, Vantage captured what she knew into 208 behavioral Playwright tests the modernized app must pass before it ships — her institutional knowledge, converted into an acceptance contract no future rebuild can violate. Multi-tenant from day one; UCR is tenant 1, ACHP onboards next.

718
Locked tests
309
Workflows
924
SME sessions
Read the Vantage case study

Migration validation

FDM Snowflake

Multnomah County's SQL Server → Snowflake financial data-mart migration (Workday / Questica sources). A six-agent Claude Code pipeline grounds in the source schema, generates validation scripts, runs adversarial cross-model review via Gemini, verifies against live data three times, and triages defects. Delivered the County's build-vs-buy decision to standardize on vendor-native tooling.

~200
Tables
6
Agents
3×
Review
View on program hub

Why this is a specialty

What makes legacy testing hard

  • The code is thirty years of accumulated decisions

    UCR is three decades of stored procedures, trigger chains, and business logic by people who moved on. An index that can serve a greenfield app can't serve this; the AI has to read, cluster, and annotate in a form the test generators can actually use.

  • The knowledge lives in humans, not documents

    The analyst who's run the benefits workflow for 36 years knows which fields matter in which order. No document captures that. Extracting it structured enough to compile into tests requires a session-capture workflow, typed extraction, and SME confirmation — not flat RAG.

  • Rebuilds break what people quietly depend on

    A modernized UCR will replace stored procedures with services, change query shapes, rename columns. The tests are the contract that says: whatever you change underneath, the workflows the humans depend on must still behave the same way. 208 Playwright tests at 98.1% compile-clean is the UCR rebuild's acceptance gate.

  • Migrations need validation, not just generation

    FDM Snowflake isn't a web app test platform; it's a data validation platform. One model generates validation scripts. Another (different model family) audits them adversarially. The live run happens three times, with defect triage before the generated validations are allowed to stand. That multi-model routing is the architecture, not a feature.

  • Human review is the design, not the exception

    Every test enters the locked suite through a human gate. Every self-healed selector proposal is reviewed before it's applied. The AI accelerates the authoring and the healing; the humans remain accountable for what passes. This is the governance pattern that makes the work defensible under audit.

The Vantage fleet

Eight agents, one loop

Five agents generate and curate tests; three operate the platform and keep the suite running as the application evolves. The count is fixed — a ninth requires an Architecture Decision Record. Every one is live and running in production against Multnomah County's Universal Client Registry.

Test-authoring agents · produce a reviewable test

CODEX

Live

Codebase intelligence. Indexes symbols from the application's source, detects drift when code changes, and answers structural queries. Other agents query CODEX rather than parsing the codebase themselves.

typed primitives · find_symbol · impact_of

TESTGEN

Live

Test generation. Produces Playwright specs from code analysis, SME knowledge, and corpus material. Outputs a draft a human curates — never a test that runs unreviewed.

DOM-grounded PageObjects

KNOWLEDGE

Live

SME capture. Extracts typed rows from recordings, documents, and meeting notes. Feeds TESTGEN's generators and CODEX's annotations.

typed extraction, not flat RAG

GUARDIAN

Live

Regression watch. Runs the locked suite against the live application, streams results, and detects regressions. Orphan reaper for runs that never reach a terminal state.

queued runner · heartbeat · run-status machine

INTEGRATION

Live

Vendor seam and contract testing — the boundary between the application and external systems. Shipped alongside the first legacy→modernized migration tenant.

contract tests at the legacy seam

Operate / heal / orchestrate · keep the suite running as the app lives

MEDIC

Live

Self-healing selectors. When GUARDIAN catches a selector failure, MEDIC reads the DOM snapshot at the failure point and proposes a PageObject diff for human review — rather than failing the test and waiting for someone to notice.

DOM-diff proposals · heal-on-review loop

OPERATOR

Live

Orchestration and per-tenant configuration. Watches the other agents, applies deterministic rules, routes decisions to the right human, and manages tenant config and briefing cadence. Each tenant has its own.

one conversational cockpit per application

VANTAGE ENGINEER

Live

Platform-tier roll-up. Summarizes what changed across every tenant, surfaces what needs a decision, and gives the human a cockpit view across the fleet.

cross-tenant briefings · platform-wide state

The loop

The test-authoring and run loop

The agents exist to serve one loop. The locked-green count is the metric it produces: how many tests have been verified against the live application and are currently passing. The loop is human-in-the-loop at both decision points — a draft that has not been reviewed is never run as part of the locked suite, and a heal that has not been approved is never applied.

The Vantage test loopA source change in the application under test drives CODEX re-indexing, a recorded drift event, TESTGEN draft generation, human review, locking into the suite, a GUARDIAN run, and a MEDIC heal proposal when a selector breaks. Results update the locked-green trust signal.Source change in the application under testCODEXre-indexes the changed filesDrift event recordedwhat changed · what is affectedTESTGENgenerates a draft spec, grounded in thereal DOM via PageObjectsHUMAN REVIEWconfirm · edit · rejectConfirmed spec is LOCKED into the suitean unreviewed draft never runs in the locked suiteGUARDIANruns the locked suite against the live appIF A SELECTOR BREAKSMEDICproposes a heal — a PageObject diffHUMAN approves or rejects the healResults recorded — trust signal updatesthe locked-green countdrift re-enters the loophuman gatehuman gate

Multi-tenant boundary

One platform, many applications

Every tenant-partitioned table carries a tenant_id UUID; PostgreSQL row-level security enforces isolation. An unscoped read returns zero rows rather than another tenant's data — verified live across 78 of 80 tenant-partitioned tables under FORCE RLS.

TenantApplication
UCR LegacyUniversal Client Registry — the county's case-management system
ACHPAdult Care Home Program — licensing and inspections

Substrate

The stack underneath

LayerTechnology
AgentsPython 3.12
FrontendReact 18 · Vite · TypeScript
DatabasePostgreSQL · Cloud SQL · RLS
Test runnerPlaywright
HostingGoogle Cloud Run
ModelVertex AI (Claude / Gemini)

If you're writing an RFP that uses the word “testing”, “validation”, or “migration”

Start with a 30-minute working session. Describe the system, the retiring team, and the deadline. The firm assesses fit against current program commitments and recommends the shape of the work — Architecture & Build Sprint for a scoped platform, flagship Embedded Program Lead for an ongoing one, or a referral elsewhere if the fit isn't right. You receive a written summary within two business days you can forward internally.