01
The mandate
[CLASSIFIED: client name not cleared for publication.] A $4B enterprise data analytics software company with 1,000+ customers engaged Fast Code AI to find defects in its main product before release rather than after it. Scope: one codebase, 350+ components, ~1.5k pull requests per release. Deliverable: agent review on every pull request and risk-based defect hunts, running in the client's CI. Working rule: no finding enters the client's issue tracker without verification.
02
The problem
Review, testing and release were manual. Peers reviewed each pull request, a regression cycle ran before each release, a go or no-go meeting decided the release, and defects that escaped were found by customers. AI coding tools were already in place for writing code; checking it was unchanged. Our analysis of the client's issue tracker found 136 failure patterns behind past defects. 127 had no test. Four causes recurred. Three in four of the riskiest areas sat where a feature meets another system. 40+ settings with 600+ values ruled out exhaustive testing, so the combinations that had failed before had to be chosen deliberately. Fixes targeted the single defect, leaving the same mistake in neighbouring code untested. Knowledge of the fragile areas sat with senior engineers, not in the suite.
03
The deployment
Phase 1 went live in the client's CI within four months. It moves two decisions from people to agents, with evidence. Agent review runs on every pull request. Risk-based defect hunts target the areas and failure patterns with no test. Every finding is verified before it is filed, and a defect hunt is reported only with a failing test that reproduces the bug. Verified findings go into the client's issue tracker, where the client's engineers set priority. Release decisions stay with people.
The work maps onto the layers of the framework in part 1: knowledge (failure patterns and risk areas from the client's issue tracker, matched against its tests), workflow (findings filed into the client's own tracker), agents (review and defect hunting), verification and control (verification before filing, reproducing tests, people deciding releases) and measurement (findings counted only after verification, priority set by the client). Agents run inside the client's CI. Model and infrastructure details are available under executed NDA.
04
Scientific foundation
The published research behind the deployment's main design decisions.
Its finding that the 20% of files predicted to be most fault-prone held 83% of the faults on average is the basis for hunting defects by risk rather than evenly across 350+ components.
Its finding that failures are triggered by interactions among a few parameters is why 40+ settings with 600+ values can be tested by choosing combinations deliberately rather than exhaustively.
Its result, that language models can write a test reproducing a reported bug (for 251 of 750 Defects4J bugs), is what makes a reproducing failing test a workable requirement for every defect hunt.
Its finding that models struggle to correct their own answers without external feedback is why findings are verified against external evidence, such as a reproducing test, before they are filed.
05
Outcomes
Defects filed in the first six weeks: 67, each counted only after verification and entry into the client's tracker. Rated P0 or P1 by the client's engineers: 39. Of 33 findings resolved at the time of writing, 25 were confirmed real (fixed, or duplicates of known defects) and 3 were not defects. Found by: customers, after release, before the deployment; agents, before release, after it. Status: active, Phase 1 live in the client's CI.
Source: engagement records and the client's issue tracker.
Read next
What makes a pipeline AI-native, and the guardrails and evals that keep its agents safe.
Part 3 · How we compareWhere each type of provider fitsAI code review and context platforms, hyperscalers, building in-house and Fast Code AI, and how an engagement starts.
Get your pipeline audited
Fast Code AI takes frontier AI into production for enterprises, with teams in India, Germany and the UAE. We will audit your pipeline inside your environment, find the failure patterns your suite doesn't cover, and measure everything against your own baseline. What an audit covers, or ask for one.