127 of 136 failure patterns had no test

AI-native SDLC, part 2. Agent review and risk-based defect hunting in the CI pipeline of a $4B enterprise data analytics software company: 67 verified defects filed in six weeks, 39 of them P0 or P1. Mandate, problem, deployment, scientific foundation and outcomes.

Share

Share on LinkedInShare on FacebookShare on X

01

The mandate

[CLASSIFIED: client name not cleared for publication.] A $4B enterprise data analytics software company with 1,000+ customers engaged Fast Code AI to find defects in its main product before release rather than after it. Scope: one codebase, 350+ components, ~1.5k pull requests per release. Deliverable: agent review on every pull request and risk-based defect hunts, running in the client's CI. Working rule: no finding enters the client's issue tracker without verification.

02

The problem

Review, testing and release were manual. Peers reviewed each pull request, a regression cycle ran before each release, a go or no-go meeting decided the release, and defects that escaped were found by customers. AI coding tools were already in place for writing code; checking it was unchanged. Our analysis of the client's issue tracker found 136 failure patterns behind past defects. 127 had no test. Four causes recurred. Three in four of the riskiest areas sat where a feature meets another system. 40+ settings with 600+ values ruled out exhaustive testing, so the combinations that had failed before had to be chosen deliberately. Fixes targeted the single defect, leaving the same mistake in neighbouring code untested. Knowledge of the fragile areas sat with senior engineers, not in the suite.

03

The deployment

Phase 1 went live in the client's CI within four months. It moves two decisions from people to agents, with evidence. Agent review runs on every pull request. Risk-based defect hunts target the areas and failure patterns with no test. Every finding is verified before it is filed, and a defect hunt is reported only with a failing test that reproduces the bug. Verified findings go into the client's issue tracker, where the client's engineers set priority. Release decisions stay with people.

Plan
Build
Review
Test
Release
Found by
Beforemanual lifecycle
Tickets and refinement
Engineers code with AI tools
Peers review each pull request
Regression testing before release
Manual go or no-go
Customers, after release
With Fast Codein four months
Tickets and refinement
Engineers code with AI tools
Agent review on every pull request
Risk-based defect hunts, verified with a failing test
People decide, on verified findings
Agents, before release

The work maps onto the layers of the framework in part 1: knowledge (failure patterns and risk areas from the client's issue tracker, matched against its tests), workflow (findings filed into the client's own tracker), agents (review and defect hunting), verification and control (verification before filing, reproducing tests, people deciding releases) and measurement (findings counted only after verification, priority set by the client). Agents run inside the client's CI. Model and infrastructure details are available under executed NDA.

04

Scientific foundation

The published research behind the deployment's main design decisions.

IEEE TSE 2005Predicting the Location and Number of Faults in Large Software Systems

Ostrand, T. J., Weyuker, E. J., Bell, R. M. IEEE Transactions on Software Engineering 31(4).

Its finding that the 20% of files predicted to be most fault-prone held 83% of the faults on average is the basis for hunting defects by risk rather than evenly across 350+ components.

IEEE TSE 2004Software Fault Interactions and Implications for Software Testing

Kuhn, D. R., Wallace, D. R., Gallo, A. M. IEEE Transactions on Software Engineering 30(6).

Its finding that failures are triggered by interactions among a few parameters is why 40+ settings with 600+ values can be tested by choosing combinations deliberately rather than exhaustively.

ICSE 2023Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction

Kang, S., Yoon, J., Yoo, S. IEEE/ACM International Conference on Software Engineering.

Its result, that language models can write a test reproducing a reported bug (for 251 of 750 Defects4J bugs), is what makes a reproducing failing test a workable requirement for every defect hunt.

ICLR 2024Large Language Models Cannot Self-Correct Reasoning Yet

Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., Zhou, D. International Conference on Learning Representations.

Its finding that models struggle to correct their own answers without external feedback is why findings are verified against external evidence, such as a reproducing test, before they are filed.

05

Outcomes

4 months
to Phase 1, live in the client's CI
67
defects filed in the first six weeks
39
of them P0 or P1
76%
of resolved findings confirmed real

Defects filed in the first six weeks: 67, each counted only after verification and entry into the client's tracker. Rated P0 or P1 by the client's engineers: 39. Of 33 findings resolved at the time of writing, 25 were confirmed real (fixed, or duplicates of known defects) and 3 were not defects. Found by: customers, after release, before the deployment; agents, before release, after it. Status: active, Phase 1 live in the client's CI.

Source: engagement records and the client's issue tracker.

Read next

Get your pipeline audited

Fast Code AI takes frontier AI into production for enterprises, with teams in India, Germany and the UAE. We will audit your pipeline inside your environment, find the failure patterns your suite doesn't cover, and measure everything against your own baseline. What an audit covers, or ask for one.

Want to know more about AI ML Technology

Incorporate AI ML into your workflows to boost efficiency, accuracy, and productivity. Discover our artificial intelligence services.

Read More Blogs

View All

arrow right
logologo
Request an AI summary of Fast Code AI
  • Head Office
  • Jbr Tech Park, Plot No. 77,
    6th Rd, EPIP Zone,
    Whitefield, Bengaluru,
    Karnataka 560066
  • UAE Office
  • First Floor, Incubator Building,
    Masdar City SE4505,
    Abu Dhabi
  • German Office
  • Königstraße 26
    Stuttgart 70173
  • Email
  • arjun@fastcode.ai
  • Phone
  • +91 85530 38132

© Copyright Fast Code AI 2026. All Rights Reserved