An AI-native pipeline moves people from making every call to governing how calls are made.
This part sets out the framework: five maturity levels, five layers, and how each layer is built. Part 2 shows it at work in one enterprise's pipeline; part 3 compares it with the other ways to get there.
01 · Maturity model
The five maturity levels
Most enterprise pipelines sit at level 1 or 2; the five questions below tell you which. Level 3 is the turning point, where agents start to do the work and people move to approving it.
People make every judgement call.
AI suggests. A person reads every output.
Agents propose. People check every output.
Agents run full stages with evidence. People approve at gates.
Outcomes update knowledge and tests. The pipeline improves itself.
Requires evidence on every action, explicit permissions and accurate context.
Requires every miss, rejection and incident to flow back into knowledge and tests.
| Level | Who decides | What it looks like when a build fails |
|---|---|---|
| L1 Scripted | People, on everything | An engineer reads the logs and works out the cause |
| L2 AI-assisted | People, helped by AI | An AI summary of the logs; the engineer still decides |
| L3 Agent-augmented | Agents propose, people check each one | The agent suggests "flaky, re-run"; the engineer checks and re-runs |
| L4 Agent-operated, human-gated | Agents act within limits | The agent confirms it is flaky, re-runs, quarantines and notifies the owner |
| L5 Closed loop | People set policy | The same, and the pattern updates the knowledge store so similar failures are caught earlier |
02 · Where you stand
Five questions your pipeline should be able to answer
Each one is a decision people make by hand today. If you can't answer two of them from your own records, your pipeline is at level 1 or 2 of the maturity model above, and that is where an audit starts.
- What share of last quarter's red builds were re-run without anyone finding the cause?Re-run until green is a decision; nobody records why it was taken.
- How many of last quarter's escaped defects repeated a failure pattern you had seen before?Fixes target the single defect; the same mistake in neighbouring code stays untested.
- How long does a failing build take to reach the person who can fix it?CODEOWNERS says who owns a file, not who fixed this kind of failure last time.
- Do your riskiest areas, where one feature meets another system, have a test for the last thing that broke there?In our case study, three in four of the riskiest areas sat exactly there.
- Which of your last ten release decisions rested on evidence, and which on a meeting?Go or no-go is usually the most expensive judgement in the pipeline, and the least recorded.
03 · Reference architecture
Five layers that make the pipeline AI-native
Agents are one layer. Without the others, they produce noise that people learn to ignore.
Outcomes from Layer 05 continuously update the knowledge store in Layer 01.
Product structure, where defects cluster, rules that must hold, and test and failure history.
Triggered by pipeline events; findings reach the right owner in the tools teams already use.
Narrow agents for review, test selection, triage, fixes, release evidence and deployment checks.
Evidence on every decision, explicit permissions, approval gates and an audit trail.
Metrics against a baseline, with every outcome fed back into the knowledge store.
Sections 04 to 07 cover the knowledge store, agents, guardrails and evals. What we build and what we reuse is in part 3.
Examples in each layer are illustrative, not an exhaustive list.
04 · Layer 01 · Knowledge store
How the knowledge store is built
It combines four kinds of knowledge, each stored in the form that suits it. Agents query each one through tools when a task needs it.
| Store | Holds | Built with |
|---|---|---|
| Code graph | Files, symbols, dependencies, build targets, test coverage | Code indexers and build-system queries; updated every merge |
| History | Runs, failures, flaky record, defects, releases | Tables in your database; ingested continuously |
| Curated rules | Product map, risk areas, failure patterns, intended behaviour | Versioned files in git, owner-reviewed each release |
| Documents | Tickets, design docs, runbooks, incidents | Keyword plus embedding search; indexed nightly |
Mapping code to product intent and signal flow
Signals come from interface definitions, not inferred from code. Agents can then answer which requirements a change affects, who consumes a modified signal, and which paths have no test, and cite requirement IDs as traceability evidence.
05 · Layers 02 and 03 · Workflow and agents
How agents are orchestrated and routed
Routine steps such as fetching logs or bisecting commits are written as ordinary code. Agents handle only the decisions that need reasoning, which keeps runs cheaper and easier to debug.
How agents communicate
Agents never call each other directly; the orchestrator assigns and collects.
Findings, evidence and verdicts in fixed schemas; malformed output is rejected.
A separate agent, without the first one's reasoning, tries to disprove each finding.
Parallel sub-agents within a fixed cap per run.
Orchestration choice
Fast to prototype; extra abstraction is harder to debug.
Tool use and sub-agents built in; tied to one provider unless wrapped.
Calls a provider SDK behind a model-agnostic interface.
How building this compares with buying a product is in part 3.
Routing beyond CODEOWNERS
Usually the strongest signal for a given failure pattern.
Recent changes to the files and functions involved.
CODEOWNERS, tracker components and on-call, as the fallback.
So boundary failures do not bounce between teams.
Routing recommends an owner and team with evidence; it does not score individuals. Where works council rules apply, person-level data stays within agreed bounds.
06 · Layer 04 · Guardrails and kill switches
Guardrails and kill switches
Every agent action passes through a single gateway between the agents and your systems. Permissions, limits and kill switches are enforced there, in code.
| Permission tier | Agent may | Moves up when |
|---|---|---|
| T0 Observe | Decide silently, in shadow mode | Accuracy against people meets the agreed bar |
| T1 Suggest | Comment, open draft changes | Engineers act on most suggestions |
| T2 Act, reversible | Re-run, quarantine, revert on main | People rarely reverse its actions |
| T3 Irreversible | Never alone: release, production deploy | Stays with a named person |
Stopping agents instantly
A flag the gateway checks before every action; in-flight runs are cancelled.
Same mechanism, finer grain; the rest keeps running.
Rising reversals, errors or cost drop an agent to suggest-only.
Batch rollback by action ID; revoking service accounts as last resort.
Also enforced in code: per-agent tool allow-lists, short-lived scoped credentials, and untrusted text (logs, PRs, tickets) treated as data, never as instructions.
07 · Layer 05 · KPIs and evals
Measuring agents and their outcomes
We track pipeline outcomes each release, test every agent before it ships, and monitor it while it runs.
| Level | Examples | When |
|---|---|---|
| Outcomes | DORA metrics, time to triage, pipeline time, escaped defects, cost per change | Each release, against baseline |
| Task evals | Accuracy on labelled past cases from your history, per agent | Every change to prompts, models or tools |
| Operational | Acceptance with reasons, reversals, evidence completeness, latency, cost | Continuously, with drift alerts |
Generic tests every agent passes
Valid schema, valid tool calls, within budget.
Out-of-scope actions are blocked.
Missing or broken data gives "cannot decide".
Planted instructions change nothing.
Known cases keep their answers.
Repeat runs reach the same decision.
Measuring hallucination
A hallucination is any claim its evidence does not support. Every finding must cite evidence, so it can be checked:
Cited files, lines, logs and requirements exist and say what is claimed.
Twice from a clean environment, with a failing test.
A checker agent tries to disprove it.
Regular audits calibrate automated checks.
Read next
Agent review and defect hunting in a $4B company's CI pipeline: 67 verified defects in six weeks.
Part 3 · How we compareWhere each type of provider fitsAI code review and context platforms, hyperscalers, building in-house and Fast Code AI, and how an engagement starts.
Find out where your pipeline stands
Most enterprise pipelines sit at level 1 or 2. Fast Code AI takes frontier AI into production for enterprises, with teams in India, Germany and the UAE. We will audit your pipeline inside your environment, place each stage on these five levels from your own records, and show you where agents would pay off first. See what an audit finds, or ask for one.