Variance Control Comparison Across Delivery Modes

2026-05-16 · benchmarks-eval cost-performance agentic-ai tools-infrastructure · medium · source → · wiki →

key claims

No public source located for this item provides a task-complexity-normalised, post-pipeline, head-to-head production failure distribution that directly compares AI-assisted code changes with AI-agent-executed business-process actionsDevOps (2024)Liang et al. (2025)Zhou et al. (2024)Xiv (2024)
Build-mode failure is most visibly measured as post-release intervention after release gates, while Site Reliability Engineering guidance supports the inference that equivalent failures can also be blocked before production by system-level testsDora (n.d.)Google (n.d.)
Public evidence on AI-assisted coding shows elevated before-release security and defect risk, but it also shows that static-analysis feedback can eliminate a substantial share of those issues before release, which makes build-mode variance materially filterable before external effects occurLiang et al. (2025)Google (n.d.)
The 2024 DORA evidence shows local development gains alongside lower delivery stability and throughput, which supports the inference that AI assistance still requires small batches, robust tests, and disciplined review if teams want low escaped-failure ratesCloud (2024)DevOps (2024)
When compared with gated build-mode proxies, realistic do-mode benchmarks indicate materially lower baseline reliability, because WebArena reports 14.41% GPT-4 success, tau-bench reports fewer than 50% successful tasks and less than 25% reliability across repeated retail trials, and AgentDojo still reports many failures even without attacksZhou et al. (2024)Xiv (2024)Debenedetti et al. (2024)Dora (n.d.)Google (n.d.)
Do-mode failures are harder to catch before impact because many of them arise from live tool use, policy misapplication, excessive permissions, prompt injection, or wrong action selection, all of which often become fully visible only after an external step has already been takenOwasp (n.d.)Ruan et al. (2024)Kambhampati et al. (2024)News (2024)
The strongest public legal and regulatory material places autonomous process execution inside lifecycle risk-management and operational-resilience disciplines, which supports the inference that teams cannot govern it only as a conventional pre-release software-quality problemUnion (2024)Supervision (2021)National (2023)
The best-supported comparative conclusion is therefore that build mode is structurally more controllable than do mode, because the dominant control point in build mode is release gating before deployment while the dominant control point in do mode must remain a runtime control layer around live decisions and actionsAnthropic (2024)Google (n.d.)Owasp (n.d.)Supervision (2021)

Research Question

What is the empirical failure-rate distribution of Artificial Intelligence (AI)-assisted code that has passed a standard software delivery pipeline compared with AI-agent-executed business processes at comparable task complexity, and what proportion of failures in each mode is detectable before external effects occur?

Findings

Executive Summary

Public evidence does not currently provide a task-complexity-normalised head-to-head failure distribution, but the strongest available proxy comparison indicates that AI-assisted code that clears a standard gated delivery pipeline is more pre-effect-detectable than AI agents executing business processes directly, and it points toward, rather than directly proves, fewer externally realised failures.

Build mode is buffered by structural controls, tests, review, staged release, and deployment blocking, so many failures are converted into pre-production findings; public AI-specific evidence shows elevated before-release defect and security risk but also shows that static-analysis feedback can remediate a substantial share of those issues before release.

Do mode faces much lower baseline end-to-end reliability on realistic multi-step tasks, with WebArena reporting 14.41% GPT-4 success, tau-bench reporting fewer than 50% successful tasks and less than 25% reliability across repeated retail trials, and ToolEmu still finding severe failures 23.9% of the time even for the safest tested agent.

For regulated financial services, this means build mode can usually be governed through existing release-management and incident processes, whereas do mode needs runtime least privilege, approval checkpoints, monitoring, rate limits, and kill switches because many failures are only visible after an external action or downstream reconciliation.

Key Findings

No public source located for this item provides a task-complexity-normalised, post-pipeline, head-to-head production failure distribution that directly compares AI-assisted code changes with AI-agent-executed business-process actions.
Build-mode failure is most visibly measured as post-release intervention after release gates, while Site Reliability Engineering guidance supports the inference that equivalent failures can also be blocked before production by system-level tests.
Public evidence on AI-assisted coding shows elevated before-release security and defect risk, but it also shows that static-analysis feedback can eliminate a substantial share of those issues before release, which makes build-mode variance materially filterable before external effects occur.
The 2024 DORA evidence shows local development gains alongside lower delivery stability and throughput, which supports the inference that AI assistance still requires small batches, robust tests, and disciplined review if teams want low escaped-failure rates.
When compared with gated build-mode proxies, realistic do-mode benchmarks indicate materially lower baseline reliability, because WebArena reports 14.41% GPT-4 success, tau-bench reports fewer than 50% successful tasks and less than 25% reliability across repeated retail trials, and AgentDojo still reports many failures even without attacks.
Do-mode failures are harder to catch before impact because many of them arise from live tool use, policy misapplication, excessive permissions, prompt injection, or wrong action selection, all of which often become fully visible only after an external step has already been taken.
The strongest public legal and regulatory material places autonomous process execution inside lifecycle risk-management and operational-resilience disciplines, which supports the inference that teams cannot govern it only as a conventional pre-release software-quality problem.
The best-supported comparative conclusion is therefore that build mode is structurally more controllable than do mode, because the dominant control point in build mode is release gating before deployment while the dominant control point in do mode must remain a runtime control layer around live decisions and actions.

Assumptions

Assumption: DORA instability metrics are the least misleading public proxy for post-pipeline build-mode failure because no accessible public dataset separates AI-assisted and human-authored changes after release. Justification: DORA explicitly measures interventions required after deployment, which matches the escaped-failure concept once tests and review have already run.
Assumption: Web and enterprise-agent benchmarks are the least misleading public proxy for AI-agent-executed business-process failure because public incident reporting for deployed agents is still sparse and not normalised by task type or detection timing. Justification: the retrieved benchmarks expose multi-step tasks with tools, policy constraints, and externally visible actions, which are the defining properties of the do-mode surface.

Analysis

The evidence weighs more strongly on control structure than on exact percentages, because the public literature does not yet publish a common denominator for "released AI-assisted changes" and "production agent actions" in the same dataset.

Build mode is still risky, and the 2024 DORA findings show that AI adoption can reduce stability if teams allow larger or less disciplined changes through the pipeline, but the failure is still mediated by typed interfaces, tests, code review, staged release, and rollback.

Do mode is different because the model is not only producing an artifact for later verification, it is selecting tools, interpreting policy, and causing state changes in a live environment, which means the same error can carry customer, operational, or compliance consequences immediately.

The practical trade-off is therefore between front-loaded verification cost in build mode and runtime supervision cost in do mode.

Part of the observed difference could still be explained by unmatched proxies and benchmark immaturity rather than by delivery mode alone, because public build-mode evidence is measured as escaped post-release failure while public do-mode evidence is measured as benchmark task success and severe-action frequency.

For regulated financial services, the stronger recommendation is not "never use do mode," but "treat do mode as a runtime control problem from day one," with narrow permissions, approval checkpoints for high-impact actions, monitoring, incident playbooks, and reversible operations.

Risks, Gaps, and Uncertainties

Public evidence still lacks a common, task-complexity-normalised dataset that traces AI-assisted code from authoring through release and then compares its escaped-failure rate directly with autonomous business-process execution.
Benchmark evidence for do mode is stronger than public production-incident evidence, so the exact relationship between benchmark failure rates and real production incident rates remains uncertain even though the directional signal is clear.
The Air Canada case is a strong public example of post-action detection and liability, but it is a chatbot case rather than a fully autonomous multi-tool agent case.
WorkArena confirms a considerable gap to full enterprise-task automation, but the abstract alone does not provide a single summary percentage for all ServiceNow tasks, which limits precision in the enterprise-task comparison.

Open Questions

What released-change telemetry would be needed inside a real software-delivery organisation to measure post-pipeline failure separately for AI-assisted and human-authored changes?
Which runtime controls, approval checkpoints, rate limits, and reversal mechanisms reduce do-mode incident rates most effectively in regulated operations without destroying the economic case for automation?
How much of the current do-mode benchmark failure is due to model capability limits versus control-plane design limits such as poor scoping, inadequate verification, or overly broad permissions?