Artificial Intelligence (AI) coding assistant deployment outcomes

Research Question

Which organisations have published or disclosed coherent AI strategies specifically targeting software engineering, what outcomes have they measured, and what does the trajectory from AI-assisted coding (Type 2) to autonomous software agents (Type 3/4) look like in practice?

Findings

Executive Summary

Organisations deploying AI coding assistants at scale achieve real but task-scoped individual productivity gains of 40–55% on well-specified coding tasks; however, these gains do not automatically translate to team-level delivery improvement without process redesign, and Google's DORA 2024 data shows higher AI adoption is associated with worse delivery stability. Google reports that over 25% of its new production code is now AI-generated, reviewed by engineers — the largest disclosed deployment. The current enterprise state is Type 2 (agentic builder, human-reviewed artefact before deployment), not Type 3 (autonomous deployment); all major financial services and technology firms maintain mandatory human review gates as a non-negotiable governance control. Autonomous coding agents have advanced rapidly on benchmarks (from 14% in March 2024 to ~80% on curated tasks by early 2026), but headline scores overstate generalised capability, and no enterprise is running Type 3 SWE autonomy at scale.

Key Findings

1. The Microsoft/GitHub/MIT RCT (2023) found that developers using GitHub Copilot completed a specified JavaScript task 55.8% faster (1h11m vs 2h41m; 95% CI: 21–89%; P=0.0017), with the strongest benefit for less experienced and older developers — but the result applies to a single bounded task and does not generalize to complex multi-file enterprise work.

2. Google's CEO disclosed in 2024 earnings calls that over 25% of Google's new code is AI-generated, reviewed by human engineers before submission to production systems across Search, Cloud, and YouTube — the largest publicly disclosed AI code generation deployment.

3. ANZ Bank deployed GitHub Copilot to 3,000 engineers following an internal controlled trial of 1,000 engineers that showed 40–55% faster code development and improvements in code quality; ANZ is the most relevant documented case for the NZ financial services sector.

4. Google's DORA 2024 report (tens of thousands of respondents) found that higher AI adoption associates with a 7.2% decline in delivery stability and a 1.5% decline in throughput at the organisational level, even while individual developer productivity improves — confirming that individual speed gains require workflow redesign to convert into team-level delivery improvement.

5. Autonomous coding agents went from 14% on SWE-bench (Devin, March 2024) to ~80% on a curated 500-issue subset and ~46% on harder multi-file enterprise tasks by early 2026, representing roughly two years of rapid progress — but a discriminative subset analysis found performance drops from 73% headline to ~11% on problems that no agent had previously solved, indicating headline leaderboard scores significantly overstate generalised capability.

6. GitHub Copilot Coding Agent (GA September 2025) and comparable agentic tools operate as Type 2 (agentic builder, human-gated): the agent creates a draft pull request; mandatory human review and branch protection rules apply before any merge to production, and this model is consistent across all major enterprise deployments reviewed.

7. No major enterprise is documented as running Type 3 (autonomous deployment with delegated authority) for production software at scale; all financial services firms (JPMorgan, Goldman Sachs, Morgan Stanley, ANZ) and major platform providers (GitHub/Microsoft) maintain mandatory human review as the non-negotiable control point for AI-generated code before deployment.

8. AI-generated code without meaningful human authorship is uncopyrightable in most jurisdictions, including the US and UK, requiring organisations to document human contributions during AI-assisted development; code provenance tracking tools have emerged specifically to flag AI-generated outputs that match open-source code with restrictive licences.

9. JPMorgan Chase (200,000+ employees, OmniAI and LLM Suite) reports 10–20% developer productivity gains from AI-assisted coding, lower than the GitHub Copilot RCT figure, consistent with the DORA finding that broader enterprise deployment yields more modest aggregate gains than controlled task experiments.

10. No NZ technology companies (Xero, Trade Me, Datacom) have published SWE AI outcome data; the closest regional case is ANZ (Australian parent), whose deployment is directly relevant to ANZ NZ engineering teams and provides a useful benchmark for NZ financial services organisations considering similar programmes.

Assumptions

• Assumption: Sundar Pichai's >25% figure is directionally accurate for Google's production code. Justification: Statement made in an investor earnings call context, where material misstatement has legal consequences; independent audit not available.
• Assumption: Absence of NZ company SWE AI disclosures reflects NZ organisational culture around case study publication, not absence of AI coding programmes. Justification: AI Forum NZ survey data shows broad AI adoption in NZ; the gap is in public outcome disclosure, consistent with NZ organisations' typical preference for private implementation.
• Assumption: ANZ Australian engineering results are applicable to ANZ NZ teams. Justification: ANZ operates as a single trans-Tasman engineering organisation; the Copilot deployment was described as bank-wide.

Analysis

Individual productivity gains from AI coding tools are real and consistent across the strongest evidence (RCT, ANZ internal trial). The consistency across diverse methods (controlled experiment, internal trial, self-reported survey) increases confidence. The DORA counterevidence (delivery stability decline) is critical: it confirms that the right question is not "does AI make developers faster?" (yes, on bounded tasks) but "does AI improve software delivery?" (not automatically, and possibly negatively without process redesign).

The Type 2 / Type 3 distinction is doing real analytical work here. All documented enterprises are operating Type 2. The agentic frontier (SWE-bench, Copilot Coding Agent) is clearly advancing toward Type 2 GA and probing Type 3 at the margins. The speed of benchmark improvement (14% → 80% in two years on curated tasks) is genuine but the discriminative subset finding is an important corrective: agents are getting better at problems that have been seen before, and the genuinely hard novel problems remain unsolved.

The IP governance finding is underweighted in most corporate AI strategy discussions. An organisation that cannot assert copyright over its AI-generated code has a weaker IP position than one that carefully documents human authorship contributions. This is a non-obvious strategic implication that deserves prominence in any NZ technology governance framework.

Risks, Gaps, and Uncertainties

• The ANZ 40–55% figure is from a controlled internal trial on structured tasks; its applicability to complex enterprise software work (e.g., architectural changes, multi-system integration) is unverified.
• No independent audit of Google's >25% AI code claim. The figure may be subject to definitional variation (AI-suggested code that is accepted vs. AI-drafted code with minimal human modification).
• NZ-specific outcome data is entirely absent from the public record. Any NZ strategy built on global evidence is extrapolating from different scale, team composition, and codebase maturity contexts.
• The SWE-bench discriminative subset finding raises questions about whether benchmark improvement trajectories overstate progress on genuinely novel software engineering problems. If true, the path from Type 2 to Type 3 may be longer than headline numbers suggest.
• Governance frameworks for AI-generated code are evolving faster than case law. The IP positions documented here reflect current rulings and may change.

Open Questions

• What is the actual productivity impact of AI coding tools on complex multi-system software work (architectural decisions, integration design) as opposed to task-completion speed on bounded coding exercises?
• At what agent capability threshold (SWE-bench score, real-world deployment rate) will enterprises begin relaxing human review gates for lower-risk production changes?
• How are NZ technology organisations (Xero, Trade Me, Datacom) actually deploying AI coding tools, and what governance frameworks are they using? A structured interview programme would be needed to answer this question.
• Does AI code review (not just code generation) address the DORA delivery stability problem — i.e., can AI-assisted review compensate for AI-accelerated code volume?

sources

• [x] GitHub Octoverse: developer productivity impact of AI coding tools (2023, 2024)
• [x] Google AI impact on engineering productivity (internal reports surfaced in press)
• [x] McKinsey "Developer Velocity" report (AI-assisted coding outcomes)
• [ ] Proxify, MomoView, Kamiwaza — cited in Jevons Paradox research as early indicators
• [x] SWE-bench evaluations: autonomous coding agent performance data
• [ ] NZ tech company case studies: Xero, Trade Me, Datacom AI SWE programmes (if disclosed)
• [ ] ACM and IEEE published SWE AI governance frameworks