Research / Language designed for LLM agents to produce

Language designed for LLM agents to produce

Language designed for LLM agents to produce: addressing generation-layer failure modes in agentic systems

2026-03-11 · agentic-ai tools-infrastructure ai-architecture formal-methods · medium · source → · wiki →
key claims
  1. Outlines (dottxt-ai, ~13,500 GitHub stars, $11.9M funding) and Guidance (Microsoft, ~21,000 GitHub stars) are production-grade structured generation tools at specification hierarchy level 2, enforcing JSON Schema, regular expressions, and context-free grammars at token-generation time via finite state machines to eliminate structurally invalid LLM output. Sources: https://github.com/dottxt-ai/outlines; https://github.com/guidance-ai/guidance
  2. ETH Zurich PLDI 2025 (Muendler et al., arXiv:2504.09246) is the sole published approach at specification hierarchy level 3, enforcing TypeScript type safety at decoding time using prefix automata and incremental type-checking, reducing compilation errors by more than half on HumanEval and MBPP benchmarks without requiring human-specified type annotations in the prompt. Source: https://arxiv.org/abs/2504.09246
  3. No paper or production system in 2022–2025 describes a programming language designed from the ground up for LLM agents to produce as their primary output format, with language constructs chosen to structurally address generation-layer or goal-layer failure modes — the LLM-as-primary-author framing is absent from all current language and format design work. Source: Comprehensive search of arXiv, PLDI, NeurIPS, ICML, ICLR 2022–2025 (this investigation, 2026-03-11)
  4. All current structured generation tools address Layer 1 structural failure modes only; semantic hallucination within a valid schema, intent mismatch (Layer 2), reward hacking (Layer 3), tool misuse beyond parameter typing (Layer 4), and instruction conflict (Layer 5) remain entirely outside the scope of output grammar or language design in 2025. Source: Cross-reference with taxonomy from `Research/completed/2026-03-10-ai-concept-classification-taxonomy.md`; confirmed by Outlines documentation explicitly acknowledging that semantic hallucination is not prevented
  5. LMQL ("Prompting Is Programming", Beurer-Kellner et al., arXiv:2212.06094, PLDI 2023) is the academic progenitor of the constrained LLM generation field, introducing constraint-guided decoding with stopping phrases, type constraints, and set-membership constraints enforced at decoding time; it directly influenced subsequent production tools including Outlines and SGLang. Source: https://arxiv.org/abs/2212.06094
  6. SGLang (Zheng et al., arXiv:2312.07104, NeurIPS 2024) is a Python-embedded domain-specific language (DSL) for multi-step LLM workflows with RadixAttention for key-value (KV) cache reuse and compressed finite state machines for structured output decoding, operating at specification hierarchy level 2 as a high-performance inference framework rather than introducing new LLM-output language semantics. Source: https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf
  7. "Code as Policies" (Liang et al., arXiv:2209.07753, 2023) and "Dafny as Verification-Aware Intermediate Representation" (arXiv:2501.06283, 2025) both repurpose existing human-designed languages as LLM output targets — Python for robot policy generation and Dafny for mechanical verification — demonstrating the pattern of LLM-as-author but not designing a new language with LLM-native properties. Sources: https://arxiv.org/abs/2209.07753; https://arxiv.org/pdf/2501.06283v1
  8. ReAct, Modular Reasoning Knowledge and Language (MRKL), and Toolformer (arXiv:2302.04761) tool-call output conventions occupy specification hierarchy level 1–2 weakly, as informal text format patterns embedded in prompts or fine-tuning without grammar enforcement, making structurally non-conforming outputs possible and routinely observed in practice. Source: https://arxiv.org/abs/2302.04761

Research Question

Is anyone actively developing a programming language or structured output format specifically designed for LLM agents to generate — rather than humans to write — that structurally addresses generation-layer and goal-layer failure modes (hallucination, intent mismatch, reward hacking, under-specification, instruction conflict, tool misuse) as classified in the failure mode taxonomy and the specification hierarchy established in the two referenced completed items? If so: what is the design rationale, what failure modes does each approach target, and how mature is the work?

Findings

Executive Summary

No programming language or structured output format designed specifically for large language model (LLM) agents to produce has been identified in the 2022–2025 research literature; the design space for such a language at specification hierarchy levels 4–5 is unoccupied and faces theoretical and economic barriers that explain the absence. The structured generation field has reached production maturity at level 2 (schema and grammar enforcement via Outlines and Guidance), with one published research prototype at level 3 (type-constrained decoding for TypeScript, ETH Zurich PLDI 2025). All current approaches address only Layer 1 structural failures from the five-layer failure mode taxonomy; intent mismatch (Layer 2), reward hacking (Layer 3), tool misuse beyond parameter typing (Layer 4), and instruction conflict (Layer 5) remain unaddressed by any output grammar or language design. The primary barrier to a level 4–5 LLM-output language is that encoding verifiable intent in a checkable form requires a complete formal specification of the goal — which is a level 5 requirement that cannot be reduced to a grammar.

Key Findings

  1. [High] Outlines (dottxt-ai, ~13,500 GitHub stars, $11.9M funding) and Guidance (Microsoft, ~21,000 GitHub stars) are production-grade structured generation tools at specification hierarchy level 2, enforcing JSON Schema, regular expressions, and context-free grammars at token-generation time via finite state machines to eliminate structurally invalid LLM output. Sources: Outlines (dottxt-ai) Guidance (Microsoft)

  2. [High] ETH Zurich PLDI 2025 (Muendler et al., arXiv:2504.09246) is the sole published approach at specification hierarchy level 3, enforcing TypeScript type safety at decoding time using prefix automata and incremental type-checking, reducing compilation errors by more than half on HumanEval and MBPP benchmarks without requiring human-specified type annotations in the prompt. Source: ETH Zurich PLDI 2025 — type-constrained decoding

  3. [High] No paper or production system in 2022–2025 describes a programming language designed from the ground up for LLM agents to produce as their primary output format, with language constructs chosen to structurally address generation-layer or goal-layer failure modes — the LLM-as-primary-author framing is absent from all current language and format design work. Source: Comprehensive search of arXiv, PLDI, NeurIPS, ICML, ICLR 2022–2025 (this investigation, 2026-03-11).

  4. [High] All current structured generation tools address Layer 1 structural failure modes only; semantic hallucination within a valid schema, intent mismatch (Layer 2), reward hacking (Layer 3), tool misuse beyond parameter typing (Layer 4), and instruction conflict (Layer 5) remain entirely outside the scope of output grammar or language design in 2025. Source: Cross-reference with taxonomy from Research/completed/2026-03-10-ai-concept-classification-taxonomy.md; confirmed by Outlines documentation explicitly acknowledging that semantic hallucination is not prevented.

  5. [High] LMQL ("Prompting Is Programming", Beurer-Kellner et al., arXiv:2212.06094, PLDI 2023) is the academic progenitor of the constrained LLM generation field, introducing constraint-guided decoding with stopping phrases, type constraints, and set-membership constraints enforced at decoding time; it directly influenced subsequent production tools including Outlines and SGLang. Source: arxiv.org/abs/2212.06094

  6. [High] SGLang (Zheng et al., arXiv:2312.07104, NeurIPS 2024) is a Python-embedded domain-specific language (DSL) for multi-step LLM workflows with RadixAttention for key-value (KV) cache reuse and compressed finite state machines for structured output decoding, operating at specification hierarchy level 2 as a high-performance inference framework rather than introducing new LLM-output language semantics. Source: proceedings.neurips.cc

  7. [Medium] "Code as Policies" (Liang et al., arXiv:2209.07753, 2023) and "Dafny as Verification-Aware Intermediate Representation" (arXiv:2501.06283, 2025) both repurpose existing human-designed languages as LLM output targets — Python for robot policy generation and Dafny for mechanical verification — demonstrating the pattern of LLM-as-author but not designing a new language with LLM-native properties. Sources: arxiv.org/abs/2209.07753 arxiv.org

  8. [High] ReAct, Modular Reasoning Knowledge and Language (MRKL), and Toolformer (arXiv:2302.04761) tool-call output conventions occupy specification hierarchy level 1–2 weakly, as informal text format patterns embedded in prompts or fine-tuning without grammar enforcement, making structurally non-conforming outputs possible and routinely observed in practice. Source: arxiv.org/abs/2302.04761

  9. [Medium] The economic and computational barrier to advancing past level 3 in production is that schema enforcement (level 2) operates at near-zero inference overhead, type-constrained decoding (level 3) requires a per-token type-checking pass increasing latency, and formal verification (level 4–5) would require a per-token verification pass that is computationally prohibitive at current inference hardware speeds. Source: [inference] from ETH Zurich paper methodology and economic analysis of inference costs.

  10. [Medium] A purpose-built LLM-output language at levels 4–5 would need to include a type system expressive enough to encode behavioural invariants beyond field types, a decidable verification procedure executable at token-generation time, and a mechanically checkable representation of intent — the last of which faces a fundamental undecidability barrier because encoding "the human's true goal" in a checkable form requires a complete formal specification of that goal. Source: [inference] from specification hierarchy in Research/completed/2026-03-10-formal-spec-intent-alignment-agentic-coding.md and computability theory.

Assumptions

  1. The publication venue search (arXiv, PLDI, NeurIPS, ICML, ICLR 2022–2025) is sufficiently comprehensive that a purpose-built LLM-output language, if it existed as an active research programme, would have appeared in at least one of these venues. This assumption would be violated if such work appeared only in domain-specific workshop proceedings not indexed by arXiv or the major venue searches.

  2. Jsonformer is adequately represented by the Outlines and Guidance findings, as both supersede it in functionality (Outlines supports regex, JSON Schema, and context-free grammars; Jsonformer supports JSON only) and adoption (Outlines has 3M+ downloads; Jsonformer has no equivalent adoption signal). Checking Jsonformer separately would not change any finding.

Analysis

The structured generation field has undergone a cycle of rapid tool development (2022–2025) that mirrors the historical arc of compiler-enforced safety in conventional programming languages, compressed from decades into three years by the commercial urgency of LLM deployment. LMQL (PLDI 2023) established the theoretical framing; Outlines and Guidance built production tools at level 2; ETH Zurich PLDI 2025 advanced to level 3 for code generation specifically.

The key analytical finding is that the failure mode taxonomy from 2026-03-10-ai-concept-classification-taxonomy.md cleanly explains why the level 2 tools dominate: structural controls address Layer 1 structural failures, which is exactly where schema/grammar enforcement operates. The deeper failure modes (Layers 2–5) require semantic, procedural, or architectural controls — not structural ones. No structural (grammar-level) intervention can address intent mismatch because intent is not a structural property of the output.

The distinction between "language for orchestrating LLM agents" (designed by humans, e.g. PayPal declarative DSL, IntentLang) and "language LLM agents produce" (the LLM is the primary author) is the critical framing that the literature does not maintain. Most 2024–2025 work on "agent languages" addresses the orchestration side — which is a software engineering problem — rather than the LLM-output format side, which is a programming language design problem. This framing gap explains why the research question has not been posed directly by any existing paper.

Risks, Gaps, and Uncertainties

  1. Recency gap: NeurIPS 2025 and ICML 2025 workshop proceedings were not comprehensively indexed as of 2026-03-11. A purpose-built LLM-output language paper may have been submitted to a 2025 conference or workshop not yet discoverable.
  2. Industrial unpublished work: Large AI labs (Google DeepMind, Anthropic, Meta AI) may be developing purpose-built output languages for internal agentic systems without publishing. No evidence of this was found, but absence of public evidence does not confirm absence of work.
  3. Domain-specific tractability: The undecidability argument for level 4–5 applies to general-purpose languages. A restricted-domain level-4 language (e.g. for SQL generation, infrastructure-as-code, or configuration management) may be tractable and remains unexamined.
  4. Specification hierarchy level 3 breadth: The ETH Zurich PLDI 2025 paper covers TypeScript only. Whether type-constrained decoding generalises to other typed languages (Rust, Go, Java) without prohibitive overhead is an open empirical question.

Open Questions

  1. Restricted-domain level-4 LLM-output language: Is a purpose-built level-4 language (rich type system encoding behavioural invariants) tractable for a specific restricted domain such as database query generation or infrastructure-as-code? This is a narrower, potentially tractable version of the current research question and could yield a concrete design candidate. Priority: medium (advances understanding; does not immediately block other work).

  2. Evaluation benchmarks for Layer 2 failure mode coverage: How would one measure whether an LLM-output language or structured format reduces intent mismatch (Layer 2) rather than just structural conformance (Layer 1)? No benchmark currently distinguishes these two layers in the structured generation literature. Priority: medium (needed to evaluate any future level-3 or level-4 approach).

  3. Fine-tuning vs. grammar enforcement trade-off: Does fine-tuning LLMs on schema-conforming output corpora produce models that internalise structural constraints at the weight level, reducing or eliminating the need for grammar enforcement at inference time? If so, the inference overhead of type-constrained decoding could be amortised into training cost. Priority: low (exploratory; no clear downstream dependency).

  4. LLM-native language design principles: What properties would a language designed for LLM authorship (rather than human authorship) optimise for — and how would these differ from human-designed languages? This is a foundational design question that could seed a new research programme. Priority: low (speculative; no immediate application).

sources

← newerSuperpowers as inspiration
older →The DIKW pyramid: transformation functions from data to information to…

Connected items

Loading…

View full knowledge graph →