Research / Agent evaluation framework

Agent evaluation framework

Agent evaluation framework: cross-repo pattern analysis, commonality detection, and regression identification

2026-03-10 · agentic-ai benchmarks-eval ai-architecture · medium · source → · wiki →
key claims
  1. Tool-mediated execution is the single universal architectural pattern across all eight surveyed agent frameworks and is the necessary precondition for any systematic evaluation: without discrete, logged tool calls, agent actions cannot be traced, replayed, or compared across versions. Confidence: high
  2. Trajectory capture — recording the full sequence of tool calls, state transitions, and intermediate outputs for each agent run — is the evaluation primitive that all other evaluation techniques depend on, and it is present in five of the eight surveyed frameworks but implemented in incompatible ways with no shared format across implementations. Confidence: high
  3. The practitioner consensus in 2025 is that automated evaluation runs gated on quality thresholds ("evals as unit tests") are expected CI infrastructure for production LLM agents, but fewer than half of the surveyed frameworks provide integrated eval tooling, forcing teams to rely on separate tools such as DeepEval, TruLens, or agentevals. Confidence: high
  4. The measurement gap between what practitioners want to measure (holistic quality including goal alignment, safety, and factuality) and what they actually measure in CI (task pass rate, latency, cost) persists because holistic metrics require LLM-as-judge evaluation — expensive, inconsistent without structured rubrics, and not integrated into most CI pipelines. Confidence: high
  5. LangGraph achieves approximately 94% task completion on complex branching workflows in cross-framework benchmarks and is the only orchestration framework with built-in time-travel debugging via DAG checkpoints, making it the strongest surveyed choice for workflows where traceability, rollback, and auditability are required. Confidence: medium (performance figure from secondary sources; architectural uniqueness from primary docs)
  6. Pydantic AI is the only surveyed framework that applies type-constrained output validation at every agent-tool boundary as a structural control, preventing Layer 1 generation failures by construction rather than by post-hoc parsing — a genuine architectural novelty absent from all other surveyed frameworks. Confidence: high
  7. OpenHands demonstrated that inference-time scaling with a learned critic model increases SWE-bench Verified resolution from 60.6% (one attempt) to 66.4% (five attempts), establishing that sampling strategy at inference time is a first-class optimisation lever independent of model capability or prompt quality. Confidence: high
  8. METR's time-horizon-of-completion metric — the maximum human-hours task duration at which an agent achieves at least 50% success, currently doubling approximately every seven months — characterises the agent's capability envelope rather than performance on a fixed benchmark, making it the most future-proof effectiveness signal identified in the survey. Confidence: high

Research Question

What evaluation framework allows systematic comparison of agent implementations across multiple repositories — identifying what problems each is solving, whether concepts are used idiomatically or in novel ways, whether the agent is effective, and how to detect whether a change to an agent made it better or worse — and what would a minimal viable implementation of such a framework look like?

Findings

Executive Summary

No universally adopted evaluation framework for cross-repo agent comparison exists in early 2026, but the field has converged on a coherent set of practices. Tool-mediated execution is the single universal architectural pattern across all surveyed frameworks and is the precondition for any systematic evaluation. The critical gap is between the metrics practitioners want (holistic quality: goal alignment, safety, factuality) and what they actually measure in CI (task pass rate, latency, cost), because holistic metrics require LLM-as-judge evaluation that most frameworks do not integrate by default. A minimal viable evaluation framework requires five components: a versioned scenario registry, a two-layer metric stack (code-based + LLM-as-judge with structured rubric), trace capture, a CI regression gate with thresholds, and a gold dataset refresh protocol.

Key Findings

  1. Tool-mediated execution is the single universal architectural pattern across all eight surveyed agent frameworks and is the necessary precondition for any systematic evaluation: without discrete, logged tool calls, agent actions cannot be traced, replayed, or compared across versions. Confidence: high.

  2. Trajectory capture — recording the full sequence of tool calls, state transitions, and intermediate outputs for each agent run — is the evaluation primitive that all other evaluation techniques depend on, and it is present in five of the eight surveyed frameworks but implemented in incompatible ways with no shared format across implementations. Confidence: high.

  3. The practitioner consensus in 2025 is that automated evaluation runs gated on quality thresholds ("evals as unit tests") are expected CI infrastructure for production LLM agents, but fewer than half of the surveyed frameworks provide integrated eval tooling, forcing teams to rely on separate tools such as DeepEval, TruLens, or agentevals. Confidence: high.

  4. The measurement gap between what practitioners want to measure (holistic quality including goal alignment, safety, and factuality) and what they actually measure in CI (task pass rate, latency, cost) persists because holistic metrics require LLM-as-judge evaluation — expensive, inconsistent without structured rubrics, and not integrated into most CI pipelines. Confidence: high.

  5. LangGraph achieves approximately 94% task completion on complex branching workflows in cross-framework benchmarks and is the only orchestration framework with built-in time-travel debugging via DAG checkpoints, making it the strongest surveyed choice for workflows where traceability, rollback, and auditability are required. Confidence: medium (performance figure from secondary sources; architectural uniqueness from primary docs).

  6. Pydantic AI is the only surveyed framework that applies type-constrained output validation at every agent-tool boundary as a structural control, preventing Layer 1 generation failures by construction rather than by post-hoc parsing — a genuine architectural novelty absent from all other surveyed frameworks. Confidence: high.

  7. OpenHands demonstrated that inference-time scaling with a learned critic model increases SWE-bench Verified resolution from 60.6% (one attempt) to 66.4% (five attempts), establishing that sampling strategy at inference time is a first-class optimisation lever independent of model capability or prompt quality. Confidence: high.

  8. METR's time-horizon-of-completion metric — the maximum human-hours task duration at which an agent achieves at least 50% success, currently doubling approximately every seven months — characterises the agent's capability envelope rather than performance on a fixed benchmark, making it the most future-proof effectiveness signal identified in the survey. Confidence: high.

  9. Safety evaluation is the largest unaddressed gap in the surveyed evaluation ecosystem: none of the frameworks integrate OWASP LLM Top 10 checks (prompt injection, guardrail bypass, excessive agency) as first-class CI evaluation targets, despite these being documented production failure modes affecting all agent architectures. Confidence: medium (inferred from absence of evidence in framework documentation).

  10. LLM-as-judge evaluation without a structured rubric produces irreproducible results; all reliable implementations require a coded evaluation prompt with explicit scoring dimensions that is itself version-controlled and subject to regression testing, functioning as a machine-readable specification of what "good output" means. Confidence: high.

  11. Benchmark saturation is a structural property of all fixed agent benchmarks, and SWE-bench Verified is already approaching 70% resolution for top agents, prompting the creation of SWE-bench-Live; any evaluation framework must include a benchmark refresh mechanism as a design principle rather than treating the initial benchmark as a permanent standard. Confidence: high.

  12. A minimal viable evaluation framework for a research loop agent requires five components: a versioned scenario registry, a two-layer metric stack (code-based + LLM-as-judge with structured rubric), trace capture, a CI regression gate with explicit failure thresholds, and a gold dataset refresh protocol — with shadow testing and A/B testing as production-grade extensions. Confidence: high (inference synthesised from primary practitioner and framework sources).

Assumptions

Analysis

The central tension in agent evaluation is the determinism-capability trade-off: more capable agents produce more variable outputs, making deterministic unit-test-style assertions inadequate and requiring probabilistic evaluation. This explains the convergence on evals-as-code with LLM-as-judge.

The divergence across frameworks in evaluation depth reflects design philosophy differences. Pydantic AI prioritises prevention (structural controls at every boundary); LangGraph prioritises operational debuggability (time-travel, checkpoints); OpenHands prioritises benchmark reproducibility (published trajectory datasets). Agno is the only framework that treats evaluation as a first-class framework concern with four distinct built-in eval types.

For a research loop agent, the architectural implication is clear: the primary evaluation mechanism must be LLM-as-judge with structured rubrics (since outputs are long-form research, not structured data or code patches). The secondary mechanism is trajectory analysis (did the protocol get followed?). The tertiary mechanism is internal consistency checking (do §2 claims appear in §6? Are sources cited?). These three mechanisms correspond exactly to the three layers of the five-component MVF specification.

Risks, Gaps, and Uncertainties

Open Questions

  1. Research loop evaluation rubric: What structured rubric should be used to LLM-judge the outputs of this repository's research loop agent? This is a direct follow-on backlog item (priority: high; blocks: implementation of any research loop eval gate).
  2. Trajectory similarity metric: Can the research loop agent's trajectory (tool call sequence, sources consulted, protocol adherence) be compared automatically across agent versions to detect regressions without full LLM-as-judge evaluation?
  3. Benchmark refresh for research agents: How should a gold dataset of research questions with known-good answers be curated and maintained for research loop evaluation? SWE-bench-Live's continuous update model offers a design pattern.
  4. Safety eval integration: What would it take to add OWASP LLM Top 10 checks to the CI pipeline for this repository's research loop agent?

Output

sources

← newerThe DIKW pyramid: transformation functions from data to information to…
older →Adversarial agents with shared goals

Connected items

Loading…

View full knowledge graph →