Agent Runner: A Standard, Open-Source Agent Harness for Evaluating Real-World Coding Agents

As LLM-based coding agents mature, evaluation has become a core bottleneck. Existing benchmarks largely rely on static tasks and capture only the model’s final output, not the intermediate reasoning steps or tool interactions that define modern agentic systems. To study these systems rigorously, we need instrumentation that reflects how agents are used and behave in actual development environments.

AgentRunner is an open-source agent harness that executes real Design Arena user prompts, traces full agent behavior, and records all tool interactions — including file edits, search operations, command execution, and multi-step planning loops. The goal is to enable reproducible, provider-agnostic evaluation of coding agents under realistic conditions.

Motivation

Limitations of existing benchmarks

Standard benchmarks evaluate isolated model completions, but production agents perform:

• Multi-file modifications
• Iterative re-prompting
• Tool-invocation chains
• Failure recovery
• Long-horizon planning

These behaviors meaningfully affect performance but are not captured by current benchmarks with static test sets.

Real user prompts as evaluation tasks

Synthetic tasks lack the variability and ambiguity of genuine user requests. Agent Runner instead uses live prompts from Design Arena's 490K+ users, yielding a benchmark that evolves with real usage and captures distributions of actual user requests and preferences.

Model-agnostic, framework-agnostic baseline

Agent Runner supports multiple LLM providers (OpenAI, Anthropic, Google, xAI, Mistral, Kimi, and more), enabling comparative studies in a pre-defined, open-source baseline. The agent can generate code for any project and deploy to Vercel.

Provider-specific optimizations

To ensure that tool calls, system prompts, and execution semantics were faithfully implemented, we coordinated directly with at least one representative — and often more — from each participating lab. This alignment was necessary because agent interfaces and tool-use protocols vary across providers, and small differences in system-prompt structure or tool schemas can meaningfully affect behavior. These configurations remain private as they are disclosed at the discretion of our collaborators. Please message contact@designarena.ai with any notes.

System Overview

AgentRunner provides:

• Unified tool interface (file operations, patching, search, bash execution)
• Sandboxed workspaces for reproducible execution
• Automatic context compaction
• Persistent session logs for detailed analysis

All tool calls, messages, and workspace changes are recorded, allowing fine-grained tracing of an agent’s decision process.

How to try it

• Kick off Agent Runner at https://www.designarena.ai/agentarena
• Repo at https://github.com/Design-Arena/agent-runner
• Use it as a CLI tool: https://pypi.org/project/agent-runner/
  • pip install agent-runner
  • agentrunner run “create a nextjs replica of Discord”