Evaluating Mobile App-Building Agents

As coding agents mature, evaluation has become a core bottleneck. Design Arena’s core goal is to find the edges of model capability by evaluating agents on real-world application building, not toy tasks. We’ve seen models steadily improving quality and reliability in our React/web arena. To keep pushing that frontier, we recently expanded beyond frontend-only apps into fullstack work with real backends and real databases (also see this post).

Today we’re introducing Mobile Arena — another sharp edge. Mobile development varies widely model-to-model: how quickly an agent reaches a usable artifact, how often it gets stuck in build/debug loops, and whether the result is complete enough to judge as a real product. Mobile Arena is Design Arena’s mobile evaluation setting. Two agents receive the same prompt, build in parallel, and are judged in a blind A/B comparison. Unlike web-only settings, the outputs include device-ready previews and artifacts we can hand off to phones: Expo preview URLs, Android APK artifacts, and—coming soon—native iOS builds.

What typical benchmarks miss

Static benchmarks often capture a small segment of what makes a fully functional app—coming up with a verifiable benchmark requires limiting focus to what is being tested. But in reality, for a model to build a fully functional mobile app it requires all of the following skills and more:

• Multi-screen navigation and state (tab bars, stack navigation, deep linking between screens)
• Native UX intuition—building apps that feel native, not like web pages (pull-to-refresh, modals, FABs, swipe gestures)
• Design interpretation when prompts are vague (choosing aesthetics, colors, and layouts)
• Toolchain fluency across platform-specific build systems (Metro/Expo CLI, Gradle/AGP)
• Dependency and version management (resolving conflicts, BOM compatibility, Kotlin imports)
• Device-aware layout (keyboard avoidance, safe-area insets, proper input types)
• Animations and gesture handling (reanimated interactions, animateAsState transitions, gesture-handler)
• Light/dark theme support and Material Design 3 theming
• Platform quirks like icon and font preloading (icons render as empty boxes without it)
• Build and packaging (Expo multi-platform exports, Gradle APK assembly)
• Failure recovery (dependency errors, build failures, Metro cache issues, Gradle conflicts)
• End-to-end completeness (loading, empty, and error states for every view; no placeholders)

The best way to verify all of these at once is to put the judgment in the hands of the person who asked for it in the first place. No automated metric can reliably weigh how much navigation matters relative to visual polish, or whether a missing error state outweighs a smoother animation—those trade-offs are subjective and context-dependent. When a user picks the version they prefer, their choice naturally encodes all of these capabilities into a single signal: the app that felt more complete, more responsive, more like what they actually wanted wins.

Methodology

Real user prompts as tasks

Mobile Arena tasks are not synthetic — they come from real prompts submitted by Design Arena's 2M+ users. This means the distribution of tasks evolves with actual usage: the kinds of apps people ask for, the level of detail they provide, and the ambiguity they leave for the agent to resolve. Synthetic benchmarks can be carefully controlled, but they tend to converge on a narrow band of complexity. Real prompts naturally cover everything from "build me a habit tracker" to multi-page specifications with custom color schemes and specific interaction flows.

Because new prompts are continuously submitted, the benchmark stays current without manual curation — it reflects what people are actually trying to build today, not a fixed snapshot.

Template-based starting state

Every agent starts from the same initial state — a pre-baked project template for its track. For Expo, that means a scaffolded React Native + TypeScript project with Expo SDK, navigation libraries, async storage, reanimated, gesture-handler, and vector icons already installed. For native Android, it's a Kotlin + Jetpack Compose project with Material Design 3, Navigation Compose, Coil for image loading, DataStore, and kotlinx-serialization pre-configured. Fixing the initial observation this way ensures that differences in agent trajectories — including time-to-preview — reflect the policy itself (edits, debugging, build decisions) rather than variance in environment setup.

Tooling as the action space

Agents interact with the workspace through a fixed set of tool calls rather than arbitrary execution. The shared tool surface across both tracks includes:

• File operations (read_file, write_file, edit_file, batch_create_files, multi_edit, delete_file, grep)
• Shell execution (bash) with a blocklist that prevents hanging commands, scaffold commands, and long-running watchers
• Web search for prompts that require real-world or current data

Each track then adds its own build and deploy primitives:

• Expo: deploy_to_vercel, which exports the project with -platform all and deploys it, producing a preview URL and triggering an Expo update for on-device preview
• Native Android: build_apk, which runs Gradle's assembleDebug and produces a downloadable APK artifact, plus set_app_icon for customizing the launcher icon

When a tool call fails — a build error, a missing dependency, a malformed edit — the error is fed back to the model so it can correct and retry on its own. Agents run autonomously until they conclude the task is done or hit an unrecoverable API error; there is no human in the loop during execution, though users are free to re-prompt and ask for more modifications after each turn. This means failure recovery is itself part of what's being evaluated: how many attempts it takes, whether the agent gets stuck in a loop, and whether it can reason its way out of a bad state.

The underlying agent harness is Agent Runner, our open-source, provider-agnostic framework for executing and tracing coding agents. It supports multiple LLM providers (OpenAI, Anthropic, Google, xAI, Mistral, and more), records every tool call, message, and build event as a full execution trace, and provides sandboxed workspaces for reproducible runs. Keeping the tool surface and harness consistent across providers makes comparisons fair — differences in output reflect the model, not the infrastructure.

Evaluation and ranking

After both agents finish, their outputs are presented side by side in a blind A/B comparison — model identities are hidden until after the user votes. The voter can interact with both outputs on their phone (Expo previews via QR, APK installs for native Android) or in a browser, then picks a winner or declares a tie.

To mitigate bias, the left/right ordering of outputs is randomized for each session, and model names are never revealed until after the vote is submitted. This ensures the ranking signal comes from the quality of the output, not from brand preference or position bias.

Rankings emerge from collective community preferences rather than curated opinions: each pairwise vote is weighted equally, and Elo scores are computed via the Bradley-Terry model, which estimates each model's relative strength through an iterative algorithm that converges when strength estimates stabilize. As votes accumulate, the rankings become increasingly stable — though new models can be added at any time without invalidating existing comparisons. This is the same ranking methodology used across all Design Arena leaderboards, which means mobile rankings can be directly compared against web and fullstack rankings to see where a model's strengths and weaknesses diverge across surfaces.

Try it out now at DesignArena.ai/mobile (React Native) or DesignArena.ai/?mode=android (Android Native)