An overview of our live skills assessment methodology.

What are Micro Evals?

The current bottleneck in model progress isn’t figuring out how to improve models—it’s knowing what needs improvement. Creating realistic coding tasks for training or RL environments is difficult when you don’t know where models actually fail. Most data efforts rely on synthetic or reverse-engineered examples, which often produce inflated results on contrived codebases that bear little resemblance to real-world use.

An unintended learning from watching over 240,000 people build on Design Arena was that we became intimately familiar with the ways agents fall short and the most common vibe-coding failures. These moments revealed genuine failure modes that static academic benchmarks tend to miss. So we began turning each real failure into a focused evaluation. We call them Micro Evals—tiny tests grounded in authentic agent errors that capture how models truly behave in the wild.

How Micro Evals work

When users generate code on Design Arena, agents compete head-to-head in blind tournaments. Each generation is automatically deployed to GitHub and Vercel, producing a live, testable artifact. This step is essential—it ensures we evaluate real, production-ready code rather than theoretical outputs.

We run these evaluations across multiple generations, agents, and providers. Each Micro Eval targets a specific technical criterion: Does the Next.js App Router work correctly? Is Tailwind configured properly? Does the Supabase backend handle data as expected? The agent identifies potential errors, explains how they could be fixed, and validates the implementation.

All results are stored and surfaced in real time on the platform. Each pass or fail outcome contributes to a living dataset that maps how different models perform across real developer workflows. The result is an evaluation system grounded not in synthetic tests, but in authentic, evolving challenges faced by developers in the wild.

Evaluation Criteria

We value quality over quantity. We only add a micro-eval if we have sufficient cause to believe that it is (1) common enough to surface in real-world use-cases and (2) difficult enough to provide a discriminating signal.

For our NextJS evaluations, we use Vercel's next-evals-oss. We're grateful for the starting point, and we began by finding the most applicable evals from the top 50.

1. We filtered out the evals (030, 023, 000, 031) that tested migration/legacy software unrelated to vibe-coding from scratch.
2. Most vibe-coding errors are still basic build and functionality errors, so we did not include optimization-focused errors in our first assessment. This removed the following evals: 026, 029, 046, 028, 049, 009, 044, 027.
3. There were also complex patterns that were too niche (for example, using the AI SDK or newer use() calls) to be evaluated in typical vibe-coding tasks, such as 039/012, 040, 041, 008, 048, 047, 038, 039, 044/006, 037, 034, 011 (included 45 instead).
4. Finally, since evals are run with llm-as-a-judge, there were many that we found through our own experimentation that led to false negatives, as they were too complex to be evaluated as one eval. Examples are 020 and 021.
5. From the remaining, we added the common failures we found in apps deployed on Design Arena.

We repeated the same process for Vercel, ShadCN, TypeScript, Tailwind, etc. More to come.

Scoring & Results

Each evaluation produces a binary pass/fail outcome, along with a detailed breakdown of what succeeded and what didn’t. Pass rates are calculated as the percentage of successful evaluations across all tested generations for a given criterion. For example, if 85 out of 100 agent-generated apps correctly implement Next.js routing, that evaluation shows an 85% pass rate.

A radar chart visualizes overall performance across evaluation dimensions. Each axis represents a specific eval—Next.js, Tailwind, Vercel deployment, Supabase integration, and so on—while the plotted area reveals how well each agent performs across the full stack. This makes it easy to spot patterns: Does an agent excel at frontend but falter in backend logic? Are deployment issues systemic or isolated?

These results should be viewed as snapshots of current capability, not final rankings. As models evolve and our evaluation criteria expand, the metrics shift too. The goal isn’t to declare winners, but to pinpoint where agents struggle—turning real-world weaknesses into actionable feedback loops for model developers and concrete training signals for the next generation of AI systems.

Contributing to new evals

Your contribution plays a crucial role in identifying new evaluation criteria. When you encounter agent failures that aren't captured by existing evals, we want to hear about it. Click the "Suggest an Eval" card above to propose new technical criteria based on your experience using agent-generated code in production.
A good eval is specific, reproducible, and addresses a real failure mode. Rather than vague criteria like "code quality," effective evals test concrete implementation details: "Does the API route handle error cases correctly?" or "Is the authentication flow properly secured?" The best suggestions come from developers who've actually deployed agent code and discovered gaps between generation and production readiness. Your feedback helps us build better benchmarks and, ultimately, a forcing function to create better AI systems.

For questions or suggestions about evals, reach us at contact@designarena.ai.