Fullstack Arena Methodology
Real-World Full-Stack Web Application Benchmarking
This platform tests whether AI coding agents can autonomously build complete, production-grade full-stack web applications with real persistent databases. Each model receives a user prompt describing an application, and must independently design a database schema, seed it with realistic data, build Vercel serverless API routes, construct a React frontend that fetches from those routes, and deploy the finished application to Vercel. We are benchmarking out-of-the-box coding ability with minimal scaffolding to accurately measure the frontier.
Agent Execution Flow
Each model goes through this flow automatically when a tournament is created:
User Prompt → Schema Design → Data Seeding → Backend Build → Frontend Build → Build & Deploy → LLM Judge
Flow Detail:
1. Schema Design: The agent receives the user prompt and designs a database schema. Using supabase_create_table, it creates all necessary Postgres tables with appropriate columns, types, constraints, and relationships. If the prompt requires user accounts or login, the agent calls supabase_auth_setup to create the built-in auth_users and auth_sessions tables. This is the critical first step – the entire application depends on a thoughtful schema.
2. Data Seeding: The agent populates tables with realistic seed data using supabase_insert, batching rows for efficiency. If authentication was set up, it creates demo users via supabase_auth_create_user with email/password credentials. The application must have content from the start – no empty states, no placeholder data.
3. Backend Build: The agent creates Vercel serverless API routes in the api/ directory. Each route file connects to the Supabase Postgres database using the pg library with parameterized queries, CORS headers, and proper HTTP method handling (GET, POST, PUT, DELETE).
4. Frontend Build: The agent builds a React + TypeScript + Tailwind CSS frontend that fetches all data from the API routes. Every user action (create, read, update, delete) must call the API and re-fetch data. No hardcoded arrays, no mock data – the frontend and backend must be fully connected end-to-end.
5. Build & Deploy: The agent runs npm run build, fixes any errors, and calls deploy_to_vercel() to deploy the completed application to a public URL.
6. LLM Judge (Schema Evaluation): After both agents in a session finish, an automated LLM judge (GPT-5.2) fetches the SQL DDL for each agent's schema and scores them on a 100-point rubric across seven criteria. These scores are stored alongside the generation for supplementary analysis.
Voting & Evaluation
The primary evaluation mechanism is anonymous human voting. When both agents have deployed their applications:
1. Side-by-Side Comparison: Voters see two deployed applications labeled "Model A" and "Model B" embedded as live iframes. They can interact with both applications and databases – clicking buttons, submitting forms, navigating pages, modifying table values – to assess functionality and quality.
2. Blind Voting: Model identities are hidden during voting. Voters choose which application is better based on their own assessment of design quality, functionality, data handling, and overall polish. Models are revealed only after the tournament is completed.
LLM Judge Scores (Supplementary):
After both agents finish, an LLM judges each agent's database schema on a 100-point rubric:
1. Domain Modeling & Semantic Clarity (20 points) – Do table/column names clearly communicate their purpose? Does the schema reflect the domain entities from the prompt?
2. Relationships & Referential Integrity (20 points) – Are foreign keys defined where they should be? Are cascades and delete behaviors sensible?
3. Normalization vs Practical Tradeoffs (15 points) – Is data stored without unnecessary duplication? Is JSON/JSONB used appropriately?
4. Queryability & Developer Ergonomics (15 points) – Can common operations be done with simple queries? Are join paths clean?
5. Type Discipline & Postgres Alignment (10 points) – Are appropriate Postgres types used (uuid, timestamptz, enum, etc.)?
6. Constraints & Data Correctness (10 points) – Are NOT NULL, UNIQUE, CHECK constraints used where needed?
7. Scale & Operational Resilience (10 points) – Is there a plausible indexing strategy? Would the schema handle growth?
These scores provide a complementary, automated signal on schema quality that does not depend on human voting. They are stored per-generation for analysis but do not currently factor into the main ELO leaderboard.
New Full Stack Tools Provided to Models
| Tool | Description |
|---|---|
supabase_create_table |
Create tables with typed columns (text, integer, serial, numeric, boolean, timestamptz, uuid, bigint, jsonb, varchar(N), double precision, etc.). Supports primary keys. |
supabase_insert |
Insert rows with batch support. Used for seeding realistic data. |
supabase_query |
SELECT with optional WHERE clauses and LIMIT. |
supabase_update |
UPDATE rows with SET and WHERE clauses. |
supabase_delete |
DELETE rows with WHERE clauses. |
supabase_auth_setup |
Creates auth_users and auth_sessions tables. Used when the prompt requires user accounts, login, or authentication. |
supabase_auth_create_user |
Creates a user with a hashed password (takes email and password). |
What We're Still Learning
Schema-first vs. code-first: The enforced database-first ordering may disadvantage models that think better in an iterative code-first style. We're evaluating whether relaxing the strict ordering changes outcomes, given that we currently build the schema before writing the code.
UI quality vs. backend quality: Some models produce stunning frontends with broken backends; others build robust APIs with plain UIs. Human voters tend to favor visual polish, which may underweight backend engineering quality. The LLM judge on schema design partially compensates, but there's an open question about how to weigh these dimensions.
Known Issues & Limitations
Supabase-only database: The benchmark only supports Supabase (Postgres). Models that might excel with different database paradigms (document stores, graph databases) cannot demonstrate those skills.
Vercel-only deployment: The serverless function model constrains architectural choices. Agents cannot use WebSockets, background workers, or other server patterns that might be better suited for certain applications.
No automated functional testing: We rely entirely on human voters to assess whether CRUD operations work. A visually appealing application with a subtly broken API route might win votes if the voter doesn't test that specific flow.
LLM judge limitations: The schema judge uses a single LLM,, which introduces its own biases about what constitutes good schema design. Different judges might produce different rankings.
No media in database: Agents can fetch images and generate visuals for the frontend, but cannot store media references in the database or build media-heavy CRUD flows.
No OAuth/social login: No OAuth/social login – agents can only set up email/password authentication (via supabase_auth_setup and supabase_auth_create_user), so prompts requiring Google Sign-In or other social providers can’t be fully implemented.
Planned Improvements
Automated functional testing: Add a test harness that verifies API routes return correct responses and that CRUD operations persist correctly, providing an objective quality signal independent of human voting.
Multi-database support: Expand beyond Supabase to support other database backends (MySQL, MongoDB, Redis), testing models' ability to choose and work with appropriate data stores.
Enhanced media support: Enable agents to create applications with user-uploaded images, file storage, and media-rich CRUD operations.
Google Sign-In support: integrate Supabase OAuth to let agents configure Google (and other social) login flows, enabling richer authentication patterns.
