How Microsoft Research Asia used Design Arena to validate their aesthetic coding benchmark

We recently added AesCoder-4B to Design Arena: a new model trained by a team from Microsoft Research Asia. AesCoder-4B is comparatively tiny at just 4 billion parameters, but as of the time of this writing, it is beating several flagship models in the Website rankings, including Gemini 2.5 Pro, DeepSeek R1, and Sonnet 4 (Thinking). AesCoder is also significantly faster than its peers, holding one of the highest Elo-to-generation-speed ratios in the Website category. 

According to the model’s companion paper, AesCoder-4B was developed by fine-tuning Qwen3-4B-Instruct-2507 in two stages. The team first carried out supervised fine-tuning on their newly built AesCode-358K dataset. For the Python subset, they refined the VisCode-200K dataset by regenerating its code samples with Qwen3-Coder-480B-A35B-Instruct-FP8, restricting them to essential libraries such as matplotlib, seaborn, and plotly, and verifying executability through runtime validation, yielding 158,000 high-quality plotting examples. The website subset was created from 200,000 design instructions generated by GPT-4o, paired with corresponding HTML outputs from GPT-5 and Qwen3-Coder-480B-A35B-Instruct-FP8.

The team then used a GRPO algorithm integrating agentic reward feedback (GRPO-AR) to perform reinforcement learning. GRPO-AR combines agent-assigned scores in three categories deemed essential for visual coding tasks – executability, static aesthetics, and interactive aesthetics – into a single composite reward score, with respective category weights of 0.1, 0.8, and 0.1. The objective of this novel approach was to jointly optimize for functionality and aesthetics.

Citing a gap in the literature for aesthetic code benchmarks, the researchers developed their own automatic and locally runnable benchmark – OpenDesign – to evaluate their fine-tuning approach’s performance. OpenDesign uses a two-level LLM-as-a-judge framework to rank model outputs on aesthetics. Pulling from a sample of 840 prompts corresponding to real webpage design cases, HTML generations are made by the model of interest. These prompt-HTML pairs are then evaluated by a judge model (GPT-4o) through two evaluations. 

First, the HTML code is rendered into an image and assigned a static aesthetics score. Second, an interactive aesthetics agent assigns an interactive aesthetics score using the same prompt-HTML pairing. This process is repeated for each prompt in the OpenDesign pool. The final score is the average of these results across the 840 benchmark cases.

AesCoder-4B’s performance on OpenDesign placed it above every model included in their sample set (created pre-Sonnet 4.5) except Claude Sonnet 4 and GPT-5. To assess the quality of the results, the researchers cross-validated a sample of the scores with internally labeled human preference data. This route demonstrated close (80.9%) GPT-human alignment.

The researchers supplemented this approach by testing the OpenDesign scores’ correlation with Design Arena’s leaderboard rankings, finding broad agreement between their rankings and Design Arena’s. This alignment suggests that aesthetics can be approximated economically, although agreement rates between judge models and human judges are still imperfect (60-80%). The researchers also did not explore how tastes and preferences evolve over time, and whether OpenDesign’s framework can update dynamically in response.

For researchers, this opens the door to a new class of automated evaluation pipelines shown to be correlated with recent snapshots of leaderboard rankings constructed with large volumes of preference data. And for the broader community, it offers a glimpse into a future where taste-based preferences are readily quantifiable and comparable, enabling smaller teams and individual builders to validate products quickly and affordably.

We’re committed to supporting this future, and we’ve added a model request form to our leaderboard page as a step forward. Now, any team interested in benchmarking their model at scale can submit an inquiry and see where they rank. 

Congratulations to Microsoft Research Asia, Zhiyuan College, Shanghai Jiao Tong University, and Peking University for your work on AesCoder-4B. We’re excited to see how the model performs as more votes accumulate in the coming weeks!