In pursuit of a benchmark for human taste

We know when we like something, but when asked why, we stumble over our words and hallucinate reasoning. We contradict ourselves—we don’t know why we like a thing, we just do.

There exist people who understand our preferences better than we do ourselves. Let’s call them Tastemakers. Tastemakers know what we really want, not just what we say we want. Not a faster horse; a car. Great taste is a marker of how well we are understood.

Today’s frontier models still struggle to make the designs that we describe—a non-trivial percentage of the outputs on Design Arena are white text on a white background. But just teaching the models to make what the user asked for is not enough to teach models taste. Prompt adherence will never train the model to make the outcomes so ideal that we don’t even know to ask for them.

The ability to cater to an audience, sometimes in spite of what the audience says it wants, is an objective, measurable skill. This is a verifiable evaluation—you just have to ask the user whether or not they like the outcome.

This ability can't be captured by a deterministic evaluation. Static benchmarks do not reflect the authentic, diverse long-term goals and preferences of real users. They do not grow with user appetite, reflect current trends, or new learned behavior. The most accurate benchmark for a model’s ability to create what people actually want is measured at the user behavior level, not with some set of verifiable evals or synthetic RL environments.

But crowdsourced benchmarks built for the purpose of testing frontier models are not an accurate reflection of user behavior either, because the rewards assigned by the user are not sufficiently robust. People prompt for unrealistic use cases (like “hello world”) and vote for eye-catching outputs (like emoji responses), which then overfit the model to these outputs. The stakes are not there for the grader. You will get the most realistic assessment of whether a model is good at writing email responses if the person giving the feedback is actually trying to write an email.

So yes, we are building a consumer website because it is the most accurate benchmark. Counterintuitive? Yes. Hard? Definitely. But authentically capturing taste the way it should be done is something we care deeply about. So whatever, let’s do it.

We’re creating the most realistic benchmark for human preferences. If you want in, contact founders@designarena.ai.