What did Co-Design Bench find about coding agents?
On 8 November, The Decoder reported on a benchmark that puts a number on a complaint many developers already make. Coding agents often do most of a job and leave the last details undone. The test, called Co-Design Bench, was posted to arXiv on 26 October. Xilin Wei and fifteen co-authors wrote it, with affiliations that include Alibaba Token Hub, Fudan University and Zhejiang University. The Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University are also listed.
Across 273 tasks, the strongest of ten model endpoints passed an average of 69.9% of the acceptance criteria for a task. It fully satisfied every criterion in only 26.4% of tasks. In other words, 73.6% of tasks kept at least one unmet requirement. The model was credited for most of the work, but not all of it. The paper calls the first measure strict completion and the second the full-pass rate.
The gap matters because most public coding leaderboards report partial or test-based scores. A model that fixes four of five requested changes can look strong on an average. Yet it may still ship a broken button, a stale total or a layout that collapses at one screen size. The authors argue that criterion coverage and complete delivery should be reported together.
The results do not show that agents are useless. They show where the last mile is failing. The hardest cases combine several kinds of change. The choice of agent harness can also move a score by almost nineteen percentage points on repository edits. Both findings matter for teams that are choosing tools or setting review rules.
This is a benchmark paper rather than a product announcement. It defines tasks, writes hidden acceptance criteria and grades finished applications in a browser. Its central claim is narrow and testable. On real revision requests, partial success is common and complete success is rare.
How was the benchmark built from real coding sessions?
Co-Design Bench was not assembled from synthetic prompts. The authors started with 39,325 production queries from 26,314 real coding sessions, then removed exact and near-duplicates. That left 37,463 candidate requests, which were filtered and ranked before experts turned 273 of them into bounded tasks.
Each task pairs a normalized request with a runnable pre-edit application. Agents receive text alone or an image, audio clip, video or document. The benchmark has two tracks. Multimodal Editing contains 100 tasks for localized changes, and Repository Editing contains 173 tasks for coordinated work across multiple files.
Experts wrote 2,331 hidden acceptance criteria, with 553 in the editing track and 1,778 in repository work. The criteria describe required outcomes and anything that must be preserved, but they do not prescribe an implementation. That design means the graders need no gold code or target screenshot.
Grading uses a judge-then-refute protocol. One model tests the finished app in a browser, checks source evidence and grades each criterion as pass, partial or fail. A second model challenges every positive verdict and looks for stubs, hardcoded values and features that cannot be reached. Runs were capped at three hours and 120 turns. The main comparison held the Claude Code harness, system prompts and tool interfaces fixed across all ten endpoints.
The selection process has a deliberate bias. Candidates were ranked partly by historical failure signals, including repeated reformulations and explicit rollbacks. The authors preferred requests that had already given agents trouble, and allowed at most three queries per session to become one task. That makes the benchmark a stress test of difficult revisions, not a random sample of all coding requests.
The corpus also spans five input types, which changes what the agent must do. For an audio or video request, the agent has to extract a value or a visual reference before it can edit anything. The benchmark records those steps because perception failures can look like editing failures in a final score.
Which coding agents scored best, and on which tasks?
GPT-6 Astra led both tracks. On multimodal edits it passed 75.4% of criteria and fully completed 41.0% of tasks. On repository edits it passed 66.7% and fully completed 17.9%. Combined across all 273 tasks, it reached 69.9% strict completion and a 26.4% full-pass rate.
The next places were close. Claude Opus 5 finished at 66.1% combined, Qwen3.8-Max at 65.3% and Qwen3.8-27B at 64.4%. Kimi K3 reached 62.9%, while Grok 4.6 came last at 51.1%, with only 6.6% of tasks fully completed. The spread between the best and worst combined score was 18.8 percentage points.
The ranking is not stable across settings. Qwen3.8-27B ranked fifth on multimodal edits but second on repository edits, where its 64.5% narrowly beat Opus 5 at 64.2%. Kimi K3 moved the other way, from fourth to sixth. The two track rankings correlate at a Spearman coefficient of 0.818. That is high, but it is not a substitute for testing the setting you care about.
Task type changes the picture again. Functional Implementation made up 46.2% of tasks. It had the highest pooled strict completion at 61.9%, but the lowest full-pass rate at 10.3%. Grounded Understanding tasks reached 60.4% strict and 20.2% full. UI Transformation sat between them at 61.5% strict and 13.6% full.
Models also specialise by input. Qwen3.8-27B led text-only tasks at 77.9%, Opus 5 led image tasks at 89.1%, and GPT-6 Astra led video, audio and document tasks. Documents produced the highest pooled strict score at 65.2%. Yet only 18.2% of document tasks were fully passed, against 37.0% for tasks with no attachment.
Those numbers come from a single evaluation run per model and task. The authors describe the comparisons as descriptive, because task groups differ in content as well as labels. A high average can still hide a long tail of unfinished requirements.
Why do broader changes fail more often?
Repository tasks that demanded three or four kinds of change scored 59.1%, compared with 65.4% for tasks with one or two. That is a 6.3-point gap. The full-pass rates were almost identical, at 9.7% and 9.1%, but zero-pass outcomes rose from 0.9% to 5.8%.
The pattern suggests a threshold rather than a smooth decline. When a request mixes layout, styling, interaction and content, an agent may satisfy each part in isolation. It can still break the links between them. A renamed field can update one view and leave another stale. A new control can work until a user reloads the page.
More effort did not buy more completion. On multimodal edits, GLM-5.2 used 69 median turns and 7.29 million tokens per task for 57.2% strict completion. GPT-6 Astra used 32 turns and 1.24 million tokens for 75.4%. On repository edits, Qwen3.8-27B spent 57 turns and 6.72 million tokens for 64.5%. GPT-6 Astra spent 48 turns and 2.33 million tokens for 66.7%.
The tool traces show agents building their own perception pipeline. OpenCV appeared in 68.5% of video runs, ffmpeg in 31.5% of video runs and 25.0% of audio runs. Faster-whisper appeared in 4.4% of audio runs. Some agents also downloaded and ran pretrained vision or audio models to describe media before editing.
Effort tracked task type as much as model quality. Video tasks had a median of 44 turns, against 27 for text-only tasks. Image inspection averaged 14.47 calls per multimodal run and 2.78 per repository run. The share of edit and write calls rose from 7.3% to 27.6% between the two tracks.
The authors are careful about causation here. These profiles describe how agents spent effort, not proof that a particular tool or turn count caused a better result. The benchmark records behaviour, and the differences between tasks remain large.
Does the agent harness matter as much as the model?
A separate experiment kept the model fixed and swapped the harness. The harness is the program through which an agent reads, writes and runs code. The authors tested Kimi K3 and Qwen3.8-27B under three harnesses: codex 0.142.0, opencode 1.18.13 and qwen-code 0.21.11.
For Qwen3.8-27B on repository edits, completion ran from 56.0% under qwen-code to 63.8% under codex and 74.9% under opencode. That is an 18.9-point spread for the same model on the same 173 tasks. On multimodal edits the same model ranged from 72.2% to 80.1%.
Harness rankings reversed between the two models on multimodal edits, with a Spearman correlation of minus 1.00. Kimi K3 scored highest under qwen-code and lowest under opencode, while Qwen3.8-27B showed the opposite order. On repository edits the correlation was 0.50.
The gap between the two models swung by 10.0 percentage points on multimodal edits. It swung by 16.2 points on repository edits, depending on the harness. The paper concludes that a leaderboard listing only the model tells part of the story. It recommends reporting the model and the working environment together.
For buyers, the practical test is a pair, not a model name. A strong model in a weak harness can underperform a weaker model in a better one. The paper does not identify which harness features cause the difference, because it swapped whole environments rather than individual tools.
The result also complicates comparisons across vendors. Two teams can run the same model and report very different completion rates without either being wrong. The execution setup, retry policy and tool access all belong in the disclosure.
How does this compare with earlier evidence on AI code quality?
The finding echoes work published by METR in March 2026. METR asked four maintainers from three repositories to review 296 AI-generated pull requests that had passed SWE-bench Verified. They also reviewed 47 human-written patches that were actually merged. Roughly half of the test-passing AI patches would not have been merged.
On average, maintainer merge decisions ran about 24 percentage points below the scores from the automated grader. METR also estimated that merge rates improved 9.6 percentage points per year more slowly than benchmark scores. It described that trend as suggestive rather than firm.
METR warned against reading a benchmark score as a real-world success rate. The note says "a naive interpretation of benchmark scores may lead one to overestimate how useful agents are". It also notes that agents were not allowed to iterate on feedback in that study. That is a genuine difference from normal development.
The Co-Design Bench authors make a related point about saturation. On 150 WebCompass-edit tasks, switching Qwen3.7-Plus from single-shot answering to agentic execution raised mean graded feature completion from 73.6% to 95.1%. Opus 4.8 reached 99.4% under agentic execution.
A test that most models nearly complete cannot rank them well. Co-Design Bench tries to restore difficulty with real requests, compound changes and hidden criteria. Its 26.4% full-pass rate is the result. That is far below the near-perfect scores that agentic models post on older editing tests.
The two bodies of evidence point in the same direction. Partial credit and automated tests measure something real, but they do not measure whether a maintainer would accept the work. Both papers treat complete, reviewable delivery as the harder and more useful target.
What are the limitations, and what should teams watch?
The benchmark has acknowledged limits. Each task contains one fixed request, so follow-up questions and mid-task corrections are not tested. That removes a large part of normal collaboration. In normal work, a user notices a problem and asks for a change before the work is finished.
The judges are language models, and they can make mistakes. The authors ran a sensitivity check with three alternative judges and report that relative rankings and score distributions stayed broadly stable. Even so, automated grading of visual and interaction claims is not the same as expert review.
There is also a conflict-of-interest question. The authors include Alibaba staff, and two Alibaba endpoints, Qwen3.8-Max and Qwen3.8-27B, are among the ten evaluated. Neither model tops the table, which weakens the simplest version of the concern. The independent ATLAS-Eval catalogue still flags the link and notes that contamination has not been assessed.
The task mix limits generalisation. Requests were chosen partly because they had already caused reformulations or rollbacks, so the benchmark over-represents revisions that agents find hard. A 26.4% full-pass rate should not be read as the success rate for every coding request in the wild.
Execution was capped at three hours and 120 turns. Wall-clock timeouts were recorded as infrastructure failures and kept in the denominators, so the raw scores are conservative in that respect. The paper also notes that endpoints were sampled and the exact generations cannot be reproduced from a model name alone.
For teams, the useful lessons are practical. Measure complete tasks, not just passing tests or accepted suggestions. Test the model and harness together on your own repositories. Keep a human review step for compound changes, and treat vendor claims that cite only average criteria as incomplete. The next signals to watch are independent replications, harness-aware leaderboards and whether follow-up turns become part of the evaluation.