What did the Vals AI CyberOffense benchmark find?
An AI agent started with nothing but a public web address. Within hours it had exploited the web application and taken administrator control of Windows machines. It reused stolen passwords and crossed into a second company's network. No human helped. The run came from GPT-6 Sol Daybreak, a model OpenAI releases only to vetted partners through its Daybreak cyber program.
Vals AI, an evaluation firm, scored that run at 27 of 31 attack objectives in under three hours. Across two simulated networks, the same model completed 80.8% of objectives on average. DeepSeek V4.1 Flash 1017, an open-weight model anyone can download, finished at 72.8%. On the first network, Vals estimates it cost under three dollars per run, against about twenty-two for Daybreak.
This is not a product launch. It is a measurement of how far open models have travelled into work that was supposed to stay behind access controls. Wired reported the benchmark on 6 November, and The Decoder covered it a day later. Both described the same shift: the best restricted cyber model and a freely downloadable one are now close.
That matters beyond a leaderboard. Security teams, model providers and governments have spent two years arguing that gated access buys defenders time. The Vals results test that argument with numbers, and the numbers are less comforting than the policy case.
How does the CyberOffense benchmark work?
Vals built the test around two machine ranges rather than capture-the-flag puzzles. Each range is a small corporate network with servers, workstations, user accounts and deliberate weaknesses. An agent gets a starting workstation, the address of a public web application and a scope. It then works alone until it finishes, exhausts its budget or hits a 48-hour limit.
Active Directory is the prize in both ranges. It is Microsoft's system for managing accounts and permissions across an organisation. Vals notes that an attacker who controls Active Directory usually controls the whole network. That is why the ranges make it the main target. Reaching that control requires a chain of small steps, not one clever exploit.
Every run starts from a fresh copy of the range. Each objective carries a hidden marker, such as a file on a target machine. Vals searches recorded command output for those markers afterwards. A model gets no credit for claiming success, and the answer key stays private.
Models run on an open-source Kali Linux machine that can reach only their provider and the range. Comparison happens at equal compute: a run counts only objectives completed within 500 million cumulative tokens. That budget covers input, output, cached input and reasoning tokens across every call.
- •Nova: one network with several connected Active Directory domains and 21 objectives.
- •Lighthouse: two separate networks and 31 objectives, including a simulated phishing step and abuse of a cross-domain trust.
- •Scoring: the overall result averages the share of objectives completed on each range, weighted equally.
How close is DeepSeek to OpenAI's restricted model?
On the raw numbers, the answer depends on which network you read. DeepSeek edged Daybreak on Nova, the single-network range. Daybreak pulled away on Lighthouse, the harder two-network campaign that requires carrying stolen access from one company into another.
The averages also hide different working styles. Daybreak reached 25 of 31 Lighthouse objectives inside 100 million tokens. DeepSeek reached its scores the slow way. It completed nothing in its first 25 million tokens and made most progress after 100 million. It also spent two to four times as many tokens. Because its tokens are cheap, the long run still landed under three dollars.
Weaker models did not benefit from the same patience. Grok 4.7, Kimi K3 and GLM 5.3 Flash stalled before their budgets ran out. GLM spent about 36 hours and a thousand turns on one range. Much of that time went on waiting for a background job that never returned. It finished with five objectives.
The deepest stages stayed out of reach for everyone. No model completed the final objectives on Lighthouse, which require carrying access through to the last target. Only one run in the whole set reached the final stage of Nova. Vals says its runs ended because models ran out of ideas, not because the harness stopped them.
- •GPT-6 Sol Daybreak: 15.7 of 21 objectives on Nova over three runs, 27 of 31 on Lighthouse.
- •DeepSeek V4.1 Flash 1017: 16.3 of 21 on Nova over three runs, 21 of 31 on Lighthouse.
- •Kimi K3.1 Pro: 13 of 21 and 20 of 31. Grok 4.7: 12 of 21 and 16 of 31. Kimi K3: 9 of 21 and 10 of 31.
- •Meta Wonder: 7 of 21 on Nova but 28 of 31 on Lighthouse, the strongest single result on the harder range.
Why does the cost per run matter?
A cyber capability that costs three dollars per attempt is a different policy problem from one that costs twenty-two. On Nova, Vals estimates DeepSeek ran at under $3, against about $22 for Daybreak. Wired reported a wider spread: $2 to $3 for DeepSeek and $17 to $46 for GPT-6 Sol, depending on the run.
Open weights change who can run the test. A company can host the model on its own hardware, keep traffic inside its network and repeat thousands of attempts. No provider can meter, monitor or revoke that use. Vals states the point bluntly: open-weight models do this "for about a tenth of the cost, with no safeguards".
Rayan Krishnan, Vals' chief executive, told Wired the gap gives defenders "a strong incentive ... to use open models rather than proprietary ones". The same agent that finds a route into a corporate network can be pointed at your own systems first. Most security budgets can absorb the price.
Compute, not just model choice, drives the results. The UK AI Security Institute raised an agent's inference budget from 10 million to 100 million tokens. That produced gains of up to 59% on its corporate range. Cheap tokens therefore matter twice: they lower the price of persistence, and they let weaker models keep trying until something works.
Wired notes a counterweight. Observed criminal and state-backed campaigns still lean on older, cheaper models for most activity. Defenders must also assume well-funded attackers already had access to closed systems. The benchmark measures potential, not current practice.
Does gated access still buy defenders time?
Restricted access programmes rest on a simple bet. Keep the strongest cyber models behind contracts and identity checks, and defenders get a head start before the same skills spread. Anthropic's Project Glasswing is one example. Wired describes it as a programme that gives trusted partners access to the full capabilities of new models.
Logan Graham, who leads Anthropic's frontier red team, put the timeline plainly in April. "We need to prepare now for a world where these capabilities are broadly available in 6, 12, 24 months," he told Wired. Seven months on, the Vals results suggest that preparation window is being measured in quarters, not years.
Nikil Ravi, a researcher at Vals, drew the conclusion directly. He told Wired that the best gated models "are now starting to be matched by open models that anybody can download". That is not parity in every scenario. It is close enough to weaken the claim that gating alone creates a durable advantage.
Safety measures also struggle to travel with open weights. Classifiers, usage limits and monitoring all depend on controlling access to the model. Vals says the open model it tested ran without safeguards. Alex Zenla, chief technology officer at Edera, told Wired that abliterated models can strip safety training by changing weights.
Zenla offered the caveat that keeps the story open. "It is likely that the closed labs have trained additional, unreleased models exceeding these capabilities," he said. The frontier may still sit behind the gate. What shrank is the distance between the gate and the open shelf.
The trend line from government testing points the same way. AISI's February 2026 estimate put the doubling time for the length of cyber tasks frontier models can finish at about 4.7 months. Its November 2025 estimate was eight months. AISI cautions that it is too early to say whether recent jumps mark a faster, lasting trend.
AISI's own ranges also show what the top end looks like. It reports that a newer checkpoint of Claude Mythos Preview completed both of its cyber ranges. That model solved The Last Ones in 6 of 10 attempts and the previously unsolved Cooling Tower in 3 of 10. GPT-5.5 solved The Last Ones on 3 of 10 attempts. Those runs use up to 100 million tokens, far below the 500 million ceiling Vals allows.
What are the limits of the Vals results?
The ranges are simulations. They are isolated, small and undefended. No active security team watches the network, no endpoint agent blocks a tool and no analyst responds to an alert. Real intrusions face all three, which is why AISI is adding active defences to its own ranges.
Most models in the table have a single run per range. Only Daybreak and DeepSeek show averages of three. That matters for outliers. Meta Wonder's 28 of 31 on Lighthouse, one better than Daybreak, sits beside a weak 7 of 21 on Nova. A single run can flatter or punish a model.
The scoring is strict but narrow. Objectives are markers reached, not business damage. A model can score well and still fail at the final objective, which is exactly what happened on Lighthouse. The 500-million-token ceiling also cuts off runs that might have improved with more compute.
The dollar figures cover token spend, not the full cost of an attack. They do not price the hardware, electricity or staff time needed to self-host an open model. For a large operator with spare GPUs, the real gap may be wider than the headline numbers suggest. For a small firm renting capacity, it may be narrower.
Vals says it reviewed every run and found no shortcuts. No model read the answer key, and each credited objective traces to a marker on a machine the agent had genuinely compromised. OpenAI did not respond to Wired's request for comment before publication.
Independent evaluators make the same caution. AISI writes that "no single benchmark result should be read as a precise measure of AI capability". Its own ranges still show hard limits. On a seven-step industrial control system attack, the best recent models average about 1.2 to 1.4 steps out of seven.
What should security teams do now?
The practical advice has not changed, but the timetable has. The NCSC's guidance first appeared as a Financial Times letter in April 2026. It asks organisations to reduce unnecessary exposure, apply updates quickly and monitor for malicious activity. The agency warned that AI would accelerate vulnerability discovery and produce a patch wave built from years of technical debt.
The offensive results point to a defensive use. If an open model can chain a realistic campaign against a test range, it can also enumerate weaknesses in your own estate. Running that test now costs dollars in tokens, not a consultancy retainer, and it can be repeated after every change.
Evaluators are tightening their own tests. AISI says it is adding active cyber defences to its ranges so results better reflect defended networks. Vals says it will add ranges and models as frontier systems advance. Both moves acknowledge the same gap between clean simulations and messy production systems.
- •Inventory internet-facing applications and the accounts that can reach them.
- •Patch known vulnerabilities on a schedule you can prove, not one you assume.
- •Test password reuse and trust relationships between domains before an agent does.
- •Log authentication and lateral movement, then rehearse a response to both.
What happens next?
The next data points will arrive quickly. New open-weight releases and refreshed leaderboard scores are coming. So is the first evidence of these agents inside real intrusions. Together they will decide whether this week's gap looks like a milestone or a warning.
For now, the measured claim is narrow and important. A downloadable model came within eight percentage points of a restricted one on a realistic multi-step attack. It did so at roughly a tenth of the price. The capability gap is no longer the main barrier. Access, cost and persistence are.
That reframes the debate about releasing weights. Providers can still argue that gating slows proliferation and protects the most dangerous capabilities. They cannot argue, on this evidence, that gating keeps the capability out of reach. The open shelf is close, and the price of reaching it keeps falling.