Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
Another Qwen doesn't seem all that scary, a few nation-state actors have likely exfiltrated weights of much stronger non-open models and abliterated them for use in large-scale collection on targets foreign and domestic. But god forbid someone mods a video game or jailbreaks their phone without applying for Claude Mythos or Daybreak, right?
If Anthropic truly believes we are on the verge of catastrophe but nobody is listening because they've been crying wolf for years about it while they get rich then they should truly do something. By "do something" I don't mean try again to convince the world to slow down as they IPO their bleeding edge research and solution (which is creating the doubt you mention) but just say "hey, we're so sure we're months away from worldwide disaster that we're shutting down our AI operations completely in hopes we can all put some attention to this before it's too late - will you follow or work together to come up with ways to prevent this problem from happening".
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe we could stop a billion people from dying" then what good is the current approach doing them?
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
[1]: https://artificialanalysis.ai/models/qwen4-max
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
Looking forward to the "Flash" and 27B versions.
I wish there were more ~60b dense options like ripple too!
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
Models often get great scores despite being a lot worse than the frontier in practice.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Ah yes, the reverse-benchmaxers of OpenAI models.
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe we could stop a billion people from dying" then what good is the current approach doing them?
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."