"It is not close to Fable, and I do not believe it is that close to Sol."
It does seem to inch out GPT-5.5 and Opus 4.8 on a lot of benchmarks, do you think it's impossible that the gap hasn't been shrunk to 2 months? The naive reading of the benchmarks would imply something like that right?
The benchmarks aren't as valuable as they were. You see a lot of talk of “saturated” in reviews of reviews. All the Chinese models (and most US models) are designed to score higher on benchmarks than on actual real world use.
Trust the part of his essay where he quotes people using it for real tasks more than the benchmarks. But do follow the marks, they're not yet useless. They still give a signal of which areas the model is strongest.
Re: "It is actively suicidal to respond to ‘the Chinese have better models now’ with ‘then we had better sell them the compute so they can run them and also build even better ones.’ Yet every time, yes, people will argue that. Sigh. "
When should we expect nVidia to automate this argument, using a AI agent to scan the current news, and write a topical op-ed-oid arguing for nVidia to sell unlimited SOTA GPUs to the PRC? Perhaps it won't be _people_ arguing this point permanently... This sounds like a feasible task today, albeit there would be AI 'tells' in currently generated prose...
Good roundup. The question I always end up asking with a new model this size is the boring one, what does it actually take to run it yourself, because the capability headlines never mention the VRAM bill.
You'd need roughly 12 H200's, and that's probably the most you could fit in a single server / mining rig setup. It would cost roughly $400k, and would even then be Q4 (heavily quantized), so pretty slow tps.
In more human terms you might try to scrape by with ~70 4090 equivalents, but it's a bad idea bc you would need to split them up across multiple racks / rigs, and that's going to seriously kill your tps / latency and it wouldn't be usable. You need to colocate as many GPU's as you can all on one server / rack / rig, or it kills your tps (tokens per second).
Most "open" frontier models are enterprise grade / served from data centers, external or internal, and will be even more so from here on out, because even Deepseek R1 needed 5-6 H100's to run.
So, thinking this through, there are basically two ways to choose a model:
1. For some users, tasks, and budgets, the answer is "Pay for the most intelligence you can afford, at almost any price." This makes sense when either the human time is highly valuable, or conversely, when the human is no longer able to truly supervise the model. High intelligence models offer more hands-off delegation. Which is sometimes what you want. For example, a non-programmer trying to make money off an iPhone app should just pay for Fable. I have seen Fable literally one-shot genuinely useful Android apps based on a one-page description and the answers to a half-dozen questions.
2. For other users, tasks and budgets, the answer is "Pay for the cheapest model that handles your use case reasonably well." This tends to be most common either for easy use cases, or for use cases with much less delegation. For example, maybe you just need a model to write you Python scripts to transform data formats. Or an experienced programmer may want keep highly involved in the coding, and delegate just the easy/mechanical stuff. This style of work can be viable with much smaller models, including potentially local models like Qwen3.6 27B (runs on a high-end gaming GPU or two) or DeepSeek V4 Flash 284B A13B (runs in 96GB to 256GB of fast RAM). Or your can purchase "near frontier" models like GLM 5.2 of Kimi K3 via the API, especially if weights are released. This gets you slightly iffy Opus-class performance, if you choose well.
So I suspect the US labs are likely to come under margin pressure. They may continue to dominate in category (1), but the history of economics says that we should never underestimate (2).
> But yes, this is one point of potential pain, including risk for anyone using Kimi K3 commercially.
I mean, this is going to come down to "prove it", right? Especially when the weights are released and people can run it on prem if they have the resources and potential gains from doing so. I don't see a viable path to enforcing such regulations today or in the near future.
One thing I find interesting is that the discussion has shifted remarkably fast from “How good is K3?” to “What should governments do about K3?”
Dean Ball criticized the U.S. government’s restrictions on Anthropic’s Fable because they could slow American progress. Now he argues that the U.S. should create regulatory risk around Chinese open models. China, meanwhile, has imposed its own restrictions around strategically important AI assets, whether on acquisitions, frontier companies, or researcher mobility.
The stated reasons differ, but the institutional behavior is becoming increasingly similar. Once frontier AI is perceived as strategically important, states begin restricting the flow of models, talent, capital, and technology.
That makes me wonder whether “Cold War” is actually the right frame. A Cold War suggests an ideological conflict. But what we’re seeing looks more like a recurring pattern of sovereign states responding to shifts in strategic capability. If tomorrow the strongest open model came from India rather than China, I suspect many of the same pressures would emerge.
The irony is that AI may be the industry least compatible with this logic. Frontier models depend on global supply chains, global talent, global research communities, and increasingly global data sources. Yet the more globally integrated the technology becomes, the more states seem compelled to pull it back into the logic of sovereignty.
So perhaps the deeper question isn’t whether open models accelerate or decelerate AI. It’s why frontier AI is so quickly being absorbed into the logic of sovereignty. The model may change. The country may change. But the institutional response increasingly looks the same.
Unfortunately I feel like the benchmarks are diverging from reality for software engineering, because most important work is now done inside the first party harnesses, for Fable and Sol. The benchmarks say crazy stuff like that Gemini is almost as good as the other models, whereas in the harnesses it is quite obviously terrible.
"It is not close to Fable, and I do not believe it is that close to Sol."
It does seem to inch out GPT-5.5 and Opus 4.8 on a lot of benchmarks, do you think it's impossible that the gap hasn't been shrunk to 2 months? The naive reading of the benchmarks would imply something like that right?
Not better than Fable / Mythos and Mythos Preview had a limited release in early April, so it's at least that long (3.5 months).
Oh yeah you are right; weird to me that Opus 4.8 was actually released a month after the preview of Mythos.
The benchmarks aren't as valuable as they were. You see a lot of talk of “saturated” in reviews of reviews. All the Chinese models (and most US models) are designed to score higher on benchmarks than on actual real world use.
Trust the part of his essay where he quotes people using it for real tasks more than the benchmarks. But do follow the marks, they're not yet useless. They still give a signal of which areas the model is strongest.
<mildSnark>
Re: "It is actively suicidal to respond to ‘the Chinese have better models now’ with ‘then we had better sell them the compute so they can run them and also build even better ones.’ Yet every time, yes, people will argue that. Sigh. "
When should we expect nVidia to automate this argument, using a AI agent to scan the current news, and write a topical op-ed-oid arguing for nVidia to sell unlimited SOTA GPUs to the PRC? Perhaps it won't be _people_ arguing this point permanently... This sounds like a feasible task today, albeit there would be AI 'tells' in currently generated prose...
</mildSnark>
Good roundup. The question I always end up asking with a new model this size is the boring one, what does it actually take to run it yourself, because the capability headlines never mention the VRAM bill.
You'd need roughly 12 H200's, and that's probably the most you could fit in a single server / mining rig setup. It would cost roughly $400k, and would even then be Q4 (heavily quantized), so pretty slow tps.
In more human terms you might try to scrape by with ~70 4090 equivalents, but it's a bad idea bc you would need to split them up across multiple racks / rigs, and that's going to seriously kill your tps / latency and it wouldn't be usable. You need to colocate as many GPU's as you can all on one server / rack / rig, or it kills your tps (tokens per second).
Most "open" frontier models are enterprise grade / served from data centers, external or internal, and will be even more so from here on out, because even Deepseek R1 needed 5-6 H100's to run.
Do you have an opinion on how https://manifold.markets/nsokolsky/will-zvi-agree-that-kimi-k3-is-bett should resolve?
My current read is that it should resolve as NO, unless we get a second update post before the end of July.
So, thinking this through, there are basically two ways to choose a model:
1. For some users, tasks, and budgets, the answer is "Pay for the most intelligence you can afford, at almost any price." This makes sense when either the human time is highly valuable, or conversely, when the human is no longer able to truly supervise the model. High intelligence models offer more hands-off delegation. Which is sometimes what you want. For example, a non-programmer trying to make money off an iPhone app should just pay for Fable. I have seen Fable literally one-shot genuinely useful Android apps based on a one-page description and the answers to a half-dozen questions.
2. For other users, tasks and budgets, the answer is "Pay for the cheapest model that handles your use case reasonably well." This tends to be most common either for easy use cases, or for use cases with much less delegation. For example, maybe you just need a model to write you Python scripts to transform data formats. Or an experienced programmer may want keep highly involved in the coding, and delegate just the easy/mechanical stuff. This style of work can be viable with much smaller models, including potentially local models like Qwen3.6 27B (runs on a high-end gaming GPU or two) or DeepSeek V4 Flash 284B A13B (runs in 96GB to 256GB of fast RAM). Or your can purchase "near frontier" models like GLM 5.2 of Kimi K3 via the API, especially if weights are released. This gets you slightly iffy Opus-class performance, if you choose well.
So I suspect the US labs are likely to come under margin pressure. They may continue to dominate in category (1), but the history of economics says that we should never underestimate (2).
really excited to keep testing and using the model. it’s been impressive so far.
What would actually change your mind here, messy adoption telemetry from real users, downstream finetunes, or the first really bad misuse case?
> But yes, this is one point of potential pain, including risk for anyone using Kimi K3 commercially.
I mean, this is going to come down to "prove it", right? Especially when the weights are released and people can run it on prem if they have the resources and potential gains from doing so. I don't see a viable path to enforcing such regulations today or in the near future.
One thing I find interesting is that the discussion has shifted remarkably fast from “How good is K3?” to “What should governments do about K3?”
Dean Ball criticized the U.S. government’s restrictions on Anthropic’s Fable because they could slow American progress. Now he argues that the U.S. should create regulatory risk around Chinese open models. China, meanwhile, has imposed its own restrictions around strategically important AI assets, whether on acquisitions, frontier companies, or researcher mobility.
The stated reasons differ, but the institutional behavior is becoming increasingly similar. Once frontier AI is perceived as strategically important, states begin restricting the flow of models, talent, capital, and technology.
That makes me wonder whether “Cold War” is actually the right frame. A Cold War suggests an ideological conflict. But what we’re seeing looks more like a recurring pattern of sovereign states responding to shifts in strategic capability. If tomorrow the strongest open model came from India rather than China, I suspect many of the same pressures would emerge.
The irony is that AI may be the industry least compatible with this logic. Frontier models depend on global supply chains, global talent, global research communities, and increasingly global data sources. Yet the more globally integrated the technology becomes, the more states seem compelled to pull it back into the logic of sovereignty.
So perhaps the deeper question isn’t whether open models accelerate or decelerate AI. It’s why frontier AI is so quickly being absorbed into the logic of sovereignty. The model may change. The country may change. But the institutional response increasingly looks the same.
Has anyone gotten good results with Kimi Code?
Unfortunately I feel like the benchmarks are diverging from reality for software engineering, because most important work is now done inside the first party harnesses, for Fable and Sol. The benchmarks say crazy stuff like that Gemini is almost as good as the other models, whereas in the harnesses it is quite obviously terrible.
Kimi K3 still needs nvidia chips that China doesn’t have.
Find out why here
https://wavesandpositions.substack.com/p/the-xi-trump-summit-just-proved-america?r=gz32a&utm_medium=ios
Kimi K3 still needs nvidia chips that China doesn’t have.
Find out why here
https://wavesandpositions.substack.com/p/the-xi-trump-summit-just-proved-america?r=gz32a&utm_medium=ios