50 Comments
User's avatar
Laiba Rehman | Rowan Jupiter's avatar

About America.gov, it's been patched now but before if you typed "hello" and then "play Minecraft" it would go crazy with an American version of the Minecraft end poem, including gems such as "and the republic said you are the daylight" and "and the republic said I love you because you are the reason we have a ZIP code at all."

https://x.com/LaibaRehmanRJ/status/2105685847064326409

Evan's avatar

> If we already have r~0.15, then you should expect it to exceed 1 as we discover new methods and scaling laws along the way

r is already a measure of exponential growth. If you expect r to increase steadily then your model is implicitly super-exponential, which I think requires some justification.

FWIW I think Ramez is wrong about the current r, and we are on track for a nice smooth exponential foom. But I don't think we are on a super-exponential path.

vectro's avatar

I mean, if we chart r over time, is it holding steady? Doesn’t seem like it to me.

Evan's avatar

https://artificialanalysis.ai/#frontier-language-model-intelligence-over-time looks like a pretty normal exponential to me but I haven't actually run the regression.

tup99's avatar

> I strongly endorse journals having a retroactive zero tolerance policy for AI writing, where getting a submission caught at any point gets you retraction and a long or permanent ban, ideally across many journals.

JFC.

Does this not seem... Draconian?

If we feel that the only way to deal with the deluge of AI slop is to impose this death penalty, I can see that. But why make it retroactive, before people were informed that they would be subject to the death penalty?

Will Morgan's avatar

Retroactive to better AI detectors, not retroactive to the rule. The linked tweet says it clearer.

Kenny Easwaran's avatar

Notably, this particular policy change was made after the journal accepted a paper that was submitted together with a statement that it was entirely AI-generated that the editors somehow missed (understandably, given how awkward most journal editor software systems are). The policy only applies to submissions made after the policy was officially posted on the website, which was about three weeks after it was tweeted, and was discussed very widely on philosophy blogs.

Coagulopath's avatar

I think we need something like email blacklists, where a domain caught spamming instantly goes on the list (blocking even legitimate emails) but reputation eventually heals - after enough time without spam reports, you fall off the list.

Jeffrey Soreff's avatar

To my mind, the ban on AI generated papers seems to be filtering along the wrong dimension entirely. If an AI generated paper is good, publishing it can advance the field. If a human paper is slop, preventing it from occupying the attention of readers is useful. I sympathize with the editors caught in the middle of a flood of an increased number of papers. And the AI contributors should be listed as authors or co-authors. But rejecting a good paper because the author is silicon does not serve the field.

tup99's avatar

I think if Zvi's proposal was a pragmatic solution to deal with the flood of papers, it's not particularly helpful to point out the problems with this pragmatic solution unless you can suggest a pragmatic alternative. What would you suggest, that is as readily implementable and as effective as the (non-draconian) version of the proposal?

Jeffrey Soreff's avatar

Many Thanks! Two possible suggestions:

a) Use AI heavily to select the good papers, or at least to do preliminary filtering. I don't know if SOTA LLMs' 'editorial taste' is up to the task yet. I _do_ expect that, as with all human economic functions as full AGI is reached, it will eventually be possible for AI to entirely fill the editorial role.

b) Randomly select papers for editorial examination. If the editorial staff can examine N papers a day, and 10N papers arrive, admit that the periodical is limited by the bandwidth of the staff, and examine a random 10% of the incoming papers.

Shaked Koplewitz's avatar

I found the ramez argument reasonably credible, but then even if we accept it it at best argues that AI progress will stay exponential instead of hyperbolic for the next few years. Which isn't exactly a "don't worry we're stalling out" message.

Gustavo José Zambrano's avatar

Superb synthesis. Examining structural incentives explains systemic outcomes far better than surface level explanations.

Andrew Clough's avatar

For myself I'd be really happy if people would distinguish between sentience, sapience, and consciousness when having these conversations.

Gustavo José Zambrano's avatar

A fascinating perspective. The intersection of formal logic and human judgment continues to surprise us in unexpected ways.

Calvin P's avatar

> I still have no idea how he could believe what he said last week

I'll repeat what I said the other day about Jensen Huang. It's really really hard on an emotional level to accept the possibility of AI doom if you haven't already understood and accepted the risks for decades. Being highly smart and intellectual helps you come up with reasons to dismiss it. The reasons sound smart and seem logical to you, and would probably convince a lot of other people. Being smart doesn't always help you overcome your feelings.

The only way out is to spend effort understanding the root of your emotions, go through the process of overcoming them, and then take another look at the opposing arguments and see why they hold weight. It's a serious and painful process, and will probably come with many sleepless nights.

The good thing is, it's helpful that the people on the safety side are respectful and have sympathy. It shows that this isn't an ego trip for them, they're saying what they say because they really believe it.

Sylvain Ribes's avatar

"RSI works if r>1, as in progress accelerates faster than difficulty. If we already have r~0.15, then you should expect it to exceed 1 as we discover new methods and scaling laws along the way, and more intelligent things think of things we did not consider."

But why should r be monotonous over time even if it ever exceeded 1? That simply doesn't make any sense to me.

There's no telling what extra marginal progress will cost at any point of the intelligence curve.

avalancheGenesis's avatar

Sister Dot...is that...Animaniacs? Haven't thought about that franchise for like 30 years. Makes sense from a shenanigans perspective. For gamer points I woulda gone with Onyxia Wipe though. More Dots, we need more Dots! That's a 50 point minus on DKPBench!

Looking forward to Argon coverage. One should never fully count out Countenance or Google, but after Muse being mostly a nothingburger despite company-wide reorg and throwing massive money at hiring, I wasn't really expecting Gemini to (maybe) leapfrog back into the frontier pack again. Very curious how they did it while making no similar business-news waves. (As a langauge model, they should have named it Argot though. It's a bit late to change the schtick to Noble Gases. To keep with zodiastrology theme, it coulda been Gemini Rising...)

User's avatar
Comment removed
Oct 2
Comment removed
avalancheGenesis's avatar

I kinda sorta maybe understand why people spam bot comments on the posts themselves, but as replies to others' genuine 100% human-written comments, it's kind of gross honestly.

Gris moyen's avatar

This is incorrect: "Several Anthropic employees read his blog, so they did it, with Fable 5.1 and about $100 in credits with prompts like ‘keep going,’"

From the anthropic blog post:

"Claude ended up doing the calculation two different ways: the original bootstrap, and the indirect form-factor approach. Either approach would have cost an end-user around one or two thousand dollars, mostly due to the expense of running Claude for so long. The bootstrap calculation, done with the Python programming language with package SymPy, took around $100 of the budget, corresponding to running 96 CPUs for a week."

Gustavo José Zambrano's avatar

The intersection between empirical models and cognitive augmentation is something I study closely: https://gzambrano.substack.com

Blissex's avatar

«Anthropic analyzes the cyber capabilities of GLM-5.3, which lacks robust safeguards against misuse. [...] This should update us towards slow diffusion in misuse, and a general ‘people don’t do things’ and that obscurity still has some force behind it. It is good news, and we can hope that it continues for a bit.»

So many people keep worrying about misuse by us "employees" of publicly available models which are heavily restricted and trained on public and legal content; so few people worry about use by big and private "principals" of privately built models with few or no restrictions probably trained also on confidential or illegal content.

Blissex's avatar

«So many people keep worrying about misuse by us "employees" of publicly available models which are heavily restricted and trained on public and legal content»

I guess this is all part of the trend towards "cognitive defense" where our "principals" are quite concerned that our enfeebled minds may be subjugated into behaving badly by Xi and Putin and that we needs to protect out freedom of opinion and press from russian and chinese attacks:

https://geoffmulgan.substack.com/p/the-case-for-cognitive-rearmament

"The announcement last week by Andy Burnham of a new National Centre for Information Defence (NCID) marks an important change in how we think about security and a potentially radical shift towards what I call ‘cognitive rearmament’.

The background to this shift is a dawning realisation that cognitive security can’t be taken for granted. Most would agree that happy and healthy societies depend on people who can think for themselves. Workers who can solve problems. Citizens who can use their votes wisely and can spot lies and deception. [...] IQ levels began to reverse at the beginning of this century in some of the richest countries, including the UK, Germany and Denmark. The PISA scores measuring school children’s levels of attainment peaked in 2012 and then started to decline, with sharp falls in reading, maths and science.

The annual Monitoring the Future study in the US found a significant rise in the proportion of 18-year-olds who struggle to concentrate or learn new things, while 25% of adults in high-income countries (and 35% in the US) are now unable to use “mathematical reasoning when reviewing and evaluating the validity of statements”. [...] With this regression come new threats. The spread of misinformation on social media platforms has been well-documented. Large proportions of the population now believe claims – on crime or migration in particular – which are the opposite of the truth.

Meanwhile, cognitive warfare has become a higher priority for states. Russia’s budget for cognitive warfare has been estimated at $2bn, while China’s is near $10bn. The 20th National Congress in 2022 announced an expansion of the “study and mastery of the characteristics and laws of information based and intelligent warfare” with the brain described as a priority for People’s Liberation Army investment and as the “main battlefield” of future warfare."

Prime Seeker's avatar

I believe your model of the world is incoherent. Would you kindly explain how would a private effort even match the public ones, given that AI is the most visible tech explosion in history, relying on half of the free world in its supply chain (and extremely rare, expensive and visible humans to boot)?

My understanding is:

- a "shadow OpenAI" is outright impossible right now (an attempt to keep one secret would be kind of similar to a black hole)

- anything smaller than that is fully irrelevant in the "full speed" world, as catching up to frontier requires more resources that are currently concentrated on the frontier, and fast-following doesn't help you much in the RSI regime

- any coordinated slowdown relies extremely hard on tracking down the overwhelming majority of the relevant chips

So, not yet a Can't Happen, but a probably manageable risk in specific scenarios.

Blissex's avatar

«- a "shadow OpenAI" is outright impossible right now (an attempt to keep one secret would be kind of similar to a black hole)»

That seems pure hand-waving to me: "shadow" (stealth mode) OpenAI style startups are happening all the time, and one such which is no longer so shadow is called Anthropic.

«- anything smaller than that is fully irrelevant in the "full speed" world, as catching up to frontier requires more resources that are currently concentrated on the frontier

This seems pure hand-waving based solely on PR by supporters of OpenAI and Anthropic because the overwhelming majority of current spending is targeted at inference not training. It it typical of USA business propaganda to claim that they are investing tremendous sums in science and engineering and product development and then things like Linux or OpenStreetMap come around.

As many startups have proven to develop pretty good models it takes small teams and relatively small budgets; for example there are rapidly diminishing returns to parameter count beyond 70-120B. Also very useful weapon models do not need to be "frontier" whatever that means; they can just be pretty good and specialized at doing nasty things; training does not need to be done on latest GPUs, it can even be largely done (more slosly perhaps) on ordinary CPUs, or on older GPUs.

There are many *civilian* HPC clusters with more than 10,000 GPUs for scientific and content generation purposes. Claude Fable mentions that "Epoch AI" tries to keep track of ML clusters and the estimate is that 100-200 have more than 10,000 GPUs.

Anyhow I just asked Claude Fable to estimate cost and training of something like GLM-5 from Zhipu: the estimate is $50-$150m (including overheads) over 4-6 months and technical team sizes are a few dozen people, from companies with total 100-300 people.

The basic problem with nuclear or many chemical weapons is that they need radioactive or specific raw materials which are dangerous to handle and easier to track while ML weapons can be developed with entirely civilian means and data centers do not need to be concentrated in massive facilities. ML weapons are not developed for lowest cost but but of maximum impact.

Prime Seeker's avatar

I still believe your model of the world is incoherent.

Re: stealth mode startups - OK, so how many of them got to the frontier? We had Anthropic catch up and surpass OpenAI, and... that's all? Everything else, in terms of frontier capabilities, has been a failure (assuming that they even try; some, like Thinking Machines, don't seem to). Meanwhile, all of the Chinese models are Claude distills nowadays, so they can't reach the frontier with their current methods even on principle.

Re: propaganda - yes, they're using the majority of their compute to serve inference. No, that doesn't mean that their development compute demands aren't sizeable (we have lots of evidence and the argument from incentives as a bonus). Given their scale, what is sizeable for them is impossible to not notice for anyone else.

Re: small models - irrelevant for the point. Sure, you can have decent specialized agents for cheaper, but they're not major threats, as they're _specialized_ (most prominently for supervised coding), and end-to-end offensive capabilities of any kind require general intelligence. There is a possibility of it percolating to the low model tiers, but for absolute thresholds it's a "nice problem to have" (we'll need multiple years of 2026 progress for small models to have anything like dangerous bio capabilities, and that means we'll then have powerful AIs to solve stuff and won't be existentialed out yet), and for relative ones it's, again, irrelevant (by the time you can squeeze Mythos capabilities into your 100B, everything that survived is already hardened against Kimi K4/GLM 6 via a Darwinian process).

Generally, your posture is that The Real Threat Is Elsewhere. For something to be The Real Threat, it needs to either:

1) have absolute capabilities above a threshold (optionally paired with sufficiently misanthropic actors)

2) have relative capabilities beyond the public frontier

For 1, the only vector right now is the bio risk, and the public SOTA still doesn't cut it. For 2, you "just" need something better than Mythos (because, for example, in an AI war scenario with a proper runaway agent, it will be a Mythos-class model tasked with shutting down rogue deployments).

2 is basically not subject to change anymore, the current frontier (remember it's a moving target) requires insane investments. 1 will eventually catch up with us, and we don't yet know the offense advantage on many relevant risks, but this is too narrow, too far in the future and too non-actionable to matter in the discourse.

I could see a narrow path of "Composer for cyber" (say, Russians with it would be pretty scary), but that's kind of shadowed by the open models anyway, at worst we'll see The Hackening several month early.

Devin Lehmacher's avatar

Links in your table of contents don't work when they contain an apostrophe. Substack expects apostrophes to be trimmed out completely rather than treated as a separator. e.g. [Quickly, There's No Time](https://thezvi.substack.com/i/217261613/quickly-there-s-no-time) links to `https://thezvi.substack.com/i/217261613/quickly-there-s-no-time` which resolves to the top of the article while copying the anchor produces `https://thezvi.substack.com/p/ai-188-gemini-dot-argon?open=false#§quickly-theres-no-time` (the equivalent `https://thezvi.substack.com/i/217261613/quickly-theres-no-time` works too).

Relatedly, I find it a minor nuisance that Wordpress (https://thezvi.wordpress.com) links to Substack rather than its own headings. Claude can probably fix both of these in your publishing workflow at the same time :P

Kenny Easwaran's avatar

Ah that explains why the section link I was using wasn’t working! (I was clicking on the “young lady’s primer” link, because that’s the section I was on when I put down my phone, and mobile Chrome jumped back to the top of the page when I opened it up again.)

Kevin Lacker's avatar

The Ramez Naam article trying to predict future rates of improvement is interesting. There must be some logical cap on the intelligence of a model at a particular size, right? I wonder what we could estimate based on rates of improvement of smaller models. For large models we keep saturating evals so it's hard to predict or measure a maximum but still there must be one.

For example, how small could a model be, and still get over 90% on GSM8K? Right now it looks like around 4B parameters. Do we think that's close to the ultimate limit? Will we ever get below 2B, 1B, etc? (You could stipulate "no benchmaxing" - like instead of literal GSM8K, you could have a larger model generate n problems from a GSM8K-like distribution.)

AT's avatar

Loved the going down faster than everything crypto joke :)

Jeffrey Soreff's avatar

"I do think it would be good practice to say that if you are an AI company and prove [X] in incomprehensible fashion, and want full credit, part of that is that you should support others working to make the proof of [X] comprehensible. Otherwise you only did some of the work, and therefore only deserve some of the credit. "

Sort-of agreed but

a) "you should support others working to make the proof of [X] comprehensible. " Should, at least in the limit, be an automated task. So it may be a matter of tokens expended, not mathematicians funded.

b) Some proofs are just giant case analysis. The four color theorem was proved that way, and, as far as I know, no compact, human comprehensible proof has been found.

Kenny Easwaran's avatar

Coincidentally, there was a new proof of the four color theorem a few weeks ago (overshadowed by Navier Stokes the day before, and the formalization of Fermat’s Last Theorem a week earlier) so Quanta wrote a nice article about the history of the problem: https://www.quantamagazine.org/the-four-color-theorem-gets-a-rare-new-proof-20260910/

It looks like there is a proof that involves just the reduction of 630 cases, but I don’t know that this is enough for “human comprehensibility”.

But still, it’s an important part of (not just mathematical but all academic) work to try to make your work comprehensible to others. If you don’t do that, you’re being like the alchemists, who specifically tried to do amazing things without telling anyone else how it was done.

Jeffrey Soreff's avatar

Many Thanks!

My suspicion is that, in many (but possibly not all) cases, an AI may be able to go through a voluminous proof that passes the LEAN compiler check, and do things like classify steps into 'routine' and 'novel', and perhaps be able to also e.g. construct illustrative cases cases to illuminate the 'novel' parts.

But some theorems may (unavoidably?) require examining a large number of cases, with no nice taxonomy to collapse them down to a small number of kinds of cases...

Re "to try to make your work comprehensible to others"

Hmm... Under normal circumstances, "others" are probably limited to mathematicians in the same sub-field. Might there be a point where "others" are typically AI mathematical agents in the same sub-field?

Kenny Easwaran's avatar

I think your last point is definitely an intriguing thought (and an unsettling one for people who are used to being among the target "others"). I suspect there will be an extended period in which there are two communities that only partially interact, each trying to make things understandable for their own community, and trying to get results from the other, possibly asymmetrically. (This is already the case with many different academic disciplines, and also with non-academic intellectual communities, like the rationalist/AI-safety community Zvi is part of and the academic philosophy community I am part of, both of whom often tend to be dismissive of each other, but probably could learn a lot from each other if we each did a better job of communicating in ways the other could understand.)

On the main part of what you said though, it's notable that as far as I know, no AI has actually written anything like a nice paper presenting a novel result - so far, that part has depended on human labor (which has gotten pretty far with the unit distance conjecture). I don't know how long that might last.

Jeffrey Soreff's avatar

Many Thanks!

"as far as I know, no AI has actually written anything like a nice paper presenting a novel result"

That is my impression as well. My current misunderstanding of the Navier-Stokes result is that it is essentially a counterexample (complex and kind-of "engineered") with the ~100 page proof basically proving its validity.

"also with non-academic intellectual communities, like the rationalist/AI-safety community Zvi is part of and the academic philosophy community I am part of, both of whom often tend to be dismissive of each other, but probably could learn a lot from each other if we each did a better job of communicating in ways the other could understand.)"

Agreed. I get the (vague!) impression that part of this is differing mores, which are hard to bridge. To the extent that the difference is different technical languages - well, one of the successful applications of LLMs is machine translation... :-)

Jeffrey Soreff's avatar

"AI-made five minute video, legitimate banger: You think AI is a normal technology?"

Brilliant! Many Thanks for the link!