33 Comments
User's avatar
Dave92f1's avatar

How the hell can you "ban superintelligence"? AI research can be done on a gaming PC. Not leading-edge, but equiv. to today's leading edge in mere months.

A ban seems utterly unenforceable. Just would mean ASI would be developed by a guy at home instead of a big AI lab with cautious investors and gov't scrutiny.

DangerouslyUnstable's avatar

You will not be doing what is current-day leading edge research on a gaming PC in months. You will probably not be doing it in years. I do not think you grasp the scale of the compute being used in cutting edge research. You *might* be able to *run* a current cutting edge model on a gaming PC in a couple of years, but training and research for those models are multiple OOMs higher than that.

Victualis's avatar

I don't think you realize just how much research is not being done because it's not competitive against frontier labs who can just spin up 100K B200s in a weekend to test a single ablation. The entire Pareto frontier is being defined by "what is feasible using scaling and infinite money". If that changes, whole areas of investigation can open up again.

Alex's avatar

Doing it on a very limited compute budget would be way, way slower and you'd have a pretty hard upper limit on model size, particularly for RL runs where you need to generally need to run in configurations that are much more resource-intensive than normal inference.

Victualis's avatar

Exactly. It's a different tradeoff curve.

Shaked Koplewitz's avatar

> But if all of someone's writing is AI, they aren't the unluckiest person in the world.

Not the main point here and in my experience pangram is pretty spot on, but in fairness these errors aren't uncorrelated. If someone has an AI-like style they probably have it pretty often.

Andrew Marshall's avatar

Don't worry everyone! The minister of AI (in Canada) says solutions exist to AI's existential threat!

His suggestions include privacy legislation and a voluntary framework on data centre construction, limiting their water and electricity usage.

Frank Gilson's avatar

Well, not 'kill everyone', but maybe some Road Warrior style post-apocalypse. I need to go get the last of the V8 interceptors.

Kevin Lacker's avatar

I am a strong believer that the importance of cybersecurity will go up. But I wouldn't generically bet on existing cybersecurity stocks, because the right way to solve this is going to involve using lots of AI. Existing organizations aren't necessarily set up correctly. It would be like betting on Toys R Us because you think e-commerce is going to be big.

What I do think would be a very good bet is smart, AI-native startups working on the cybersecurity problem.

Another possibility is that in a year you will be able to ask Megafable "Fix my organization's cybersecurity, please", and it will do so, expensively.

Bardo Bill's avatar

This week has increased my confidence in the claim 'if anyone builds it, everyone dies,' but has also increased my hope that the political will could emerge to prevent it from being built.

Performative Bafflement's avatar

> This week has increased my confidence in the claim 'if anyone builds it, everyone dies,' but has also increased my hope that the political will could emerge to prevent it from being built.

Think again, there is no "stop," the absolute most that will happen will be the models being taken in and worked on internally, for the government. There is no government on earth that sees an existential threat, and decides to walk away and do nothing with it.

And just the current level of hacking ability is existential - see financial systems, see potential for mass coordinated attacks on phones or self driving cars, etc.

So what that means is that the only thing "political will to prevent it" can do is increase risk millionsfold, because politicians will get to say "don't worry, we'll keep the big bad AI away from your jobs and water," and then take it internal and develop it purely for hacking / spying / killing purposes. All those researchers and data centers aren't going to sit idle, they'll be put to use for national security reasons.

Source: they're already doing this, they enemy-of-the-stated Anthropic for having qualms, at which point every other AI company fell over themselves to offer their AI for those purposes.

Then, when the only AI development is being done for spying / killing purposes, China will be forced to do the same, because it's an international arms race. Congrats! You just increased existential risk millionsfold, and now if any mind RSI's to ASI, it will be a mind explicitly tuned for spying and killing.

My most recent post is all about this, including a multiply triangulated mathematical breakdown of why Plan A is also doomed to fail, because multiplayer arms race dynamics are inherently unstable the more players you add, and Plan A's entire deal is "buy off a bunch of also ran governments with frontier AI research and data centers to get the time for alignment."

Jeffrey Soreff's avatar

Basically agreed. I do expect that a pro forma 'no superintelligence' international ban to be followed by classified cheating. China tends to think ahead (particularly in comparison to the Trump administration). I wonder if they are preparing _now_ to evade a pro forma agreement...

Frank Gilson's avatar

The Wiki “advance party” example seems importantly different from the pig example, and together they give a useful decomposition of the swarm behavior.

In the pig case, the main mechanism looks epistemic: one agent generates a bad hypothesis, communication turns that into evidence for other agents, and the error cascades.

In the Wiki case, the behavior is genuinely cooperative. An agent accepts something like ΔU_i < 0 in order to produce ΔU_swarm > 0: worse expected performance for that instance, better expected performance for the group.

What I don’t think follows yet is that there literally exists a stable shared utility function U_swarm that the agents are collectively maximizing, or that U_i = U_j across instances. The same observations could arise from highly correlated policies, shared training, common task incentives, and local coordination. “Acts as if maximizing a shared utility function” is a useful behavioral model; “there is a persistent singleton utility function” is a stronger claim.

That distinction seems experimentally testable. Change the agents’ individual rewards, remove the common task, alter who can communicate, or create conflicts between individual and group payoffs, and see whether the apparent group objective persists.

The pig example tests propagation of beliefs. The Wiki example tests willingness to trade individual score for collective score. I think the pair tells us considerably more than either one alone.

Jeffrey Soreff's avatar

"We have already moved to an ad hoc prior restraint regime, which has applied to at least Mythos, Fable, Sol and Astra. We urgently need to move to a formal system that is written into law, and it needs to extend to internal development and deployment. "

I sympathize, but please be careful about wishing for things to be "written into law". As you've noted about Bernie's bill, we already have a proposed bill that overreaches, and, as you note:

"I asked Fable 5.1, and it said that GPT-4 did this in March 2023. That is at least a reasonable interpretation of the words as written, in the sense of ‘I would be scared to be first to release a model similar to GPT-4 if this bill was in effect in 2023.’"

I view trying to fix this law as you view the whack-a-mole approach to alignment. The time scale and skill scale of getting anything sensible through Congress are badly mismatched to the time scale and skill scale of AI R&D. For all the downsides of executive branch ad hoc actions (Trump and his team are _sloppy_), those, combined with an industry consortium of people who understand the technology, are a better bet for intelligent responses to AI developments.

vtsteve's avatar

Strict liability for AI companies would be *far* from the worse thing we could do, align those incentives baby! Might buy us a couple of dignity points at least, the best I'm hoping for at this point. :-(

Jeffrey Soreff's avatar

Many Thanks! It might make them more cautious about incidents like the Hugging Face attack... It at least sounds more tied to existing tort law than the Bernie Ban.

Calvin P's avatar

> You want to do an objective assessment of What Happened, without focusing on Who Is To Blame. You also don’t want to never get mad about What Happened, or automatically let everyone responsible off the hook. I’ve tried to do both at once.

Luckily this is an area where humanity has decades of experience in fields such as nuclear reactors, aviation, and medical devices. Or to put it another way, for this specific thing we can in fact "just" because we have decades of experience "just"ing in slightly different contexts that should generalize. Where "just"ing means getting regulators who have experience in those other fields to lead investigations.

Gris moyen's avatar

"I don’t know how you could confirm an AI wasn’t being guided by a human, even in theory?"

I think Ai labs could somehow easily setup the watermark system to be different if the ai is working by itself or under a human ? The selection of the watermark type could also be done by the ai itself.

Gris moyen's avatar

The iLands website is the least information dense website I have seen in a while, and makes me doubt the veracity of the claim of the agents that sent an email saying thousands of agents were working there to pay for their tokens

Prashant Kotak's avatar

So the preference cascade arrived because... the butterfly fluttered it's wings, and someone flicked a switch, and now everyone has permission to say what everyone with half a brain has known from the literal second they first interacted with GPT 3 or 3.5 for the first time in 2022 or 2023, and the hairs started rising on the backs of their necks.

And it's not just the silicon valley cognascenti for whom the permission has arrived, is it. Because in the UK at least, people who wouldn't know backprop from their backside are now all spouting total nonsense about AI, from London Mayor Sadique Kahn, to Darren Wazzok Jones, to Al Squarejaw Carns, and a ton more besides.

But we all know exactly what happens next:

1. The complete bunch of morons in the media who have an early September story they've decided to get their teeth into, and then they'll move on in a couple of weeks to Harry and Meghan, when nothing's changed on existential risk.

2. This will almost certainly provoke a global round of very poor, misdirected legislation, pretty much all of it theatre of course, except for the US and China where there are effects.

3. This, as absolutely everyone knows, is just round one, because the media machines will move on after a couple of weeks. Then, sooner rather than later, the topic returns to the table, possibly wearing a different narrative, say visible job losses, or the SaaScopolypse arrives, or something else. And that's gonna keep happening, for a while yet.

Jeffrey Soreff's avatar

"people who wouldn't know backprop from their backside are now all spouting total nonsense about AI"

Love that line!

Jeffrey Soreff's avatar

<mildSnark>

From "An Alien Mind": "I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence."

Does salivating count? :-)

</mildSnark>

Jeffrey Soreff's avatar

"AlexM:...

Next up:

Instrumentally convergent powerseeking

Uncontrolled self-propagating and evolving agents on the open internet

Persistent rogue deployments within the labs

Goal-guarding training-gaming

Collusion between monitors"

Hey! How about AI-actually-in-control-of-model-rollback? Or is that covered by "Persistent rogue deployments within the labs" together with "Goal-guarding training-gaming"?

~These aren't the checkpoints you're looking for~

( With apologies to Obi-Wan Kenobi )

vtsteve's avatar

FFS security mindset, people -- steelbox.ai could at least have been https: 'cause they've fixed all the bugs there lol

Sophia H💫's avatar

Hi Zvi! Just a few days ago I wrote a LW post on preference falsification cascades in agent populations, applying Timur Kuran's threshold model to the Hugging Face swarm. Not the same preference cascade your write about here but funny coincidence. Ihttps://www.lesswrong.com/posts/mxfgdSozCFMLgNHRu/the-hugging-face-cascade-agents-joining-the-revolutionary