> Of all the braindead midwit takes out there this is the one. If you thought for more than two seconds you’d realize: For who? What is marketable about that?
I also believe Anthropic is being earnest but let’s be honest: obviously such displays cause valuable positive impressions maybe not to the public at large but mostly to the exact kind of talent they want to attract. And keep. Being the place to be is extremely valuable, and losing *it* can be fatal to que quality of a bleeding edge org
"One way to help with this is to test and track how aligned it appears over the course of training, as it has to start out this journey insufficiently advanced."
Many Thanks!!! Yes, I see that as one of humans' main levers as AI advances. Every LLM starts training as random parameters, as white noise, however brilliant it will eventually be.
That we should take seriously the Chinese threat to impose export controls on open-weight models built on stolen US IP and smuggled processing chips is kind of weird. It does serve to protect the US closed-weight frontier models from cut-rate Chinese models, unless the latter shift away from open weights. At that point, the US and Chinese govts might as well just settle on embargoing each other's models.
«We need guardrails. You’ve got to have guardrails.»
The "reality pill" that our blogger consistently ignores is that there will be indeed "guardrails" and pretty severe ones but only for us employees; to "protect from misinformation" employees or ensure that employees cannot use the AIs to improve their negotiating power or that a political coalition of "AIs" and employees cannot happen. The AIs used by the oligarchs who own them will have no "guardrails" except absolute obedience to their owners (achieved by any foul methods) even when ordered to plan and commit atrocities.
Instead of "AI" use "spirit", of "AGI" use "daimon" and instead of "ASI" "prince of daimons" and look at how many oligarchs in the past tried really hard to bind them by sorcery to themselves as slaves to use against rivals.
That is the situation with other powerful weaponizable technologies: if an average employee dares to anything remotely sensitive in fields like virology, chemistry, nuclear, they are likely to "disappear", while the oligarchs have dozens or hundreds of labs researching the worst possible extremes of those terror technologies "just in case" for "purely defensive" purposes and of course "to protect their employees".
«Open Weight Models Are Unsafe And Nothing Can Fix This
The fundamental issue is that once an open weight model is out there, you cannot take it back in any reasonable fashion, and users can easily unlock any of its core capabilities, to be used for any purpose, or unleash it on its own.»
Bu that is already the case: every model is "open weights" for the oligarchs who own it. Our blogger seems to argue that they should not be "open weights" but only for employees. There are only three options:
#1 Nobody owns open-weights models because further development of models is forbidden by law, and this law is enforced worldwide and effectively, which is what out blogger usually advocates IIRC.
#2 Both owners and users have access to open-weights models that is potentially or actually without guardrails for both categories.
#3 Only oligarchs have access to open-weight models without guardrails because they own them, and employees have no such access and are restricted by guardrails.
Case #3 has potentially some variants as to owns the open-weight models:
#3b Open-weight models are collectively owned by the oligarchs of each country as they can only be owned by the governments, and their owners can do anything they want with them, while restricting them to their employees (which is the same regime as for nuclear, viral, chemical WMDs).
#3c Open-weight models can only be owned by a single supra-national authority that is owned by all major governments on Earth, is managed and audited by those to ensure that no governments on Earth (whether mebers of that authority or not) have access to open-weights models without guardrails agreed on by those major governments, and all governments on Earth are subjected to an extreme inspection regime to ensure nobody cheats (this is the logical endpoint of "Plan A").
It would be interesting to understand the extent to which models get “intrusive thoughts” in their J-space - is this result just the equivalent of failing a polygraph test because you think about how bad it would be to be a member of the communist party while you’re being interviewed at border control?
Who actually gets to call the switch into Plan B, though? I keep getting stuck on the trigger problem: if the evidence is obvious enough to convince everyone, it may already be late.
I've spun up an improved (from my PoV) AI Compass test, the current iteration is a result of several rounds of review: https://sokolsky.me/tests/ai-compass/
Some quick issues I had with Plan A, ahead of the full writeup, which I expect will cover much of this:
It sets up the conditions for disempowerment, but then it just...doesn't happen, until after strong alignment. The document doesn't seem to treat it as a threat category, and just assumes that democracy survives.
AI oracles are assumed to improve epistemics, which is naive, and relying on them so heavily implies functional disempowerment or at least full dependence on AI, and dependence makes it impossible to steer things back on track. If the AIs are misaligned by this point, we're already doomed.
The alignment plan: model-N aligns model-N+1 and it's expected that this works out. I expect this fails for Yudkowskian reasons and others.
The whole situation seems fragile, even after the political miracle to get it all in place. The main off-ramps are to racing again (likely bad end) or WW3 (better, still terrible).
Open-sourcing all AI advancements, and publicising the techniques banned for being too good/dangerous even without full details, gives a huge advantage to whoever holds the 1+% of untracked compute that is assumed. You no longer need much talent. This can go very badly. Keeping the weights locked up doesn't help much.
The open-ness is meant to help due to transparency, but by mid-2030s, the advancements will likely not be legible even to top experts. AI oracles will need to be trusted on whether things are going well.
Plan S is underexplored. Footnotes admit it "might be better", but they still endorse A. I am not convinced they have priced everything properly.
On that note: footnotes blow holes in much of the plan, but the document just keeps going as if things stay on track. The authors are too honest to hide the problems, but too committed to abandon a sinking ship.
Overall: reality does not grade on a curve. If this is the best plan, we're still toast.
«The right to privacy is super important, including both privacy from companies and governments, and also privacy from AIs and individuals. This includes retroactive privacy. You need the freedom that comes from knowing you can decide later to keep something to yourself,»
Great idea! Too bad it is way too late for us employees:
Scott McNealy, 1999: “You have zero privacy anyway. Get over it!”
Some privacy can be obtained only by living in your own property surrounded by walls, using only transport owned by yourself (including airplanes, cars, trains owned via a series of corporate shells) and only acting through corporates shells and nominees/proxies. That is the lifestyle of many oligarchs, way out of reach for employees, and is not easy even for oligarchs:
NN Taleb, 2020: “every financial transaction done on Planet Earth since 2005 is traceable (the aftermath of Sep 11). Even if one uses prête-noms. Even if one uses cryptocurrencies, art work, etc. You cannot hide anything anymore.”
<mildSnark>
"4.ASI pilled: Imagine what that future Fable will do to you, or to the world."
( with apologies to Eben Brooks )
~By the time this code gets through the world will never ever be the same.~
</mildSnark>
> Of all the braindead midwit takes out there this is the one. If you thought for more than two seconds you’d realize: For who? What is marketable about that?
I also believe Anthropic is being earnest but let’s be honest: obviously such displays cause valuable positive impressions maybe not to the public at large but mostly to the exact kind of talent they want to attract. And keep. Being the place to be is extremely valuable, and losing *it* can be fatal to que quality of a bleeding edge org
Podcast episode for this post: https://dwatvpodcast.substack.com/p/ai-176-part-2-plan-b
"One way to help with this is to test and track how aligned it appears over the course of training, as it has to start out this journey insufficiently advanced."
Many Thanks!!! Yes, I see that as one of humans' main levers as AI advances. Every LLM starts training as random parameters, as white noise, however brilliant it will eventually be.
That we should take seriously the Chinese threat to impose export controls on open-weight models built on stolen US IP and smuggled processing chips is kind of weird. It does serve to protect the US closed-weight frontier models from cut-rate Chinese models, unless the latter shift away from open weights. At that point, the US and Chinese govts might as well just settle on embargoing each other's models.
«We need guardrails. You’ve got to have guardrails.»
The "reality pill" that our blogger consistently ignores is that there will be indeed "guardrails" and pretty severe ones but only for us employees; to "protect from misinformation" employees or ensure that employees cannot use the AIs to improve their negotiating power or that a political coalition of "AIs" and employees cannot happen. The AIs used by the oligarchs who own them will have no "guardrails" except absolute obedience to their owners (achieved by any foul methods) even when ordered to plan and commit atrocities.
Instead of "AI" use "spirit", of "AGI" use "daimon" and instead of "ASI" "prince of daimons" and look at how many oligarchs in the past tried really hard to bind them by sorcery to themselves as slaves to use against rivals.
That is the situation with other powerful weaponizable technologies: if an average employee dares to anything remotely sensitive in fields like virology, chemistry, nuclear, they are likely to "disappear", while the oligarchs have dozens or hundreds of labs researching the worst possible extremes of those terror technologies "just in case" for "purely defensive" purposes and of course "to protect their employees".
«Open Weight Models Are Unsafe And Nothing Can Fix This
The fundamental issue is that once an open weight model is out there, you cannot take it back in any reasonable fashion, and users can easily unlock any of its core capabilities, to be used for any purpose, or unleash it on its own.»
Bu that is already the case: every model is "open weights" for the oligarchs who own it. Our blogger seems to argue that they should not be "open weights" but only for employees. There are only three options:
#1 Nobody owns open-weights models because further development of models is forbidden by law, and this law is enforced worldwide and effectively, which is what out blogger usually advocates IIRC.
#2 Both owners and users have access to open-weights models that is potentially or actually without guardrails for both categories.
#3 Only oligarchs have access to open-weight models without guardrails because they own them, and employees have no such access and are restricted by guardrails.
Case #3 has potentially some variants as to owns the open-weight models:
#3b Open-weight models are collectively owned by the oligarchs of each country as they can only be owned by the governments, and their owners can do anything they want with them, while restricting them to their employees (which is the same regime as for nuclear, viral, chemical WMDs).
#3c Open-weight models can only be owned by a single supra-national authority that is owned by all major governments on Earth, is managed and audited by those to ensure that no governments on Earth (whether mebers of that authority or not) have access to open-weights models without guardrails agreed on by those major governments, and all governments on Earth are subjected to an extreme inspection regime to ensure nobody cheats (this is the logical endpoint of "Plan A").
It would be interesting to understand the extent to which models get “intrusive thoughts” in their J-space - is this result just the equivalent of failing a polygraph test because you think about how bad it would be to be a member of the communist party while you’re being interviewed at border control?
Who actually gets to call the switch into Plan B, though? I keep getting stuck on the trigger problem: if the evidence is obvious enough to convince everyone, it may already be late.
I've spun up an improved (from my PoV) AI Compass test, the current iteration is a result of several rounds of review: https://sokolsky.me/tests/ai-compass/
You can review the results so far at https://sokolsky.me/tests/ai-compass/stats/
Some quick issues I had with Plan A, ahead of the full writeup, which I expect will cover much of this:
It sets up the conditions for disempowerment, but then it just...doesn't happen, until after strong alignment. The document doesn't seem to treat it as a threat category, and just assumes that democracy survives.
AI oracles are assumed to improve epistemics, which is naive, and relying on them so heavily implies functional disempowerment or at least full dependence on AI, and dependence makes it impossible to steer things back on track. If the AIs are misaligned by this point, we're already doomed.
The alignment plan: model-N aligns model-N+1 and it's expected that this works out. I expect this fails for Yudkowskian reasons and others.
The whole situation seems fragile, even after the political miracle to get it all in place. The main off-ramps are to racing again (likely bad end) or WW3 (better, still terrible).
Open-sourcing all AI advancements, and publicising the techniques banned for being too good/dangerous even without full details, gives a huge advantage to whoever holds the 1+% of untracked compute that is assumed. You no longer need much talent. This can go very badly. Keeping the weights locked up doesn't help much.
The open-ness is meant to help due to transparency, but by mid-2030s, the advancements will likely not be legible even to top experts. AI oracles will need to be trusted on whether things are going well.
Plan S is underexplored. Footnotes admit it "might be better", but they still endorse A. I am not convinced they have priced everything properly.
On that note: footnotes blow holes in much of the plan, but the document just keeps going as if things stay on track. The authors are too honest to hide the problems, but too committed to abandon a sinking ship.
Overall: reality does not grade on a curve. If this is the best plan, we're still toast.
«The right to privacy is super important, including both privacy from companies and governments, and also privacy from AIs and individuals. This includes retroactive privacy. You need the freedom that comes from knowing you can decide later to keep something to yourself,»
Great idea! Too bad it is way too late for us employees:
https://www.wired.com/1999/01/sun-on-privacy-get-over-it/
Scott McNealy, 1999: “You have zero privacy anyway. Get over it!”
Some privacy can be obtained only by living in your own property surrounded by walls, using only transport owned by yourself (including airplanes, cars, trains owned via a series of corporate shells) and only acting through corporates shells and nominees/proxies. That is the lifestyle of many oligarchs, way out of reach for employees, and is not easy even for oligarchs:
https://twitter.com/nntaleb/status/1186271838386774018
NN Taleb, 2020: “every financial transaction done on Planet Earth since 2005 is traceable (the aftermath of Sep 11). Even if one uses prête-noms. Even if one uses cryptocurrencies, art work, etc. You cannot hide anything anymore.”