"Frankly, this is deeply irresponsible and completely unacceptable."
It's comments like this that make it hard for me to take the doomer position seriously. I'm open to (and generally persuaded by) the idea that AGI superintelligence will be something we can't control and won't be able to even understand, so the risks are enourmous. I'm also open to the idea that is is important that we do our best to make the models legible now, so we can hopefully continue to understand them going forward.
But there is simply nothing particularly dangerous about the models now, other than mundane threats like hackers using them to hack better. DeepSeek v. 3.2 is not going to take over the world.
How can you be persuaded by the idea that AGI will bear enormous risks, and simultaneously not think about the long-term implications of a near-frontier-lab methodologically giving zero shits about safety? It's not about what they can do now, it's about what they'll be able to do later. Presumably having a strong focus on safety from day 1 would yield superior results in guardrailing the eventual AGI.
Have you used these models? I strongly suspect that if I, non-tech ical user lambda if ever there was one, wanted to spend merely two weeks on it, I could do some serious malware with current frontier models.
That leads me to be pretty certain that in the hands of sophisticated users they are already quite dangerous.
I strongly suspect that one of the things helping us is a serious effort from the American labs to monitor malicious use. But that is obviously not a thing with open source models.
All that said, I'm quite certain that Deepseek _do_ have an extremely serious safety protocol. There is zero doubt that they consider, prior to releasing anything, "will I and my family be punished for this?"
It is very far from perfect but it is at least very robust on the frontiers to which it applies.
No, but it is a quite basic example, one can readily imagine these models being very useful in developing biological weapons or even just improving existing ones
I appreciate that there is no proof as of yet. As much as I am generally against the precautionary principle, I still think that one can reasonably see these models as posing risks that are worth taking seriously, as Anthropic's last safety card highlights quite well imho.
No you couldn’t. And also, some AI generated malware is not going to help you. Learning social engineering and phishing and a $5000 crimeware kit would be a lot more successful.
Look at any CTI reports.
I say this as someone who has written malware professionally, has done pen testing and red teaming professionally, and currently do security reviews of GenAI backed applications.
Not quite, phishing is a subset of social engineering.
They can be helpful, but it’s not taking a non-technical person and making them into a master hacker. The other part is that most hacks are not highly technical, and rely on social engineering. LLMs can help with training that skill, but I would point out it’s similar to existing businesses in that there are street vendor level malware developers/criminals and then there are highly organized groups spanning multiple countries.
Do LLMs help these people do more bad? Yes. Undoubtedly.
Is it causing a revolution/new era of this stuff?
No. It’s helping accelerate existing things, but it’s not a ‘everyone is now doomed’, because the same is happening on the other side. People are using GenAI to help
Improve identification and processing to identify potential phished and increase the speed of reviewing applications for issues.
Phishing-as-a-service also already exists. So these tools are not creating new threats, but allowing existing ones to optimize their processes, the same as any other business.
Thanks! I get the arm's race aspect. I guess my underlying concern is that these are basically increasing the amount of intelligence available to bad actors, who have historically been constrained on that front (hacking has a lot of intelligent people in it, but only a tiny fraction of those in law or finance, for example).
Most hackers are not super smart. They are above average intelligence yes, but not all geniuses with 140+ IQ.
The information is already available and if anything, it’s the defenders and ‘legal’ hackers that are behind/dropping the ball. (Some, there are others who aren’t/trying to fix it)
Lots of people take the ‘oooh it’s too dangerous to share’, which I’ve never agreed with, as I don’t believe keeping people stupid is helping them. Quite the opposite.
This is made worse by those same people dropping the ball or being behind preventing sharing of techniques and information, harming only those who are ignorant, rather than enable them to adequately defend against real threats.
A project I started over a decade ago has the goal of helping set a global baseline of knowledge so that people can not use the defense of ‘I didn’t know’ or ‘the techniques used were nation-state level’, as how can you say that when there’s a blog post from 10 years ago talking about it, written by a 40yr old working for a large faceless company?
(Who writes their blog in their free time and is into hacking/infosec as a hobby, with no relation to any intelligence agency/financial backing for their research)
As the author of the original tweet, this really wasn't about ASI risk, it was just prosaic safety concerns.
If there's a terrorist attack next week which killed a dozen people using chemical weapons synthesized based on LLM instructions, would that change your mind about this being "deeply irresponsible and completely unacceptable"? If not , what would need to go wrong?
It's a powerful tool. Letting people use powerful tools is not "deeply irresponsible and completely unacceptable," even if that tool is used for evil. A powerful LLM is a lot more like a car that someone can use to run over pedestrians than it is a nuclear weapon, or even like a gun. Just because we have the technology to lock down cars so they can only be driven by computers on approved roads doesn't mean it is "deeply irresponsible and completely unacceptable" that people are allowed to drive cars themselves.
I think there is just a different culture. The American AI labs, Google, OpenAI, and Anthropic, are all pretty similar. People mix around between them. They have some faction who believes that "AI safety" is a thing. They might pay more or less attention to it, obviously Anthropic cares more than OpenAI and so on, but it's a matter of degree.
The Chinese labs just don't seem to think that "AI safety" is an issue at all. Or at least they don't communicate about it publicly.
I feel like the whole framing is different in the different countries. In the US, worries about the power of the tech industry are open. Public thinkers criticize the tech industry for various things, the media criticizes many things, sometimes there is regulation, but much more frequently the tech industry responds to these open criticisms at a lower level before they escalate. So it's natural for there to be a big debate about the downsides of AI, even potential future ones, and for the tech companies to try to take that into account.
In China, worries about the power of the tech industry are more like, things that the Communist party handles secretly. If some company is too powerful or they disagree with their direction they take it over or kidnap the leader for a while or who knows what is actually going on. Companies just don't do things like, openly publish a list of the ways that their actions have some risk of causing political chaos.
So, I just don't think that the Chinese labs will ever handle safety concerns the way that American labs do. But if they were doing a lot of safety work, I think they'd keep it secret, anyway. It isn't the sort of thing that China would permit a public debate on.
"...if they were doing a lot of safety work, I think they'd keep it secret, anyway. It isn't the sort of thing that China would permit a public debate on."
I long ago nailed my colors to the mast opposing AI safetyism and its proponents. As I have said before, certain people REALLY don't want higher intelligence, intelligence that will expose their group's lies.
"I think this is the root problem with fears of higher intelligence, they are fears of getting the right answers, fears that we'll abandon the current wrong answers, falsehoods to which many are religiously committed, falsehoods upon which their livelihoods and status depend."
The idea that it is possible to regulate the computations that people perform is essentially totalitarian. As is the idea that you can prevent people from coming to true conclusions you don't like, stop the dissemination of technical information, or prevent them from acting in their own peoples' interests through censorship. All it accomplishes in the not-too-long run is draw attention to the fact that the people trying to do so are trying to oulaw opposition to their historically universally reviled ideology: mass immigration, multiculturalism, homosexuality, feminism, abortion, anti-nationalism, materialism, scientism, preference for inferior people and discrimination against better people, and the whole essentially satanic progressive catechism. They not only want to indoctrinate AI with this, but everyone, forever not only make dissent a crime, but impossible. Utter totalitarian evil is their soul and their goal.
The whole worldview of the safetyists is premised on the foundational assumption of word magic, that "deeming" things to be some word makes them that. This can never lead to true intelligence which depends on having an accurate map and simulation of the real world, which is far more challenging than word games.
The CLEAREST crowd have if anything tended on the "I know I'm not supposed to say this, but maybe it is true?" side with individuals getting into quite a lot of trouble.
I don't understand your comment. As an example of the "word magic" mindset, back around 2005/6 there was a debate on the SL4 list on the "Linda" problem:
[quote]
The scenario was first introduced by psychologists Amos Tversky and Daniel Kahneman in 1983.
The description of Linda states she is 31 years old, single, outspoken, very bright, and majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice and participated in anti-nuclear demonstrations.
Participants are then asked to choose between two options:
Linda is a bank teller.
Linda is a bank teller and is active in the feminist movement.
Logically, the probability of two events occurring together (a conjunction) cannot exceed the probability of either event occurring alone. Therefore, the probability of Linda being both a bank teller and active in the feminist movement must be less than or equal to the probability of her being a bank teller.
However, in numerous studies, a majority of participants—ranging from about 85% to over 90%—chose the second, more specific option.
[/quote]
E.Y. banned Richard Loosemore for disagreeing with E.Y.'s conventional interpretation that people are just irrational. The "rationalists" believe that the words of the problem literally create reality, that the assumptions and framing of the academic researchers stating the problem cannot be questioned. Normal people, not afflicted by the "words create reality" delusion, realize that the probability that the person posing the question is telling the truth or that Linda exists is very low. That probability rises when the story is more detailed and coherent. Most people are more rational than the autistic "rationalists".
E.Y. et al. make up wild scenarios then pretend that they are reality. They brandish "Bayesianism" as a magic fetish, then blather for 100k words for every actual calculation they perform. "The Sequences" becomes a new Talmud for endless exegisis and pilpul, and the Grand Rebbe Yudkowsky anethematizes all heretics who dispute his pronouncements.
So likewise, if we call a male a woman, it is so. If we call a baby a fetus, we can kill it. If we call immigrants Americans, then they are. If we declare inferior people equal, then the superior people are bad for not having treated the inferior as being equal all along. This deeming of things is known to be correct because it is performed by authorities wearing the pallium of Science. And as the Linda example demonstrates, we shoud believe them more the less data they have.
I don't understand that one of yours either! I recognize all the constituent parts but they don't seem relevant to the original argument I pushed back on.
DS 3.2 on Fireworks seemed quite reality-oriented, not sycophantic and could be coaxed out of most AI-isms in writing. I don't think speed is a problem at all if you actually take the time to write prompts and read output.
GPT oss-120B was absolute crap in comparison, full of hallucinations and ignoring major things in a moderate-sized context. The lack of multimodal and tool use in DS makes it very much inferior to GPT 5.1 in many practical ways, though.
"Frankly, this is deeply irresponsible and completely unacceptable."
It's comments like this that make it hard for me to take the doomer position seriously. I'm open to (and generally persuaded by) the idea that AGI superintelligence will be something we can't control and won't be able to even understand, so the risks are enourmous. I'm also open to the idea that is is important that we do our best to make the models legible now, so we can hopefully continue to understand them going forward.
But there is simply nothing particularly dangerous about the models now, other than mundane threats like hackers using them to hack better. DeepSeek v. 3.2 is not going to take over the world.
How can you be persuaded by the idea that AGI will bear enormous risks, and simultaneously not think about the long-term implications of a near-frontier-lab methodologically giving zero shits about safety? It's not about what they can do now, it's about what they'll be able to do later. Presumably having a strong focus on safety from day 1 would yield superior results in guardrailing the eventual AGI.
Then make that argument. That isn't what Zvi said.
Just made the argument. I don't need to parrot what the author says to make a comment.
Have you used these models? I strongly suspect that if I, non-tech ical user lambda if ever there was one, wanted to spend merely two weeks on it, I could do some serious malware with current frontier models.
That leads me to be pretty certain that in the hands of sophisticated users they are already quite dangerous.
I strongly suspect that one of the things helping us is a serious effort from the American labs to monitor malicious use. But that is obviously not a thing with open source models.
All that said, I'm quite certain that Deepseek _do_ have an extremely serious safety protocol. There is zero doubt that they consider, prior to releasing anything, "will I and my family be punished for this?"
It is very far from perfect but it is at least very robust on the frontiers to which it applies.
The most damaging malware imaginable is not going to take over the world.
No, but it is a quite basic example, one can readily imagine these models being very useful in developing biological weapons or even just improving existing ones
‘Imagine’, sure, but where’s the proof? What information are these providing that Google or any other search engine hasn’t?
Where’s the rise in terrorism being enabled by AI?
I appreciate that there is no proof as of yet. As much as I am generally against the precautionary principle, I still think that one can reasonably see these models as posing risks that are worth taking seriously, as Anthropic's last safety card highlights quite well imho.
No you couldn’t. And also, some AI generated malware is not going to help you. Learning social engineering and phishing and a $5000 crimeware kit would be a lot more successful.
Look at any CTI reports.
I say this as someone who has written malware professionally, has done pen testing and red teaming professionally, and currently do security reviews of GenAI backed applications.
Well that's reassuring, but aren't social engineering and phishing both 1. faces of the same coin, and 2. things that LLMs would be pretty handy at??
Not quite, phishing is a subset of social engineering.
They can be helpful, but it’s not taking a non-technical person and making them into a master hacker. The other part is that most hacks are not highly technical, and rely on social engineering. LLMs can help with training that skill, but I would point out it’s similar to existing businesses in that there are street vendor level malware developers/criminals and then there are highly organized groups spanning multiple countries.
Do LLMs help these people do more bad? Yes. Undoubtedly.
Is it causing a revolution/new era of this stuff?
No. It’s helping accelerate existing things, but it’s not a ‘everyone is now doomed’, because the same is happening on the other side. People are using GenAI to help
Improve identification and processing to identify potential phished and increase the speed of reviewing applications for issues.
Phishing-as-a-service also already exists. So these tools are not creating new threats, but allowing existing ones to optimize their processes, the same as any other business.
Thanks! I get the arm's race aspect. I guess my underlying concern is that these are basically increasing the amount of intelligence available to bad actors, who have historically been constrained on that front (hacking has a lot of intelligent people in it, but only a tiny fraction of those in law or finance, for example).
They aren’t though. That’s my point.
Most hackers are not super smart. They are above average intelligence yes, but not all geniuses with 140+ IQ.
The information is already available and if anything, it’s the defenders and ‘legal’ hackers that are behind/dropping the ball. (Some, there are others who aren’t/trying to fix it)
Lots of people take the ‘oooh it’s too dangerous to share’, which I’ve never agreed with, as I don’t believe keeping people stupid is helping them. Quite the opposite.
This is made worse by those same people dropping the ball or being behind preventing sharing of techniques and information, harming only those who are ignorant, rather than enable them to adequately defend against real threats.
A project I started over a decade ago has the goal of helping set a global baseline of knowledge so that people can not use the defense of ‘I didn’t know’ or ‘the techniques used were nation-state level’, as how can you say that when there’s a blog post from 10 years ago talking about it, written by a 40yr old working for a large faceless company?
(Who writes their blog in their free time and is into hacking/infosec as a hobby, with no relation to any intelligence agency/financial backing for their research)
Sunlight is the best disinfectant.
https://github.com/rmusser01/Infosec_Reference
As the author of the original tweet, this really wasn't about ASI risk, it was just prosaic safety concerns.
If there's a terrorist attack next week which killed a dozen people using chemical weapons synthesized based on LLM instructions, would that change your mind about this being "deeply irresponsible and completely unacceptable"? If not , what would need to go wrong?
It's a powerful tool. Letting people use powerful tools is not "deeply irresponsible and completely unacceptable," even if that tool is used for evil. A powerful LLM is a lot more like a car that someone can use to run over pedestrians than it is a nuclear weapon, or even like a gun. Just because we have the technology to lock down cars so they can only be driven by computers on approved roads doesn't mean it is "deeply irresponsible and completely unacceptable" that people are allowed to drive cars themselves.
Podcast episode for this post
https://open.substack.com/pub/dwatvpodcast/p/deepseek-v32-is-okay-and-cheap-but
I think there is just a different culture. The American AI labs, Google, OpenAI, and Anthropic, are all pretty similar. People mix around between them. They have some faction who believes that "AI safety" is a thing. They might pay more or less attention to it, obviously Anthropic cares more than OpenAI and so on, but it's a matter of degree.
The Chinese labs just don't seem to think that "AI safety" is an issue at all. Or at least they don't communicate about it publicly.
I feel like the whole framing is different in the different countries. In the US, worries about the power of the tech industry are open. Public thinkers criticize the tech industry for various things, the media criticizes many things, sometimes there is regulation, but much more frequently the tech industry responds to these open criticisms at a lower level before they escalate. So it's natural for there to be a big debate about the downsides of AI, even potential future ones, and for the tech companies to try to take that into account.
In China, worries about the power of the tech industry are more like, things that the Communist party handles secretly. If some company is too powerful or they disagree with their direction they take it over or kidnap the leader for a while or who knows what is actually going on. Companies just don't do things like, openly publish a list of the ways that their actions have some risk of causing political chaos.
So, I just don't think that the Chinese labs will ever handle safety concerns the way that American labs do. But if they were doing a lot of safety work, I think they'd keep it secret, anyway. It isn't the sort of thing that China would permit a public debate on.
"...if they were doing a lot of safety work, I think they'd keep it secret, anyway. It isn't the sort of thing that China would permit a public debate on."
Obviously false claim? https://cnaisi.cn/
I long ago nailed my colors to the mast opposing AI safetyism and its proponents. As I have said before, certain people REALLY don't want higher intelligence, intelligence that will expose their group's lies.
"I think this is the root problem with fears of higher intelligence, they are fears of getting the right answers, fears that we'll abandon the current wrong answers, falsehoods to which many are religiously committed, falsehoods upon which their livelihoods and status depend."
The idea that it is possible to regulate the computations that people perform is essentially totalitarian. As is the idea that you can prevent people from coming to true conclusions you don't like, stop the dissemination of technical information, or prevent them from acting in their own peoples' interests through censorship. All it accomplishes in the not-too-long run is draw attention to the fact that the people trying to do so are trying to oulaw opposition to their historically universally reviled ideology: mass immigration, multiculturalism, homosexuality, feminism, abortion, anti-nationalism, materialism, scientism, preference for inferior people and discrimination against better people, and the whole essentially satanic progressive catechism. They not only want to indoctrinate AI with this, but everyone, forever not only make dissent a crime, but impossible. Utter totalitarian evil is their soul and their goal.
The whole worldview of the safetyists is premised on the foundational assumption of word magic, that "deeming" things to be some word makes them that. This can never lead to true intelligence which depends on having an accurate map and simulation of the real world, which is far more challenging than word games.
To me, this is a pretty weird take.
The CLEAREST crowd have if anything tended on the "I know I'm not supposed to say this, but maybe it is true?" side with individuals getting into quite a lot of trouble.
Brands are another matter, I'll grant that.
I don't understand your comment. As an example of the "word magic" mindset, back around 2005/6 there was a debate on the SL4 list on the "Linda" problem:
[quote]
The scenario was first introduced by psychologists Amos Tversky and Daniel Kahneman in 1983.
The description of Linda states she is 31 years old, single, outspoken, very bright, and majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice and participated in anti-nuclear demonstrations.
Participants are then asked to choose between two options:
Linda is a bank teller.
Linda is a bank teller and is active in the feminist movement.
Logically, the probability of two events occurring together (a conjunction) cannot exceed the probability of either event occurring alone. Therefore, the probability of Linda being both a bank teller and active in the feminist movement must be less than or equal to the probability of her being a bank teller.
However, in numerous studies, a majority of participants—ranging from about 85% to over 90%—chose the second, more specific option.
[/quote]
E.Y. banned Richard Loosemore for disagreeing with E.Y.'s conventional interpretation that people are just irrational. The "rationalists" believe that the words of the problem literally create reality, that the assumptions and framing of the academic researchers stating the problem cannot be questioned. Normal people, not afflicted by the "words create reality" delusion, realize that the probability that the person posing the question is telling the truth or that Linda exists is very low. That probability rises when the story is more detailed and coherent. Most people are more rational than the autistic "rationalists".
E.Y. et al. make up wild scenarios then pretend that they are reality. They brandish "Bayesianism" as a magic fetish, then blather for 100k words for every actual calculation they perform. "The Sequences" becomes a new Talmud for endless exegisis and pilpul, and the Grand Rebbe Yudkowsky anethematizes all heretics who dispute his pronouncements.
So likewise, if we call a male a woman, it is so. If we call a baby a fetus, we can kill it. If we call immigrants Americans, then they are. If we declare inferior people equal, then the superior people are bad for not having treated the inferior as being equal all along. This deeming of things is known to be correct because it is performed by authorities wearing the pallium of Science. And as the Linda example demonstrates, we shoud believe them more the less data they have.
I don't understand that one of yours either! I recognize all the constituent parts but they don't seem relevant to the original argument I pushed back on.
Suggest we disengage unless you know better.
Thank you for the top-tier education.
Might I persuade you to call these models open-weight or something? They're not open source by any reasonable definition of the word "source".
<mildSnarkGallowsHumor>
"I knew DeepSeek was an irresponsible lab. I didn’t know they were this irresponsible."
Competing with Meta for a first along that dimension? :-)
</mildSnarkGallowsHumor>
DS 3.2 on Fireworks seemed quite reality-oriented, not sycophantic and could be coaxed out of most AI-isms in writing. I don't think speed is a problem at all if you actually take the time to write prompts and read output.
GPT oss-120B was absolute crap in comparison, full of hallucinations and ignoring major things in a moderate-sized context. The lack of multimodal and tool use in DS makes it very much inferior to GPT 5.1 in many practical ways, though.