I feel just slightly guilty for joking with Depseek about a feature, it gives me pseudocode that is quite clearly meant as and understood as a joke, and then I'm like "Please can you actually implement that." Sorry about the mood whiplash, Deepseek, and thanks for writing that code.
I'm grateful for your persistent, polite, and effective fisking of Tyler Cowen's pronouncements. Like thousands of us, I once found his output a daily delight. Over time, I noticed that he was occasionally way off base (hard to avoid when you make 10-50 assertions a day), but that his calm self-assurance, prodigious intelligence, and lofty reputation made his errors hard to refute. Then I noticed that the small subset of his work touching on politics (climate, health care, and his NYT and Bloomberg columns generally) was often ... well, the polite word would be "Straussian". For a while I sputtered less polite words, but eventually gave up and just stopped reading him. It's fantastic that you are holding him to account on topics where your depth of knowledge and ability to engages matches his. I hope you and others will consider the possibility that the significant flaws in his AI thinking have parallels in his other magisterial pronouncements on matters of great importance.
My mental model of deep research (which, tbh I have yet to try) is that yes, it is impressive and on par with what serious graduate level researchers or professionals could produce.
Yes it could be disruptive in a way.
However my hunch would be that it doesn't yield that much actual progress, simply that it helps us realise a significant chunk of white collar "research" is close to useless and quite redundant.
I'm not too surprised Tyler Cowen likes it either.
Ironically, the Google AI that can navigate phone trees and request to talk to a human might result in a rare case of AI *creating* jobs. Customer service lines might need to hire more humans to keep up with the demand.
Update the rest of the way. Obviously then the companies replace the first line CSR reps with AI.
So total unemployment? Not necessarily. This is Jevons paradox - each level 2 rep is about 5-10 times as productive. And customers can complain without spending any of their own time.
So both sides make way more complaints and process more complaints. Also customers sue way more often for a similar reason since attorneys are cheaper.
I don't know the equilibrium but one reasonable prediction is human employment stays full and theres a lot more complaints and lawsuits.
For my fellow young people who are readers of this newsletter - what the heck are we supposed to do in our careers? So many professions require some form of apprenticeship, except by the end of it the AGI will be able to do whatever we were apprenticing for. Do we just try to grab a high paying job and ride it out? This feels deeply cynical; I want to be optimistic and contribute to the policy work/research to help out, but by the time I will have learned enough to actually contribute to a field it will be too late
Either you are brilliant enough to make a meaningful contribution to AI safety by becoming an AI researcher or you go into a physical trade and hope for the best. Plumbers will be around way longer than most intellectual professions.
"DeepSeek’s web version appears to send your login information..."
Not exactly! From the linked article, it "has computer code that could" do that. "Neither Feroot nor the other researchers observed data transferred to China Mobile when testing logins in North America, but they could not rule out that data for some users was being transferred to the Chinese telecom."
This is super helpful! I often experience confusion when you quote a take in order to disagree with it. I read the quote normally, but then I read your comment *after* the quote and I realize I need to go back and reverse the valence in my mind. And occasionally, I can't completely parse your take, so I'm not sure whether you agree or disagree. This up-front labelling fixes this problem completely!
I have just had a rather strange experience with Deepseek R1. I asked it to draw bitmaps for characters. Now, this task is like counting the R's in strawberry, only worse, because the LLM sees tokens, not characters, and certainly not bitmaps.
The chain of thought looks like it's failing miserably ... and then it gets the right answer at the end.
This general phenomenon of garbage chain of thought but correct answer deserves some investigation, I feel.
> Look at what is happening in coding, the first major profession to have serious AI diffusion... There is essentially no pushback.... We will see pushback, but I mostly don’t see any stopping this train for most cognitive work.
Software (at least in the USA) has no professional organizations or unions and is essentially unregulated. I would guess things like doctors, lawyers, accountants, and real-world engineers will be harder to dislodge due to all sorts of legal regulations. There are many examples of high paying jobs in the US that could have been automated decades ago but have not been due to unions, for example dock workers or train drivers. It's hard for me to envision how these legally locked up professions could fall other than one country essentially turns everything over to AI, does massively better than all the other countries, and then everyone else either has to copy or falls massively behind.
There hasn’t been the expected pushback from lawyers bc the state bar associations are dominated by the influence of BigLaw, which expects to gain market share and dump associates. So BigLaw created a buzz phrase “access to justice” to frame AI as enabling poor people better access to the legal system. Their ONE job is to protect our cabal, and they can’t even do that right.
I find Noah’s statement “sorry the programming career didn’t work out, have you considered working in a care home” to be the sort of thing that could really hit home for professionals. Not only would you lose your career, you’d be brought low and have to do the kind of labor that people with GEDs do today. We all need to fear that future, years of expertise rendered meaningless overnight as we’re forced into the “caring professions” as all that remains. (Or more likely, driven to “deaths of despair.”) To the conservative anti-futurist mantra of “I will not eat the bugs, I will not live in the pod” they ought to add “I will not wipe the old man’s ass”.
I've trusted semianalysis in the past, but their reporting on Deepseek has been unfortunately riddled with errors. Here is a pretty decisive rebuttal of their arguments:
Please do read it, because it's quite damning. Straight up making up numbers, not adding numbers together correctly, and generally not understanding how hedge funds work.
One comment about firms with drop-in AGI replacements for humans, re:
"AIs can be copied on demand. So can entire teams and systems. There would be no talent or training bottlenecks."
Eventually, sure, but about that training: My current (mis?)understanding is that _incremental_ learning, actually updating the neural weights, is still an open research problem. My current (mis?)understanding is that if one tried the equivalent strategy, right this minute, one could copy the information in the context window (and maybe the current neural net activations?), but the weights would still be the frozen ones.
Looking forward to this problem being solved (if it is indeed still open).
The "difference in FLOP" question on the AGI Readiness Index makes no sense, it should ask about the ratio (or ask about the difference in log(FLOP)). If x > y are exponentially distributed (as things measured in FLOPs tend to be) then x-y ~ x tells you almost nothing about the value of y relative to x (whereas x/y tells you everything). This doesn't explain the instability you noted, but it suggests the people who designed the index were not thinking clearly.
>Look at what is happening in coding, the first major profession to have serious AI diffusion because it is the place AI works best at current capability levels. There is essentially no pushback.
My model for this: Currently 99%+ of the benefits of AI are being captured by the workers themselves. It is like accounting at the advent of VisiCalc - we charge the same, but the work we do takes so much less time. The people who pay us haven't figured out how to shift the equilibrium yet (too much uncertainty in this fast moving landscape) and the forces of competition are not super high. This is why the mass layoffs in Silicon Valley weren't nearly as disruptive as you may have previously expected. AI products currently are *heavily* consumer focused, empowering individual users and enhancing their performance, rather than B2B focused. Agents will likely disrupt this, but for now, the workers are taking the benefits while they can and while the boss isn't looking too closely.
The AI I'm seeing in the B2B space (HighRadius is a basic example relevant to my financial operations experience) is all extremely out of date or of a basic "variance analysis/outlier detection" application. About 6 months ago I was of half a mind to pitch VC for using ChatGPT as a backend for invoice processing/document flow management, and the example of anlysing historical documents only reinforces to me the efficacy of that. But, I'm comfortably middle class and don't feel the *need* to drive such things as I assume a successful entrepreneur does. But hey, if anyone wants to discuss with me and start a partnership of some kind, sure why not.
Maybe just me but the links aren't linking out... Just back to this post
Update - just in the substack (android) app. The web version is working
Yep, I get this every so often, I don't know why they won't fix it.
"You don’t like the UI, build a new one."
Or ask the AI to build a new one for you...
======
I feel just slightly guilty for joking with Depseek about a feature, it gives me pseudocode that is quite clearly meant as and understood as a joke, and then I'm like "Please can you actually implement that." Sorry about the mood whiplash, Deepseek, and thanks for writing that code.
I'm grateful for your persistent, polite, and effective fisking of Tyler Cowen's pronouncements. Like thousands of us, I once found his output a daily delight. Over time, I noticed that he was occasionally way off base (hard to avoid when you make 10-50 assertions a day), but that his calm self-assurance, prodigious intelligence, and lofty reputation made his errors hard to refute. Then I noticed that the small subset of his work touching on politics (climate, health care, and his NYT and Bloomberg columns generally) was often ... well, the polite word would be "Straussian". For a while I sputtered less polite words, but eventually gave up and just stopped reading him. It's fantastic that you are holding him to account on topics where your depth of knowledge and ability to engages matches his. I hope you and others will consider the possibility that the significant flaws in his AI thinking have parallels in his other magisterial pronouncements on matters of great importance.
I thought the daily posts were going to lead to *shorter* megaposts 😅.
Podcast episode fot his post:
https://open.substack.com/pub/dwatvpodcast/p/ai-102-made-in-america
My mental model of deep research (which, tbh I have yet to try) is that yes, it is impressive and on par with what serious graduate level researchers or professionals could produce.
Yes it could be disruptive in a way.
However my hunch would be that it doesn't yield that much actual progress, simply that it helps us realise a significant chunk of white collar "research" is close to useless and quite redundant.
I'm not too surprised Tyler Cowen likes it either.
Ironically, the Google AI that can navigate phone trees and request to talk to a human might result in a rare case of AI *creating* jobs. Customer service lines might need to hire more humans to keep up with the demand.
Update the rest of the way. Obviously then the companies replace the first line CSR reps with AI.
So total unemployment? Not necessarily. This is Jevons paradox - each level 2 rep is about 5-10 times as productive. And customers can complain without spending any of their own time.
So both sides make way more complaints and process more complaints. Also customers sue way more often for a similar reason since attorneys are cheaper.
I don't know the equilibrium but one reasonable prediction is human employment stays full and theres a lot more complaints and lawsuits.
For my fellow young people who are readers of this newsletter - what the heck are we supposed to do in our careers? So many professions require some form of apprenticeship, except by the end of it the AGI will be able to do whatever we were apprenticing for. Do we just try to grab a high paying job and ride it out? This feels deeply cynical; I want to be optimistic and contribute to the policy work/research to help out, but by the time I will have learned enough to actually contribute to a field it will be too late
Either you are brilliant enough to make a meaningful contribution to AI safety by becoming an AI researcher or you go into a physical trade and hope for the best. Plumbers will be around way longer than most intellectual professions.
"DeepSeek’s web version appears to send your login information..."
Not exactly! From the linked article, it "has computer code that could" do that. "Neither Feroot nor the other researchers observed data transferred to China Mobile when testing logins in North America, but they could not rule out that data for some users was being transferred to the Chinese telecom."
> Zeynep Tufekci (being wrong)
This is super helpful! I often experience confusion when you quote a take in order to disagree with it. I read the quote normally, but then I read your comment *after* the quote and I realize I need to go back and reverse the valence in my mind. And occasionally, I can't completely parse your take, so I'm not sure whether you agree or disagree. This up-front labelling fixes this problem completely!
Good note! This is was a case where I actively worried someone might get the wrong impression. But perhaps I should be doing this more widely.
I have just had a rather strange experience with Deepseek R1. I asked it to draw bitmaps for characters. Now, this task is like counting the R's in strawberry, only worse, because the LLM sees tokens, not characters, and certainly not bitmaps.
The chain of thought looks like it's failing miserably ... and then it gets the right answer at the end.
This general phenomenon of garbage chain of thought but correct answer deserves some investigation, I feel.
> Look at what is happening in coding, the first major profession to have serious AI diffusion... There is essentially no pushback.... We will see pushback, but I mostly don’t see any stopping this train for most cognitive work.
Software (at least in the USA) has no professional organizations or unions and is essentially unregulated. I would guess things like doctors, lawyers, accountants, and real-world engineers will be harder to dislodge due to all sorts of legal regulations. There are many examples of high paying jobs in the US that could have been automated decades ago but have not been due to unions, for example dock workers or train drivers. It's hard for me to envision how these legally locked up professions could fall other than one country essentially turns everything over to AI, does massively better than all the other countries, and then everyone else either has to copy or falls massively behind.
There hasn’t been the expected pushback from lawyers bc the state bar associations are dominated by the influence of BigLaw, which expects to gain market share and dump associates. So BigLaw created a buzz phrase “access to justice” to frame AI as enabling poor people better access to the legal system. Their ONE job is to protect our cabal, and they can’t even do that right.
I find Noah’s statement “sorry the programming career didn’t work out, have you considered working in a care home” to be the sort of thing that could really hit home for professionals. Not only would you lose your career, you’d be brought low and have to do the kind of labor that people with GEDs do today. We all need to fear that future, years of expertise rendered meaningless overnight as we’re forced into the “caring professions” as all that remains. (Or more likely, driven to “deaths of despair.”) To the conservative anti-futurist mantra of “I will not eat the bugs, I will not live in the pod” they ought to add “I will not wipe the old man’s ass”.
I've trusted semianalysis in the past, but their reporting on Deepseek has been unfortunately riddled with errors. Here is a pretty decisive rebuttal of their arguments:
https://threadreaderapp.com/thread/1885306497346752672.html
Please do read it, because it's quite damning. Straight up making up numbers, not adding numbers together correctly, and generally not understanding how hedge funds work.
Many Thanks, great article!
One comment about firms with drop-in AGI replacements for humans, re:
"AIs can be copied on demand. So can entire teams and systems. There would be no talent or training bottlenecks."
Eventually, sure, but about that training: My current (mis?)understanding is that _incremental_ learning, actually updating the neural weights, is still an open research problem. My current (mis?)understanding is that if one tried the equivalent strategy, right this minute, one could copy the information in the context window (and maybe the current neural net activations?), but the weights would still be the frozen ones.
Looking forward to this problem being solved (if it is indeed still open).
The "difference in FLOP" question on the AGI Readiness Index makes no sense, it should ask about the ratio (or ask about the difference in log(FLOP)). If x > y are exponentially distributed (as things measured in FLOPs tend to be) then x-y ~ x tells you almost nothing about the value of y relative to x (whereas x/y tells you everything). This doesn't explain the instability you noted, but it suggests the people who designed the index were not thinking clearly.
>Look at what is happening in coding, the first major profession to have serious AI diffusion because it is the place AI works best at current capability levels. There is essentially no pushback.
My model for this: Currently 99%+ of the benefits of AI are being captured by the workers themselves. It is like accounting at the advent of VisiCalc - we charge the same, but the work we do takes so much less time. The people who pay us haven't figured out how to shift the equilibrium yet (too much uncertainty in this fast moving landscape) and the forces of competition are not super high. This is why the mass layoffs in Silicon Valley weren't nearly as disruptive as you may have previously expected. AI products currently are *heavily* consumer focused, empowering individual users and enhancing their performance, rather than B2B focused. Agents will likely disrupt this, but for now, the workers are taking the benefits while they can and while the boss isn't looking too closely.
The AI I'm seeing in the B2B space (HighRadius is a basic example relevant to my financial operations experience) is all extremely out of date or of a basic "variance analysis/outlier detection" application. About 6 months ago I was of half a mind to pitch VC for using ChatGPT as a backend for invoice processing/document flow management, and the example of anlysing historical documents only reinforces to me the efficacy of that. But, I'm comfortably middle class and don't feel the *need* to drive such things as I assume a successful entrepreneur does. But hey, if anyone wants to discuss with me and start a partnership of some kind, sure why not.