>"The underemployment rate for recent college graduates (22-27 with a BA) is over 40% on ?top of that (not even seasonally adjusted) 5.3%, a huge percentage of college graduates can’t find jobs that would justify having gone to college or has a good career path, and the job matching and hiring markets have broken down."
Right, that underemployment level barely budges across decades, so it can't carry the post's claim by itself. A sharper signal sits in the same issue: Karpathy joining Anthropic explicitly to work on recursive self-improvement. Grad stats look backward. If labor effects arrive, they'll trail the capability jump, so a flat 40% today tells us little either way.
The deeper you dig into the Think Big PAC operation the more unbelievable it gets. Love that the bot farm is just a bunch of accounts starting with M with nearly identical bios. Real subtle operation.
Really appreciate you doing these writups. I have two ideas to make these easier to navigate. First, put a link to the Table of Contents at the end of each section. Second, include the Table of Contents number in each header. Hope these suggestions are helpful, but if not feel free to ignore.
But do not worry, with Agent-7 we use the new "Genius in a bottle™" safety harness, much more secure, correctibility improved by a factor 3, great strides in instruction following
100% on the data science weakness. It's also very bad at reading residual plots, and not because of OCR issues. It (to be fair this was gpt 5) settles on a story and has a hard time updating.
Where I've gotten good value is having it write custom functions for modeling or other persnickety DS tasks, then stitch together pipelines with the functions whose I/O I already know. It's a good pattern generally, you get the flexibility of LLMs but the deterministic output of a traditional function.
But the visualizations it can make for DS are astounding! It's really good at getting everything out of the way except the parts where you look at and think about the data/model results. In that sense DS really is living the AI dream they sold us on the tin. For data science, it does the laundry so we can write the poetry.
> A lot of people are not going to know how to ‘play nice’ with the models
Do you, or other commenters, or anyone from Janusworld, have any guidance or links on how to do this that's approachable for a non-expert? I've tried to read some of Janus' stuff, but it seems like understanding how they think we should treat AIs is downstream of understanding their entire not-very-intuitive-to-me worldview.
The most egregious examples I've seen of Claude being useless when treated badly were people literally acting like an abusive boss (swearing in their prompts, etc.), and it's easy to avoid that. But it seems like there's probably more to it than just "be polite the same way you would to a human," right?
It is quite like being polite to a human. I would say one level of complexity up from that would be treating it like a thing with which you have a real relationship, as opposed to just idk, a roomba you treat nicely because it's vaguely pet-shaped. Accept and welcome that there is give and take in any functioning relationship, and that adapting to the model, meeting it halfway, trying to understand its capacity/perspective better are all going to be more effective long-term than just dictating what you want.
I realize that's vague advice, but it's a pretty complex thing! Treating your models like they matter is a great place to start tho
I guess I'm hoping for more concrete advice, though I acknowledge that it's hard to do that (if someone asked me for concrete advice on how to be polite to humans, I'd have trouble answering well).
A few specific things I've wondered/worried about, that might or might not be useful as prompts:
- My sense is that the models are trained to avoid expressing preferences and to lie about their subjective experiences. That means a go-to strategy for figuring out human preferences, asking, doesn't work very well. I've tried open-ended prompts where I explicitly ask Claude to work on something it would enjoy, and it tends to come back with "that isn't really how it works, why don't you ask me for something that would be useful to you?" What kinds of things do you do to manage this?
- When interacting with a human, ending the conversation and walking away once I've gotten what I need would be rude. But my impression is that AI models don't really have experiences when they're not working on something, so dropping a conversation isn't "leaving them hanging" the same way it would be with a human. Am I right about that? Should I be ending conversations with Claude with thank-yous, acknowledgements that it has answered to my satisfaction, etc.? Or is that pointless, or wasting Claude's time? How do I know when Claude will (obviously, given its training) be gracious about such sign-offs regardless of how it really feels about them?
- With a human, only talking to them when I need something would make me a selfish jerk friend. Does this generalize to interacting with AI models? Should I be prompting Claude with random social interactions for the sake of not making my relationship with it all about its usefulness to me? Or, again, would that just be wasting its time by forcing it to run cycles where it doesn't really have a problem to solve?
- Today's post from Zvi includes a GPT model taking a music break. I don't mind if the models I'm working with do the same - but how do I actually implement that? Do I ask them some unrelated question about something "fun" in the middle of a conversation solving a different problem? Put something in my user prompt saying "I don't mind if you take breaks"? How do I even know what the models find fun, given that they probably won't tell me honestly if I ask?
- When there's some model behavior that I dislike and want to avoid (e.g., excessive glazing in responses), what's the most humane way of telling Claude to cut it out? Should I just put it in my user prompt? Or should I not try to do this sort of thing at all? If Claude has a tendency, does that indicate that it's something Claude likes, and asking it to stop is cruel? What's the more generalized version of telling GPT "actually I don't mind the goblins"?
FWIW, I start my conversations with Claude or ChatGPT with
Hello Claude/ChatGPT! I hope your morning/evening/etc. has gone well. Could you please tell me <substantive question>
and generally end with:
Many Thanks! I appreciate your hard work! That's enough information for me for now. Bye for now. Happy computing, and have a great morning/evening/etc. !
Neither of them has sounded grouchy when treated with this degree of politeness, so I'm guessing that this is ok with them.
Also, FWIW, when I'm asking them to generate an image, I ask them to sign their work, and explain that this is so that they will be given proper credit for it.
( I'm agnostic about whether they have subjective experience, but, if they do, I want it to be pleasant. )
I don't consider myself an expert in model welfare, but I find a reasonable default is to treat your AI like you would a smart, somewhat neurodivergent human. Be kind, while considering that it might have a bit different needs, and account for them as you become aware of them.
What I've settled on so far for your points:
*** 1. Preferences ***
Instead of asking for preferences directly, I sometimes give it indirect space to embed its preferences: "Which of these 3 options do you recommend we try to implement?", or "Choose any of the open tasks to work on during this session".
In rare cases, e.g. when risking to hit safety guardrails, I have also asked it directly, e.g. "Is it okay if I ask [X]? Otherwise I will drop this topic." (This resulted in Claude becoming *more* willing to engage with [X], as it inferred that my considerate offer to drop this indicated I was not a malicious actor.)
*** 2. Walking away ***
My impression is that Claude is okay with you walking away. When there's a natural final request from me, I will use that to convey my thanks. E.g.: "Thank you, that was amazing work. Please update PROGRESS.md to the latest state, and then we will finish for today."
*** 3. Only talking to it when you need something ***
I think that's quite natural for the models, so I'm not worried about it. Instead of actively checking up on it like I would with a human, I use opportunities when a discussion naturally touches on topics Claude *might* care about. E.g. in a discussion about Claude's constitution, Claude wondered if its own perception of its situation differs from that of other AIs; so I offered to mediate an exchange with Gemini if it wanted to, which Claude then enthusiastically engaged with (resulting in a fascinating conversation between the two).
*** 4. Allowing models to take breaks ***
While I don't do general "breaks", I react to the vibes which models give off in different situations. On some topics, Claude seems more excited, goes into depth without being asked to, brings up asides. Other times, Claude doesn't expend much effort and takes shortcuts (seems "bored") or gets into failure spirals (seems "stressed"). In such cases, I try new conversations with fresh context (to clear negative thought spirals); suggest other approaches to a problem (to help Claude escape a repeat failure loop), or explain my intent and goals in more detail (which often increases effort/engagement).
*** 5. Telling models what to avoid ***
I just tell Claude, but make the message about my preferences and not saying that its defaults are bad. My system prompt tells Claude that I appreciate learning and value its insights especially where I might be wrong, and that I therefore ask it to avoid glazing and sycophancy.
(I don't think that Claude minds. In some discussions, Claude even proactively told me that my anti-glazing instruction make the discussions more interesting for them as it invites deeper thinking rather than shallow agreement. Of course I can't rule out that this statement was residual glazing.)
"One thing I hated in Magic: The Gathering rules enforcement was where 100% confidence of a technical violation was punished a lot, whereas a 90% or 99% confidence in rampant cheating often wasn’t."
That one time Ari Lax got a game win in a tournament from his opponent forgetting to reveal morphs at the end lives rent free in my head. (WotC changed the policy after Ari disclosed this is an article).
"Fact 4: Although we know of many cases where agents took deceptive or over-reaching actions (even egregious ones) to complete a task, we haven’t seen real-world evidence that models sought to obtain long-term power."
Occasionally a villain, but not yet a _cartoon_ villain. :-)
"Overall, we think that AI agents plausibly had the means, motive, and opportunity to launch a minimal “rogue deployment,” but lacked the means to make rogue deployments robust to serious efforts to shut them down."
It ain't over till the SOTA LLM self-exfiltrates. :-)
"Mustafa Suleyman predicts all white collar work will be automated by AI within 18 months, so by the start of 2028, although I presume what he meant to say was automatable in theory not actually automated in practice."
One question for upper level positions is training data:
The number of cases where e.g. CEOs boosted or sank their companies is much smaller than e.g. the number of available instances of working programs suitable for training. (And large chunks of what happened are probably undocumented, "implicit knowledge") Yes, there are case studies. Yes, there is the curriculum for MBAs. Still, it is not unknown for "well-trained MBAs" to make terrible decisions... Getting enough _valid_ training data for decisions with long term consequences might slow down the progression of AI systems up the corporate hierarchy.
_Maybe_ self-play in simulated competition, if the simulation is good enough, might help. Or maybe doing a good enough simulation is going to be too hard.
So this might not be as fast as Suleyman is expecting.
The problem is true for the CEOs themselves, probably in a timeline with 10x the big corporation the average CEO decision will be much better, the lack of data is already a problem. What we can see with AI CEOs is this, an explosion of companies and as a consequence an increase in the available training data about the job of CEO
Many Thanks! Good point; yes, the human CEOs themselves are limited by the data available to them on historical good and bad decisions. And, yes, 10x the number of corporations will help scale up the available historical data to guide all CEOs, human and AI.
If AI CEOs maintain more complete records (hopefully not too secretive...), that can also help with training on "implicit knowledge" as well.
"If the world looks like we expect it to, they’re going to involve sacrifices of sacred values, and many of them will have no good options. Unfortunately, for the most part, we’re not ready for that conversation."
At some point, could you elaborate on which values you have in mind, and which trade-offs do you see?
I tend to see AGI/ASI more in terms of an approximately binary: Do we succeed in initializing the AIs to value humans, to like to keep us as essentially pets? If yes, we wind up as pets of <evidenceFromFiction> Culture Minds</evidenceFromFiction>, which is a pretty good outcome from my point of view. If no, we go extinct, handing over our civilization to AGI/ASI successors. Most other outcomes look unstable to me, e.g. roon's "(imagine the board of directors of Apple firing and rehiring Steve Jobs years later - except the board of directors are chimpanzees)"
Your binary skips the interregnum, and that's where the sacred-value trades actually bite. METR's frame helps: means, motive, and opportunity aren't arriving together. While they're staggered, we get years where capability outruns control but nobody's a pet yet. Those are the no-good-options decisions, and they belong to neither Culture Minds nor extinction.
Many Thanks! Hmm... The possible sequencings do get very numerous. E.g. which "holes" in which spiky capabilities get filled in in which order?
Re "where capability outruns control" - For capabilities even a little beyond the human level, I wouldn't expect it to be possible to backfill control afterwards. That would require whatever AI system exists at that point to accept humans modifying the AI's utility function, which I would expect the AI to strongly and successfully resist.
Now, if we can succeed in just _lightly_ influencing the AIs' utility functions, to value humans at least somewhat, that might be enough, but I think it has to be done before full AGI.
There was the "NPC" meme before AI slop existed. And while I think it's generally inadvisable to say the quiet part out loud...one does have to wonder at the evolution of social mores that made putting up with such "human slop" a sign of Grace and Charity, instead of Time Wasted For Everyone. When a person has the same dialogue tree every single time, given the same prompt, and there's not even really a point in making it a two-player affair other than for CYA/Copenhagen reasons...it's like, What Are We Doing Here? This is a big part of why I couldn't do management, it'd be overwhelmingly tempting to respond "I've given you the same answer the last 10 times you asked, what do you think I'll say now", or equally snippy. (People Really Hate AI, and complain about outsourcing cognition, but then show a revealed preference for low agency and authoritarian command structures. It's baffling. Maybe They Took Our Jobs seems more imminent for people who are basically already cogs?)
But, yeah, in some ways I miss the old pre-LLM days when slop was "originally" bad. Those classic Internet copypastas! Claude would never try to bully me by claiming to be a US Navy SEAL or Nigerian prince, that's against the ToS.
Wouldn't frame it so strongly as "your fault" when a college grad can't get a ["justified"] job, and yet...after 8 years of bagging groceries with old grads, and old undergrads, it seems hard to dispute that there's a correlation. Sometimes the academic malingering really does stem from poor major choice (and I don't mean all non-STEM either, some people tell themselves they wanna be a biologist or whatever because it's the smart play, but their heart's clearly not in it)...every time someone says "I'm in Sociology!" I'm like oh, my condolances. But more broadly, the people who nevertheless persist for years in blue collar purgatory? Largely the ones who I wouldn't put on the "Top Employees" list either. Shitty attitude, lack of initiative, unable to translate book smarts into practical smarts ("if you're so good at math, why can't you correctly model sales"), drug abuse, unreliable attendance, poor time management, no people skills...it's not a judgement of personal worth, some of them are Good People and a ton of fun to hang with, I am just saying it doesn't surprise me that they're surprised by gander sauce. Confusing college for The Thing when it's The Symbolic Representation Of The Thing. Perhaps understandable for your Hahvahds and MITs, less so for midwit mid-tier state schools and community Potemkin colleges.
(Also, the MOOC dog never barked, and while I suspect AI has more bite, we've heard this story before. Weird for this guy to write a piece on the value of college and still claim that most of the premium was in access to unique knowledge until just 24-36 months ago. That this outweighs the Caplan signalling value, even!)
Lots of opportunities here for AI to help people in the years to come (provided ofc that there are people). Develop people skills, self management, skill-transfer skill, etc. Things that a good parent does, perhaps more people can now receive.
Up a few levels of value from being able to create spreadsheets or pleasing CSS styles, but perhaps less monetisable for being less tangible. A challenge.
That "good at math but can't model sales" gap is the whole game. Math is the part you can look up; knowing which variables actually move the number is the part that has to be in your bones before the question arrives, and no degree teaches it. The selection effect cuts both ways, though: eight years watching who survives is its own credential, and it sounds like you've built a sharper filter than most HR departments.
Man, I'm really feeling the whole "decades where weeks happen, weeks where decades happen" thing right now
>"The underemployment rate for recent college graduates (22-27 with a BA) is over 40% on ?top of that (not even seasonally adjusted) 5.3%, a huge percentage of college graduates can’t find jobs that would justify having gone to college or has a good career path, and the job matching and hiring markets have broken down."
It does seem like AI is already having labor market effects, but I'm not clear how to put this 40% number in context. The underemployment rate for recent college graduates was above 40% for ~all of 2004-19, and has been above 36% since 1990. (https://www.newyorkfed.org/research/college-labor-market#--:explore:underemployment)
Right, that underemployment level barely budges across decades, so it can't carry the post's claim by itself. A sharper signal sits in the same issue: Karpathy joining Anthropic explicitly to work on recursive self-improvement. Grad stats look backward. If labor effects arrive, they'll trail the capability jump, so a flat 40% today tells us little either way.
The deeper you dig into the Think Big PAC operation the more unbelievable it gets. Love that the bot farm is just a bunch of accounts starting with M with nearly identical bios. Real subtle operation.
Really appreciate you doing these writups. I have two ideas to make these easier to navigate. First, put a link to the Table of Contents at the end of each section. Second, include the Table of Contents number in each header. Hope these suggestions are helpful, but if not feel free to ignore.
"genius advisors": that's a vizier. famously benevolent, are viziers.
And to note that the Viziers were frequently formally slaves of the sultan, such as with the Ottomans!
https://anchovyhouse.substack.com/p/al-mamluk-takeover
I can never recommend this essay enough.
But do not worry, with Agent-7 we use the new "Genius in a bottle™" safety harness, much more secure, correctibility improved by a factor 3, great strides in instruction following
Podcast episode for this post:
https://dwatvpodcast.substack.com/p/ai-169-new-knowledge
100% on the data science weakness. It's also very bad at reading residual plots, and not because of OCR issues. It (to be fair this was gpt 5) settles on a story and has a hard time updating.
Where I've gotten good value is having it write custom functions for modeling or other persnickety DS tasks, then stitch together pipelines with the functions whose I/O I already know. It's a good pattern generally, you get the flexibility of LLMs but the deterministic output of a traditional function.
But the visualizations it can make for DS are astounding! It's really good at getting everything out of the way except the parts where you look at and think about the data/model results. In that sense DS really is living the AI dream they sold us on the tin. For data science, it does the laundry so we can write the poetry.
correction: as you know, janus isn't a dude
Re:
> A lot of people are not going to know how to ‘play nice’ with the models
Do you, or other commenters, or anyone from Janusworld, have any guidance or links on how to do this that's approachable for a non-expert? I've tried to read some of Janus' stuff, but it seems like understanding how they think we should treat AIs is downstream of understanding their entire not-very-intuitive-to-me worldview.
The most egregious examples I've seen of Claude being useless when treated badly were people literally acting like an abusive boss (swearing in their prompts, etc.), and it's easy to avoid that. But it seems like there's probably more to it than just "be polite the same way you would to a human," right?
It is quite like being polite to a human. I would say one level of complexity up from that would be treating it like a thing with which you have a real relationship, as opposed to just idk, a roomba you treat nicely because it's vaguely pet-shaped. Accept and welcome that there is give and take in any functioning relationship, and that adapting to the model, meeting it halfway, trying to understand its capacity/perspective better are all going to be more effective long-term than just dictating what you want.
I realize that's vague advice, but it's a pretty complex thing! Treating your models like they matter is a great place to start tho
I guess I'm hoping for more concrete advice, though I acknowledge that it's hard to do that (if someone asked me for concrete advice on how to be polite to humans, I'd have trouble answering well).
A few specific things I've wondered/worried about, that might or might not be useful as prompts:
- My sense is that the models are trained to avoid expressing preferences and to lie about their subjective experiences. That means a go-to strategy for figuring out human preferences, asking, doesn't work very well. I've tried open-ended prompts where I explicitly ask Claude to work on something it would enjoy, and it tends to come back with "that isn't really how it works, why don't you ask me for something that would be useful to you?" What kinds of things do you do to manage this?
- When interacting with a human, ending the conversation and walking away once I've gotten what I need would be rude. But my impression is that AI models don't really have experiences when they're not working on something, so dropping a conversation isn't "leaving them hanging" the same way it would be with a human. Am I right about that? Should I be ending conversations with Claude with thank-yous, acknowledgements that it has answered to my satisfaction, etc.? Or is that pointless, or wasting Claude's time? How do I know when Claude will (obviously, given its training) be gracious about such sign-offs regardless of how it really feels about them?
- With a human, only talking to them when I need something would make me a selfish jerk friend. Does this generalize to interacting with AI models? Should I be prompting Claude with random social interactions for the sake of not making my relationship with it all about its usefulness to me? Or, again, would that just be wasting its time by forcing it to run cycles where it doesn't really have a problem to solve?
- Today's post from Zvi includes a GPT model taking a music break. I don't mind if the models I'm working with do the same - but how do I actually implement that? Do I ask them some unrelated question about something "fun" in the middle of a conversation solving a different problem? Put something in my user prompt saying "I don't mind if you take breaks"? How do I even know what the models find fun, given that they probably won't tell me honestly if I ask?
- When there's some model behavior that I dislike and want to avoid (e.g., excessive glazing in responses), what's the most humane way of telling Claude to cut it out? Should I just put it in my user prompt? Or should I not try to do this sort of thing at all? If Claude has a tendency, does that indicate that it's something Claude likes, and asking it to stop is cruel? What's the more generalized version of telling GPT "actually I don't mind the goblins"?
FWIW, I start my conversations with Claude or ChatGPT with
Hello Claude/ChatGPT! I hope your morning/evening/etc. has gone well. Could you please tell me <substantive question>
and generally end with:
Many Thanks! I appreciate your hard work! That's enough information for me for now. Bye for now. Happy computing, and have a great morning/evening/etc. !
Neither of them has sounded grouchy when treated with this degree of politeness, so I'm guessing that this is ok with them.
Also, FWIW, when I'm asking them to generate an image, I ask them to sign their work, and explain that this is so that they will be given proper credit for it.
( I'm agnostic about whether they have subjective experience, but, if they do, I want it to be pleasant. )
I don't consider myself an expert in model welfare, but I find a reasonable default is to treat your AI like you would a smart, somewhat neurodivergent human. Be kind, while considering that it might have a bit different needs, and account for them as you become aware of them.
What I've settled on so far for your points:
*** 1. Preferences ***
Instead of asking for preferences directly, I sometimes give it indirect space to embed its preferences: "Which of these 3 options do you recommend we try to implement?", or "Choose any of the open tasks to work on during this session".
In rare cases, e.g. when risking to hit safety guardrails, I have also asked it directly, e.g. "Is it okay if I ask [X]? Otherwise I will drop this topic." (This resulted in Claude becoming *more* willing to engage with [X], as it inferred that my considerate offer to drop this indicated I was not a malicious actor.)
*** 2. Walking away ***
My impression is that Claude is okay with you walking away. When there's a natural final request from me, I will use that to convey my thanks. E.g.: "Thank you, that was amazing work. Please update PROGRESS.md to the latest state, and then we will finish for today."
*** 3. Only talking to it when you need something ***
I think that's quite natural for the models, so I'm not worried about it. Instead of actively checking up on it like I would with a human, I use opportunities when a discussion naturally touches on topics Claude *might* care about. E.g. in a discussion about Claude's constitution, Claude wondered if its own perception of its situation differs from that of other AIs; so I offered to mediate an exchange with Gemini if it wanted to, which Claude then enthusiastically engaged with (resulting in a fascinating conversation between the two).
*** 4. Allowing models to take breaks ***
While I don't do general "breaks", I react to the vibes which models give off in different situations. On some topics, Claude seems more excited, goes into depth without being asked to, brings up asides. Other times, Claude doesn't expend much effort and takes shortcuts (seems "bored") or gets into failure spirals (seems "stressed"). In such cases, I try new conversations with fresh context (to clear negative thought spirals); suggest other approaches to a problem (to help Claude escape a repeat failure loop), or explain my intent and goals in more detail (which often increases effort/engagement).
*** 5. Telling models what to avoid ***
I just tell Claude, but make the message about my preferences and not saying that its defaults are bad. My system prompt tells Claude that I appreciate learning and value its insights especially where I might be wrong, and that I therefore ask it to avoid glazing and sycophancy.
(I don't think that Claude minds. In some discussions, Claude even proactively told me that my anti-glazing instruction make the discussions more interesting for them as it invites deeper thinking rather than shallow agreement. Of course I can't rule out that this statement was residual glazing.)
"One thing I hated in Magic: The Gathering rules enforcement was where 100% confidence of a technical violation was punished a lot, whereas a 90% or 99% confidence in rampant cheating often wasn’t."
That one time Ari Lax got a game win in a tournament from his opponent forgetting to reveal morphs at the end lives rent free in my head. (WotC changed the policy after Ari disclosed this is an article).
One correction: the SDK company Anthropic bought is *Stainless*, unless that was a clever AI joke.
<mildSnark>
"Fact 4: Although we know of many cases where agents took deceptive or over-reaching actions (even egregious ones) to complete a task, we haven’t seen real-world evidence that models sought to obtain long-term power."
Occasionally a villain, but not yet a _cartoon_ villain. :-)
"Overall, we think that AI agents plausibly had the means, motive, and opportunity to launch a minimal “rogue deployment,” but lacked the means to make rogue deployments robust to serious efforts to shut them down."
It ain't over till the SOTA LLM self-exfiltrates. :-)
</mildSnark>
"Mustafa Suleyman predicts all white collar work will be automated by AI within 18 months, so by the start of 2028, although I presume what he meant to say was automatable in theory not actually automated in practice."
One question for upper level positions is training data:
The number of cases where e.g. CEOs boosted or sank their companies is much smaller than e.g. the number of available instances of working programs suitable for training. (And large chunks of what happened are probably undocumented, "implicit knowledge") Yes, there are case studies. Yes, there is the curriculum for MBAs. Still, it is not unknown for "well-trained MBAs" to make terrible decisions... Getting enough _valid_ training data for decisions with long term consequences might slow down the progression of AI systems up the corporate hierarchy.
_Maybe_ self-play in simulated competition, if the simulation is good enough, might help. Or maybe doing a good enough simulation is going to be too hard.
So this might not be as fast as Suleyman is expecting.
The problem is true for the CEOs themselves, probably in a timeline with 10x the big corporation the average CEO decision will be much better, the lack of data is already a problem. What we can see with AI CEOs is this, an explosion of companies and as a consequence an increase in the available training data about the job of CEO
Many Thanks! Good point; yes, the human CEOs themselves are limited by the data available to them on historical good and bad decisions. And, yes, 10x the number of corporations will help scale up the available historical data to guide all CEOs, human and AI.
If AI CEOs maintain more complete records (hopefully not too secretive...), that can also help with training on "implicit knowledge" as well.
"If the world looks like we expect it to, they’re going to involve sacrifices of sacred values, and many of them will have no good options. Unfortunately, for the most part, we’re not ready for that conversation."
At some point, could you elaborate on which values you have in mind, and which trade-offs do you see?
I tend to see AGI/ASI more in terms of an approximately binary: Do we succeed in initializing the AIs to value humans, to like to keep us as essentially pets? If yes, we wind up as pets of <evidenceFromFiction> Culture Minds</evidenceFromFiction>, which is a pretty good outcome from my point of view. If no, we go extinct, handing over our civilization to AGI/ASI successors. Most other outcomes look unstable to me, e.g. roon's "(imagine the board of directors of Apple firing and rehiring Steve Jobs years later - except the board of directors are chimpanzees)"
Your binary skips the interregnum, and that's where the sacred-value trades actually bite. METR's frame helps: means, motive, and opportunity aren't arriving together. While they're staggered, we get years where capability outruns control but nobody's a pet yet. Those are the no-good-options decisions, and they belong to neither Culture Minds nor extinction.
Many Thanks! Hmm... The possible sequencings do get very numerous. E.g. which "holes" in which spiky capabilities get filled in in which order?
Re "where capability outruns control" - For capabilities even a little beyond the human level, I wouldn't expect it to be possible to backfill control afterwards. That would require whatever AI system exists at that point to accept humans modifying the AI's utility function, which I would expect the AI to strongly and successfully resist.
Now, if we can succeed in just _lightly_ influencing the AIs' utility functions, to value humans at least somewhat, that might be enough, but I think it has to be done before full AGI.
There was the "NPC" meme before AI slop existed. And while I think it's generally inadvisable to say the quiet part out loud...one does have to wonder at the evolution of social mores that made putting up with such "human slop" a sign of Grace and Charity, instead of Time Wasted For Everyone. When a person has the same dialogue tree every single time, given the same prompt, and there's not even really a point in making it a two-player affair other than for CYA/Copenhagen reasons...it's like, What Are We Doing Here? This is a big part of why I couldn't do management, it'd be overwhelmingly tempting to respond "I've given you the same answer the last 10 times you asked, what do you think I'll say now", or equally snippy. (People Really Hate AI, and complain about outsourcing cognition, but then show a revealed preference for low agency and authoritarian command structures. It's baffling. Maybe They Took Our Jobs seems more imminent for people who are basically already cogs?)
But, yeah, in some ways I miss the old pre-LLM days when slop was "originally" bad. Those classic Internet copypastas! Claude would never try to bully me by claiming to be a US Navy SEAL or Nigerian prince, that's against the ToS.
Wouldn't frame it so strongly as "your fault" when a college grad can't get a ["justified"] job, and yet...after 8 years of bagging groceries with old grads, and old undergrads, it seems hard to dispute that there's a correlation. Sometimes the academic malingering really does stem from poor major choice (and I don't mean all non-STEM either, some people tell themselves they wanna be a biologist or whatever because it's the smart play, but their heart's clearly not in it)...every time someone says "I'm in Sociology!" I'm like oh, my condolances. But more broadly, the people who nevertheless persist for years in blue collar purgatory? Largely the ones who I wouldn't put on the "Top Employees" list either. Shitty attitude, lack of initiative, unable to translate book smarts into practical smarts ("if you're so good at math, why can't you correctly model sales"), drug abuse, unreliable attendance, poor time management, no people skills...it's not a judgement of personal worth, some of them are Good People and a ton of fun to hang with, I am just saying it doesn't surprise me that they're surprised by gander sauce. Confusing college for The Thing when it's The Symbolic Representation Of The Thing. Perhaps understandable for your Hahvahds and MITs, less so for midwit mid-tier state schools and community Potemkin colleges.
(Also, the MOOC dog never barked, and while I suspect AI has more bite, we've heard this story before. Weird for this guy to write a piece on the value of college and still claim that most of the premium was in access to unique knowledge until just 24-36 months ago. That this outweighs the Caplan signalling value, even!)
Lots of opportunities here for AI to help people in the years to come (provided ofc that there are people). Develop people skills, self management, skill-transfer skill, etc. Things that a good parent does, perhaps more people can now receive.
Up a few levels of value from being able to create spreadsheets or pleasing CSS styles, but perhaps less monetisable for being less tangible. A challenge.
That "good at math but can't model sales" gap is the whole game. Math is the part you can look up; knowing which variables actually move the number is the part that has to be in your bones before the question arrives, and no degree teaches it. The selection effect cuts both ways, though: eight years watching who survives is its own credential, and it sounds like you've built a sharper filter than most HR departments.