The Canadian government invoking the Emergencies Act to (among other things) freeze the bank accounts of people protesting against vaccine mandates is, I think, a provable example of a government debanking their political enemies.
"Those Canadians ought to have a a proper constitution that forbids this" is one possible response. yes...
Well China has already been ahead of the curve on that front, but Canada was probably the most notable Western example though it remains to been seen whether Australia will follow suit.
Canada has bigger problems anyways with their Online News and Streaming Acts.
Thank *you* Zvi, for all you write and collate, your blog continues to be one of the best overviews of "What's going on in [subject]", bringing together so much from so many places.
Also, thank you anyone who listen and supports in creating my multi voiced podcast conversions, case in point, podcast episode for this post!:
I gotta say, Zvi, your newsletters have kinda become the highlight of my week. Informative, insightful, tightly written, well-rounded, fun writing style, and actually quite neutral and balanced despite your own personal views and sympathies (if only most other journalists could at least try to hold themselves to that standard!).
I’m more on the AGI-skeptical side (though I acknowledge I could very well be wrong), but I always appreciate the framing you provide and how you articulate the arguments on both sides, it helps me to get and maintain a broader perspective.
So yeah, thanks for putting in the time and effort to write these each week (and not just for AI… yesterday’s newsletter on the Jones Act was quite eye opening; the housing and dating ones are fun, too).
Thank you and happy thanksgiving! Unfortunately, like most weeks, it had some disturbing updates but at least, it seems that there have been no major capability jumps.
Here is to hoping for a good future where humanity survives.
I do note that industrialized dehumanization/successionism has become more prominently mentioned as an existential risk and one can hope that pushes in more, as well as AI brainwashing.
So happy Anthropic added profile preferences to Claude for this reason. Lots of time will be saved from giving instructions again at the beginning of each new chat.
A real beautiful side benefit is that insofar as the model tries to align to you, this removes the most deceptive and easy route of natural language manipulation and it has to actually align to your intentions.
I like/read more poetry than most people, I'm sure, but I'm definitely not a "poetry expert". This is still at the level of "the woman in that picture has six fingers and four hands and her necklace is blending into her skin". I would be shocked if you couldn't identify it.
(To be more precise: lots of human poetry is also bad; I'm not saying you could pick out AI poems from arbitrary human poems. But if you find a poet you like, and you ask the AI to write poems in their style, I am _very_ confident you could distinguish an AI poem from an actual poem by that author which you hadn't yet seen.)
I think the bigger takeaway was how dead poetry is as a literary artform. Over 90% of participants indicated that they read effectively no poetry, and over two thirds were entirely unfamiliar with their assigned poet, despite the list featuring many major literary figures who are known beyond their poetry.
Besides being able to imitate the best human persuaders, AI can just use more forms of information and methods like spaced repetition beyond humans could. Insofar as AI can be greater educators, they are great persuaders: its just manipulating information into the brains of humans.
> One frustrating conversation was about persuasion. Somehow there continue to be some people who can at least somewhat feel the AGI, but also genuinely think humans are at or close to the persuasion possibilities frontier - that there is no room to greatly expand one’s ability to convince people of things, or at least of things against their interests.
Isn’t the obvious exercise here to build and continuously refine an LLM system to make the argument?
It wouldn’t take very much in terms of donor money to fund tokens and run it open source with volunteer contributions. Even giving some fellowships to work on it might have wildly positive expected value.
On the other, I do think you could get both first-order results and an important existence proof well before hitting “hyperpersausive.” And if one is not willing to fight fire with fire, how serious is one really about the purportedly existential stakes?
I also want to say thank you. I never comment or like, but I subscribe to over 100 newsletters and yours is one of only three that come directly to my inbox. It’s been a joy to learn from you and with you.
>If Sama (the company) was getting paid by Sama (the CEO) $12.50 per hour, and only $2 per hour of that went to the workers, then something is foul is afoot.
This seems pretty standard to my experience with "specialised service firms". Both from personal experience as a trainee accountant (I billed in 6 minute increments, but when you did the maths my charge out rate was around 6x that of my hourly wage in Big4 accounting/2nd tier firms) and talking with India based employees from outsourcing firms based there on what they actually get paid, knowing what we pay the firm (think Accenture, InfoSys, Genpact, etc.).
For a quick sense check - Genpact Ltd publicly file their financial statements and their pure markup (revenue vs cost of revenue) in these suggest only a 2x markup, so I'd have to dig into how they account for what goes into "cost of revenue" to understand the gap (do team leads fall into cost of revenue, where department managers don't?, at what point on the hierarchy does it move from CoR to Overheads?). I know for BPO's there are commonly several layers of management in operations, so this still seems to track at a surface level as "reasonable" to me, if perhaps somewhat on the high end.
Thanks for the great work! I actually have the old wordpress site in my RSS feed and generally read from there, but always come here to comment.
When I worked for a tutoring company (quite a few years back), they charged clients $48/hour, and paid me $18/hour. They did at least provide the building where I met with clients, but it still always seemed egregious, and that was only a 3:1 ratio.
Can't remember if it was this blog or somewhere else but I once proposed hiring a translator for Yudkowsky and offered to do it myself if the pay was good enough.
Given the recent salience of 'Bene Gesserit voice' I am a little surprised that nobody has used that metaphor for persuasion. I think Gordon R Dickson Dorsai novels had something similar (Friendlies from memory)
I’ll say re:Windsurf, I made the change earlier this week. It seems actively better and searching and making changes given the context, but sometimes it changes way more than you asked for, so you need to be precise and check (which you should be doing anyway)
The AI roleplaying game outcomes are fascinating. Can't wait until an AI player draws something a bit more spicy like AIs having similar moral worth to humans.
That's why what you do is ask about counterfactuals - in our game we know EXACTLY what would have happened if the AI had a goal like that, we would all have been super dead.
> You don’t have many slots to spend on things like this.
TBH I mostly read your newsletter while procrastinating something else out of sheer akrasia
Re: The Week in Audio
The Canadian government invoking the Emergencies Act to (among other things) freeze the bank accounts of people protesting against vaccine mandates is, I think, a provable example of a government debanking their political enemies.
"Those Canadians ought to have a a proper constitution that forbids this" is one possible response. yes...
Well China has already been ahead of the curve on that front, but Canada was probably the most notable Western example though it remains to been seen whether Australia will follow suit.
Canada has bigger problems anyways with their Online News and Streaming Acts.
Thank *you* Zvi, for all you write and collate, your blog continues to be one of the best overviews of "What's going on in [subject]", bringing together so much from so many places.
Also, thank you anyone who listen and supports in creating my multi voiced podcast conversions, case in point, podcast episode for this post!:
https://open.substack.com/pub/dwatvpodcast/p/ai-92-behind-the-curve
I gotta say, Zvi, your newsletters have kinda become the highlight of my week. Informative, insightful, tightly written, well-rounded, fun writing style, and actually quite neutral and balanced despite your own personal views and sympathies (if only most other journalists could at least try to hold themselves to that standard!).
I’m more on the AGI-skeptical side (though I acknowledge I could very well be wrong), but I always appreciate the framing you provide and how you articulate the arguments on both sides, it helps me to get and maintain a broader perspective.
So yeah, thanks for putting in the time and effort to write these each week (and not just for AI… yesterday’s newsletter on the Jones Act was quite eye opening; the housing and dating ones are fun, too).
Cheers!
Wish you are right on AGI skepticism. Humans forever!
Thank you and happy thanksgiving! Unfortunately, like most weeks, it had some disturbing updates but at least, it seems that there have been no major capability jumps.
Here is to hoping for a good future where humanity survives.
I do note that industrialized dehumanization/successionism has become more prominently mentioned as an existential risk and one can hope that pushes in more, as well as AI brainwashing.
Fwiw, I found what seems to be a simple solution to AI charm that I am surprised hasnt been mentioned more.
Just tell it to express itself in a more stilted, non-natural language way. You'll get the benefits without it trying to schmooze.
Honestly, it seems to behave better. Neurologically, it seems easier to detect any drift in oneself via more dramatic changes as well.
So happy Anthropic added profile preferences to Claude for this reason. Lots of time will be saved from giving instructions again at the beginning of each new chat.
A real beautiful side benefit is that insofar as the model tries to align to you, this removes the most deceptive and easy route of natural language manipulation and it has to actually align to your intentions.
> But it’s not something I expect I could identify, nor do I have any real understanding of what it is or why I should care?
Have you actually looked at any of the poems the AI wrote in the paper? Because they're not just a little bit bad. They are very bad. https://x.com/colin_fraser/status/1861333943724769355
I like/read more poetry than most people, I'm sure, but I'm definitely not a "poetry expert". This is still at the level of "the woman in that picture has six fingers and four hands and her necklace is blending into her skin". I would be shocked if you couldn't identify it.
(To be more precise: lots of human poetry is also bad; I'm not saying you could pick out AI poems from arbitrary human poems. But if you find a poet you like, and you ask the AI to write poems in their style, I am _very_ confident you could distinguish an AI poem from an actual poem by that author which you hadn't yet seen.)
I think the bigger takeaway was how dead poetry is as a literary artform. Over 90% of participants indicated that they read effectively no poetry, and over two thirds were entirely unfamiliar with their assigned poet, despite the list featuring many major literary figures who are known beyond their poetry.
I'm surprised that the idea of humans being at the persuasion frontier is there given the paper like this:
https://www.psychologytoday.com/us/blog/emotional-behavior-behavioral-emotions/202403/ai-is-becoming-more-persuasive-than-humans
Besides being able to imitate the best human persuaders, AI can just use more forms of information and methods like spaced repetition beyond humans could. Insofar as AI can be greater educators, they are great persuaders: its just manipulating information into the brains of humans.
> One frustrating conversation was about persuasion. Somehow there continue to be some people who can at least somewhat feel the AGI, but also genuinely think humans are at or close to the persuasion possibilities frontier - that there is no room to greatly expand one’s ability to convince people of things, or at least of things against their interests.
Isn’t the obvious exercise here to build and continuously refine an LLM system to make the argument?
It wouldn’t take very much in terms of donor money to fund tokens and run it open source with volunteer contributions. Even giving some fellowships to work on it might have wildly positive expected value.
"And all you needed to do to persuade people about the dangers of hyperpersuasive AI was to build a hyperpersuasive AI."
Overall, I agree this is a net positive but lol
On the one hand, I know, right?
On the other, I do think you could get both first-order results and an important existence proof well before hitting “hyperpersausive.” And if one is not willing to fight fire with fire, how serious is one really about the purportedly existential stakes?
I agree, and fwiw, I have been using AI quite successfully to help me make arguments on existential risk.
I also want to say thank you. I never comment or like, but I subscribe to over 100 newsletters and yours is one of only three that come directly to my inbox. It’s been a joy to learn from you and with you.
>If Sama (the company) was getting paid by Sama (the CEO) $12.50 per hour, and only $2 per hour of that went to the workers, then something is foul is afoot.
This seems pretty standard to my experience with "specialised service firms". Both from personal experience as a trainee accountant (I billed in 6 minute increments, but when you did the maths my charge out rate was around 6x that of my hourly wage in Big4 accounting/2nd tier firms) and talking with India based employees from outsourcing firms based there on what they actually get paid, knowing what we pay the firm (think Accenture, InfoSys, Genpact, etc.).
For a quick sense check - Genpact Ltd publicly file their financial statements and their pure markup (revenue vs cost of revenue) in these suggest only a 2x markup, so I'd have to dig into how they account for what goes into "cost of revenue" to understand the gap (do team leads fall into cost of revenue, where department managers don't?, at what point on the hierarchy does it move from CoR to Overheads?). I know for BPO's there are commonly several layers of management in operations, so this still seems to track at a surface level as "reasonable" to me, if perhaps somewhat on the high end.
Thanks for the great work! I actually have the old wordpress site in my RSS feed and generally read from there, but always come here to comment.
When I worked for a tutoring company (quite a few years back), they charged clients $48/hour, and paid me $18/hour. They did at least provide the building where I met with clients, but it still always seemed egregious, and that was only a 3:1 ratio.
Can't remember if it was this blog or somewhere else but I once proposed hiring a translator for Yudkowsky and offered to do it myself if the pay was good enough.
Given the recent salience of 'Bene Gesserit voice' I am a little surprised that nobody has used that metaphor for persuasion. I think Gordon R Dickson Dorsai novels had something similar (Friendlies from memory)
I’ll say re:Windsurf, I made the change earlier this week. It seems actively better and searching and making changes given the context, but sometimes it changes way more than you asked for, so you need to be precise and check (which you should be doing anyway)
Thank you Zvi. Love your work.
The AI roleplaying game outcomes are fascinating. Can't wait until an AI player draws something a bit more spicy like AIs having similar moral worth to humans.
That's why what you do is ask about counterfactuals - in our game we know EXACTLY what would have happened if the AI had a goal like that, we would all have been super dead.