For anything that you're working on, I think there is a moral obligation to ask yourself "Would I be excited if my kids were using this?"
If the answer is "Definitely no.", then you also have a moral obligation to stop working on that thing. Full stop.
I think this aligns with most people's moral intuition, but smart people are unfortunately really good at rationalizing behavior that aligns with their personal interest. That often takes the form of "If I don't do it (or my company doesn't do it), then someone else would do it anyway, so I might as well do a good version where I have control of the outcomes." My primary counterpoints would be:
- People in tech love to talk about the scarcity of talent, and the amazing power of agency. You Can Just Do Things, but this is so rare that it is very high impact when it happens. If you have scarce talent, knowledge, and agency, you don't have to use it to do negative things!
- By choosing to do it, you are actively setting an example and social norms that doing it is acceptable. Don't be part of that.
- "I will do this bad thing, because I will maintain control and make it less bad than it otherwise would be" is a plan that rarely survives contact with reality. Your behavior shapes your own character, and your values will often evolve to match your behavior, rather than vice-versa.
- If your "value above replacement" in your current role is really so low, what are you even doing there?
This especially applies to AI researchers, software engineers, product managers, and others in tech working on frontier capability right now. If you are uncomfortable with the end-uses or potential outcomes of what you are building, please just stop and find something else to work on. Your skills are in enough demand that you will certainly find a comfortable income doing something that you think is a benefit to the world.
I am literally an IP Attorney (I am not *your* IP attorney and this is not legal advice) and yeah, I think Zvi's take on the copyright issues here is spot on. OpenAI does not get carte blanche to violate copyright just because they have some (narrowly-tailored!) opt-out button. That's not how this works, that's not how any of this works.
I honestly have no idea what their in-house attorneys were smoking. My best guess would be something about the DMCA safe harbor provisions of 17 USC 512(c), but I would be *extremely* skeptical that those apply in a context in which OpenAI is literally (and knowingly!) generating the infringing content. That was supposed to be the whole point behind the training / generation fair use distinction.
Does the same apply (perhaps with even more force?) to trademarks. I have a very vague recollection that trademark owners have an obligation to actively defend their trademarks, IIRC.
In general, trademark owners do have an obligation to police their marks, but the point of trademark is specifically to associate goods and services with a specific provider, so the more common suits and disputes you see are cases of knock-offs or confusingly similar marks from two different companies. Any infringement by Sora et al. is likely to be a lot more incidental and less likely to implicate the mark owner's exclusive rights to use the mark in commerce in connection with their goods and services, whereas copyright infringement is more of a per se breach of rights but also doesn't come with the same policing obligations. If you have a visible logo in a piece of media that isn't disparaging the mark or implicating its market that's not really going to the heart of what trademark is about.
That said, definitely get a more fulsome opinion from a practitioner on this one. I do some TM work but it's not my primary practice.
"the point of trademark is specifically to associate goods and services with a specific provider, so the more common suits and disputes you see are cases of knock-offs or confusingly similar marks from two different companies. Any infringement by Sora et al. is likely to be a lot more incidental"
Good point! An image of a can of Coca-Cola in a SORA video doesn't purport to be an actual can of soda...
Can you clarify a bit more on why TikTok-style short videos are harmful? I've never really used any social media platform aside from Reddit, and don't have children, so I don't have any direct experience with this kind of thing. My outside perspective, however, is that while Twitter seems very harmful due to the way it structurally promotes outrage and tribal divisions (you can often tell whether someone you've just met is a heavy Twitter user, just from the pretty distinctive quirks it promotes), TikTok, by contrast, seems pretty anodyne, aside from the worrying possibility of subtle CCP propaganda.
My understanding is that the biggest concern there is addiction. Of course, we should distinguish between heavy use of an entertainment medium because we find that it improves our subjective quality of life, and heavy use that we regret because something has manipulatively compelled us to use it. A lot of video games use Skinner-box mechanics to compel people to keep playing even when they aren't having fun. I can kind of see how the TikTok algorithm might function similarly- users feel bored and frustrated scrolling through low-quality content, but feel irrationally compelled to do so by the rare possibility of good content. Is that actually the experience of users in practice?
You also mentioned a concern that users can become too socially isolated. Would that concern vanish if TikTok were entirely high-quality entertainment that users didn't regret consuming? Or is the concern really that the quality and availability of entertainment in our culture is too high, leading us to neglect other important things? If it's the latter, do you think we should reduce the availability of other very entertaining things, like high-quality movies, video games and novels?
A bit of searching turned up this 2024 report: https://www.who.int/europe/news/item/25-09-2024-teens--screens-and-mental-health, which finds that about 11% of children have problematic social media use (with a similar number for problematic video game use). Another 2023 study at https://pmc.ncbi.nlm.nih.gov/articles/PMC10773792/ suggests that while screen time and addiction are correlated, screen time isn't a very good predictor for addiction, since a lot of people with substantial screen time don't show symptoms of addiction. That, of course, doesn't let the platforms off the hook- most cigarette smokers also don't get lung cancer.
There does seem to be some evidence that social media might just be a proximate cause of addiction symptoms, rather than an ultimate cause, however- that is, if social media didn't exist, a lot of the people experiencing these symptoms might still experience them with another medium. See, for example, a twin study investigating a genetic cause of addition at: https://pmc.ncbi.nlm.nih.gov/articles/PMC9261223/, a study at: https://pmc.ncbi.nlm.nih.gov/articles/PMC10562828/, which investigates substitution between addictive behaviors, and this study at: https://pmc.ncbi.nlm.nih.gov/articles/PMC8787656/, which I think is arguing that underlying disorders like depression and anxiety strongly predict social media addiction. Note that I've only skimmed through these, so the evidence might not be super strong. If social media isn't the ultimate cause of social media addiction, however, that seems like it actually might let the platforms off the hook somewhat.
On the subject of Sora 2 in particular, I played around with it last night- generating a bunch of imaginary sci-fi movie scenes (which it was great at) and trying to get it to generate flythroughs of surreal landscapes from abstract Midjourney images (which failed completely after a couple dozen attempts). After a couple of hours, I hit the 100 video limit and briefly checked out the feed of other people's generations, but didn't find much of interest there. I guess my usage was pretty atypical, but as a single data point: I found it pretty genuinely fun and not obviously something I should regret.
Anecdotally, with TikTok, it seems optimized for destroying the attention span of its users like nothing else that has ever existed. The short-form video clones that have followed largely have the same impacts.
And it is, as you say, highly addictive. This means that, put together, it has proven an extraordinarily effective tool for shredding the complex analytical and thinking capabilities of children who have been exposed to it for years.
Anecdotally, my girlfriend is a high school teacher and frequently expresses dismay at how linearly negative the trendline has been for attention spans among the kids she interacts with. She’s named examples, including kids who are unable to focus on the screen long enough to read a single PowerPoint slide… this is no longer the exception, but the rule. I know anecdotes like that are of little objective explanatory value, but I feel it is worthwhile to include.
Also anecdotal, but I will occasionally open Instagram and get sucked into the algorithm (on Reels) for hours. I’m just spending this time consuming short-form content that is largely not memorable. I have never had a single one of these sessions where I don’t feel terrible about myself afterwards, and my subjective experience is extremely negative. I’ve asked around and this experience seems extremely common for others, so at least in my social circle (mid-to-late-20s, educated, non-elite) this seems like it’s definitely occurring.
The thing I hate about studies of phones (or subcomponents thereof, like social media or specific apps) is that they always return these unimpressive results, where a nonzero-but-not-particularly-noteworthy fraction of people have Clinically Significant Problems. Sometimes the numbers can be greased with the right framing effects - Zvi loves using the example of many/most college kids saying they'd pay/be paid trivial money to delete Instagram iff everyone in their social group did the same, for example. And against all of this empirical data, blessed by going through the ritualistic motions of Science, one only has...anecdotes, lying eyes. All the people one sees in public, frequently pretzeled into near-fetal positions around their phones; the drivers so engrossed in Reels they can't be bothered to safely operate their two-tonne metal contraption; the pedestrians who walk right into traffic while scrolling on TikTok, or perhaps bump into you and then act affronted that *you* didn't move out of *their* way. The inability to hold 10 seconds of dialogue with one's cashier at the grocery store. The compulsive need seemingly everyone else has to fill every silence, every spare moment with a phone. The manufactured mimesis of desire, where sales patterns of longstanding products abruptly change enduringly due to some bullshit viral post.
One could go on. One could caveat it with "well, that's revealed preference, and if they say they're enjoying themselves unproblematically then what can you do". One could postulate a broader theory of addiction where SM is mostly a displacement effect. Eliezer once said you can defy the data on one experiment, but not several of them. All the same: as with bad benchmarks, I simply think the studies say more about their own methodological limits than the actual state of the world. Fish asking for evidence of water, etc. (NB: Matt Yglesias does bite the "entertainment got too good" bullet, e.g. https://www.slowboring.com/p/is-ever-better-video-content-breaking . We never did get Vision Pro 2.0 though...)
The "revealed preference" argument - recently-ish used by Zuck on a podcast IIRC - is obviously total garbage when the "preferred" thing is addictive or otherwise predatory. "No, giving the man a limitless supply of heroin wasn't bad, it just revealed his preference to spent the rest of his drastically shortened life in a permanent dribbling stupour. He'd just never realised that's what he wanted all along!"
"Imagine YouTube had spent a lot of its resources early on taking down copyrighted material, before anyone demanded it (they mostly didn’t at first). They would have presumably gotten sued anyway."
They wouldn't have gotten sued. You would simply never have heard of YouTube.
There was a version of YouTube that spent a lot of its resources taking down copyrighted material. It was called "Google Video". It did not get sued, because it energetically cooperated with copyright owners. But it failed in the marketplace. YouTube was far, far more popular, and in fact its breakout successes (Lazy Sunday) were copyrighted material. Google bought YouTube after losing the battle for market share, because they preferred winning plus lawsuits to losing plus no lawsuits (by a then-large-seeming factor of $1.6B).
Alone, YouTube could have gotten crushed like Napster did. But that was a good reason for them to sell. Google could force copyright owners to negotiate, since Google had plenty of money for lawyers, and in general the copyright owners didn't want to be at war with Google forever.
The same dynamic will happen here. There is a huge demand for "video fan fiction". The technology is inevitably going to become very widespread. In the meantime, while the law is unclear, the eventual winners are probably going to push the boundaries of the law very hard.
but, but, but... Doesn't the SORA social network need to actively exacerbate polarization? Isn't that an essential KPI for a social network? If a social network was launched, and it doesn't prompt an assassination, was it really launched?
Isn't one of the obvious motives that if they have their own app, at least they don't feed the other "social" media apps for free? They keep some control over their tech and already large user base and the content generated by both?
Hey guys, Sora’s been blowing up lately, but I’ve noticed a lot of videos using people’s faces or private stuff without permission—kinda crossing the line, don’t you think?
Not sure if it’s the official Sora, but sora2ai.ai actually works great—full features, no invite needed, and even a free trial. Way more fun to use!
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/sora-and-the-big-bright-screen-slop
For anything that you're working on, I think there is a moral obligation to ask yourself "Would I be excited if my kids were using this?"
If the answer is "Definitely no.", then you also have a moral obligation to stop working on that thing. Full stop.
I think this aligns with most people's moral intuition, but smart people are unfortunately really good at rationalizing behavior that aligns with their personal interest. That often takes the form of "If I don't do it (or my company doesn't do it), then someone else would do it anyway, so I might as well do a good version where I have control of the outcomes." My primary counterpoints would be:
- People in tech love to talk about the scarcity of talent, and the amazing power of agency. You Can Just Do Things, but this is so rare that it is very high impact when it happens. If you have scarce talent, knowledge, and agency, you don't have to use it to do negative things!
- By choosing to do it, you are actively setting an example and social norms that doing it is acceptable. Don't be part of that.
- "I will do this bad thing, because I will maintain control and make it less bad than it otherwise would be" is a plan that rarely survives contact with reality. Your behavior shapes your own character, and your values will often evolve to match your behavior, rather than vice-versa.
- If your "value above replacement" in your current role is really so low, what are you even doing there?
This especially applies to AI researchers, software engineers, product managers, and others in tech working on frontier capability right now. If you are uncomfortable with the end-uses or potential outcomes of what you are building, please just stop and find something else to work on. Your skills are in enough demand that you will certainly find a comfortable income doing something that you think is a benefit to the world.
I am literally an IP Attorney (I am not *your* IP attorney and this is not legal advice) and yeah, I think Zvi's take on the copyright issues here is spot on. OpenAI does not get carte blanche to violate copyright just because they have some (narrowly-tailored!) opt-out button. That's not how this works, that's not how any of this works.
I honestly have no idea what their in-house attorneys were smoking. My best guess would be something about the DMCA safe harbor provisions of 17 USC 512(c), but I would be *extremely* skeptical that those apply in a context in which OpenAI is literally (and knowingly!) generating the infringing content. That was supposed to be the whole point behind the training / generation fair use distinction.
I'm morbidly curious about a related IP matter:
Does the same apply (perhaps with even more force?) to trademarks. I have a very vague recollection that trademark owners have an obligation to actively defend their trademarks, IIRC.
In general, trademark owners do have an obligation to police their marks, but the point of trademark is specifically to associate goods and services with a specific provider, so the more common suits and disputes you see are cases of knock-offs or confusingly similar marks from two different companies. Any infringement by Sora et al. is likely to be a lot more incidental and less likely to implicate the mark owner's exclusive rights to use the mark in commerce in connection with their goods and services, whereas copyright infringement is more of a per se breach of rights but also doesn't come with the same policing obligations. If you have a visible logo in a piece of media that isn't disparaging the mark or implicating its market that's not really going to the heart of what trademark is about.
That said, definitely get a more fulsome opinion from a practitioner on this one. I do some TM work but it's not my primary practice.
Many Thanks!
"the point of trademark is specifically to associate goods and services with a specific provider, so the more common suits and disputes you see are cases of knock-offs or confusingly similar marks from two different companies. Any infringement by Sora et al. is likely to be a lot more incidental"
Good point! An image of a can of Coca-Cola in a SORA video doesn't purport to be an actual can of soda...
My guess is that their product counsel came to the exact same conclusion but were overruled.
Can you clarify a bit more on why TikTok-style short videos are harmful? I've never really used any social media platform aside from Reddit, and don't have children, so I don't have any direct experience with this kind of thing. My outside perspective, however, is that while Twitter seems very harmful due to the way it structurally promotes outrage and tribal divisions (you can often tell whether someone you've just met is a heavy Twitter user, just from the pretty distinctive quirks it promotes), TikTok, by contrast, seems pretty anodyne, aside from the worrying possibility of subtle CCP propaganda.
My understanding is that the biggest concern there is addiction. Of course, we should distinguish between heavy use of an entertainment medium because we find that it improves our subjective quality of life, and heavy use that we regret because something has manipulatively compelled us to use it. A lot of video games use Skinner-box mechanics to compel people to keep playing even when they aren't having fun. I can kind of see how the TikTok algorithm might function similarly- users feel bored and frustrated scrolling through low-quality content, but feel irrationally compelled to do so by the rare possibility of good content. Is that actually the experience of users in practice?
You also mentioned a concern that users can become too socially isolated. Would that concern vanish if TikTok were entirely high-quality entertainment that users didn't regret consuming? Or is the concern really that the quality and availability of entertainment in our culture is too high, leading us to neglect other important things? If it's the latter, do you think we should reduce the availability of other very entertaining things, like high-quality movies, video games and novels?
A bit of searching turned up this 2024 report: https://www.who.int/europe/news/item/25-09-2024-teens--screens-and-mental-health, which finds that about 11% of children have problematic social media use (with a similar number for problematic video game use). Another 2023 study at https://pmc.ncbi.nlm.nih.gov/articles/PMC10773792/ suggests that while screen time and addiction are correlated, screen time isn't a very good predictor for addiction, since a lot of people with substantial screen time don't show symptoms of addiction. That, of course, doesn't let the platforms off the hook- most cigarette smokers also don't get lung cancer.
There does seem to be some evidence that social media might just be a proximate cause of addiction symptoms, rather than an ultimate cause, however- that is, if social media didn't exist, a lot of the people experiencing these symptoms might still experience them with another medium. See, for example, a twin study investigating a genetic cause of addition at: https://pmc.ncbi.nlm.nih.gov/articles/PMC9261223/, a study at: https://pmc.ncbi.nlm.nih.gov/articles/PMC10562828/, which investigates substitution between addictive behaviors, and this study at: https://pmc.ncbi.nlm.nih.gov/articles/PMC8787656/, which I think is arguing that underlying disorders like depression and anxiety strongly predict social media addiction. Note that I've only skimmed through these, so the evidence might not be super strong. If social media isn't the ultimate cause of social media addiction, however, that seems like it actually might let the platforms off the hook somewhat.
On the subject of Sora 2 in particular, I played around with it last night- generating a bunch of imaginary sci-fi movie scenes (which it was great at) and trying to get it to generate flythroughs of surreal landscapes from abstract Midjourney images (which failed completely after a couple dozen attempts). After a couple of hours, I hit the 100 video limit and briefly checked out the feed of other people's generations, but didn't find much of interest there. I guess my usage was pretty atypical, but as a single data point: I found it pretty genuinely fun and not obviously something I should regret.
Anecdotally, with TikTok, it seems optimized for destroying the attention span of its users like nothing else that has ever existed. The short-form video clones that have followed largely have the same impacts.
And it is, as you say, highly addictive. This means that, put together, it has proven an extraordinarily effective tool for shredding the complex analytical and thinking capabilities of children who have been exposed to it for years.
Anecdotally, my girlfriend is a high school teacher and frequently expresses dismay at how linearly negative the trendline has been for attention spans among the kids she interacts with. She’s named examples, including kids who are unable to focus on the screen long enough to read a single PowerPoint slide… this is no longer the exception, but the rule. I know anecdotes like that are of little objective explanatory value, but I feel it is worthwhile to include.
Also anecdotal, but I will occasionally open Instagram and get sucked into the algorithm (on Reels) for hours. I’m just spending this time consuming short-form content that is largely not memorable. I have never had a single one of these sessions where I don’t feel terrible about myself afterwards, and my subjective experience is extremely negative. I’ve asked around and this experience seems extremely common for others, so at least in my social circle (mid-to-late-20s, educated, non-elite) this seems like it’s definitely occurring.
The thing I hate about studies of phones (or subcomponents thereof, like social media or specific apps) is that they always return these unimpressive results, where a nonzero-but-not-particularly-noteworthy fraction of people have Clinically Significant Problems. Sometimes the numbers can be greased with the right framing effects - Zvi loves using the example of many/most college kids saying they'd pay/be paid trivial money to delete Instagram iff everyone in their social group did the same, for example. And against all of this empirical data, blessed by going through the ritualistic motions of Science, one only has...anecdotes, lying eyes. All the people one sees in public, frequently pretzeled into near-fetal positions around their phones; the drivers so engrossed in Reels they can't be bothered to safely operate their two-tonne metal contraption; the pedestrians who walk right into traffic while scrolling on TikTok, or perhaps bump into you and then act affronted that *you* didn't move out of *their* way. The inability to hold 10 seconds of dialogue with one's cashier at the grocery store. The compulsive need seemingly everyone else has to fill every silence, every spare moment with a phone. The manufactured mimesis of desire, where sales patterns of longstanding products abruptly change enduringly due to some bullshit viral post.
One could go on. One could caveat it with "well, that's revealed preference, and if they say they're enjoying themselves unproblematically then what can you do". One could postulate a broader theory of addiction where SM is mostly a displacement effect. Eliezer once said you can defy the data on one experiment, but not several of them. All the same: as with bad benchmarks, I simply think the studies say more about their own methodological limits than the actual state of the world. Fish asking for evidence of water, etc. (NB: Matt Yglesias does bite the "entertainment got too good" bullet, e.g. https://www.slowboring.com/p/is-ever-better-video-content-breaking . We never did get Vision Pro 2.0 though...)
The "revealed preference" argument - recently-ish used by Zuck on a podcast IIRC - is obviously total garbage when the "preferred" thing is addictive or otherwise predatory. "No, giving the man a limitless supply of heroin wasn't bad, it just revealed his preference to spent the rest of his drastically shortened life in a permanent dribbling stupour. He'd just never realised that's what he wanted all along!"
"Imagine YouTube had spent a lot of its resources early on taking down copyrighted material, before anyone demanded it (they mostly didn’t at first). They would have presumably gotten sued anyway."
They wouldn't have gotten sued. You would simply never have heard of YouTube.
There was a version of YouTube that spent a lot of its resources taking down copyrighted material. It was called "Google Video". It did not get sued, because it energetically cooperated with copyright owners. But it failed in the marketplace. YouTube was far, far more popular, and in fact its breakout successes (Lazy Sunday) were copyrighted material. Google bought YouTube after losing the battle for market share, because they preferred winning plus lawsuits to losing plus no lawsuits (by a then-large-seeming factor of $1.6B).
Alone, YouTube could have gotten crushed like Napster did. But that was a good reason for them to sell. Google could force copyright owners to negotiate, since Google had plenty of money for lawyers, and in general the copyright owners didn't want to be at war with Google forever.
The same dynamic will happen here. There is a huge demand for "video fan fiction". The technology is inevitably going to become very widespread. In the meantime, while the law is unclear, the eventual winners are probably going to push the boundaries of the law very hard.
I feel you are correct and this is the same calculus OpenAI is working with.
<mildSnarkGallowsHumor>
but, but, but... Doesn't the SORA social network need to actively exacerbate polarization? Isn't that an essential KPI for a social network? If a social network was launched, and it doesn't prompt an assassination, was it really launched?
</mildSnarkGallowsHumor>
Isn't one of the obvious motives that if they have their own app, at least they don't feed the other "social" media apps for free? They keep some control over their tech and already large user base and the content generated by both?
Wow! I'm super glad I read all the way to the end before creating a Sora account to try it.
Probably that disclaimer about not being able to delete it without also losing your ChatGPT account should go at the top somewhere
Hey guys, Sora’s been blowing up lately, but I’ve noticed a lot of videos using people’s faces or private stuff without permission—kinda crossing the line, don’t you think?
Not sure if it’s the official Sora, but sora2ai.ai actually works great—full features, no invite needed, and even a free trial. Way more fun to use!