My 2023 ACX predictions showed a clear lack of confidence in taking on the market. I won 30 markets for an average of +185 each, and lost 12 for an average loss of -185 each. When one goes 30-12, hitting a 71% mark versus the about 58% average price initially paid, that is worth noticing. It is possible that I generally benefited from 2023 being a year where not much happened outside of AI, but I think it’s time to know what we really think.
Ah, yes! That would be the one. I knew I'd missed something but somehow couldn't figure out what. I'll try to get time to look at the wording and figure it out.
Below, language from the Resolution Criteria and Background Information sections, if helpful:
*Resolution Criteria*
This question resolves as Yes if, before January 1, 2025, a bill is introduced in either the US House or US Senate which would impose one of the following restrictions on large language models or products using large language models.
-Forbids their creation
-Sets limits on how they're trained, for example by limiting access to previously usable training data or by setting limits on the number of parameters they may be trained with.
Prevents their use for certain applications, such as interacting with customers, interfacing with other applications, or performing actions on the web.
-Restricts the ability of US citizens to use foreign LLM-based products or restricts US operated LLM products or businesses from being sold to foreign customers or entities.
- - The export restriction must be specific, and not universally applied to all exports or exports of a broad industry. For example, a blanket ban on exports to a specific country would not qualify, and neither would a blanket ban on allowing the purchase of US businesses by investors in a specific country. Introducing a ban that specifically limits the export of artificial intelligence or machine learning software to a specific country or a broad number of countries would qualify if it was known to apply to LLM products.
- - An introduced bill specifically classifying artificial intelligence software or machine learning models such that they would newly qualify for existing export bans would qualify.
Resolution will be determined according to reporting from credible sources and keyword searches of GovTrack.us.
*Background Info*
Specific legislation limiting LLMs has not been introduced in the U.S. Congress as of now. However, the ongoing discussions and emerging concerns in various sectors, including healthcare, indicate a growing awareness and potential for legislative action in the near future.
Note: There is no formal definition for LLMs, though one article describes them as follows:
Typically, a large language model contains more than 100 billion parameters and is trained using advanced algorithms on a large corpus.
I feel like "introduced to Congress" is a very poor question to ask because of how much garbage gets introduced every year without going anywhere. I wish they'd phrase it as "got to a vote in one chamber" at the very least.
I hear you. However, you might also be interested in the signal being sent by this lower-bar question. I'd be interested in your higher-bar question, however.
If you submit a question with the additional constraint you suggest (or similar), I'll get mods to look at it quickly. (https://www.metaculus.com/questions/create/) You can borrow much of the language & resolution criteria from the looser, existing question if you like. (This new question would not appear in the ACX tournament, however.)
If you decide to submit, just let me know and link to it.
> noting that I briefly saw a number by accident that was something starting with a 1 while navigating other pages, which I did my best to ignore but one is never perfect.
This is admirably scrupulous honesty and more than a little charming.
"Apollo 1 exploded, but the remaining 10 would all have counted including Apollo 13." Maybe I'm being pedantic, but I don't understand what you're trying to say, here. The Apollo mission numbering scheme is confusing and was changed after "Apollo 1" fire, but he only manned missions that "approached" the moon were Apollo 8, 10-17. That's only 9. Apollo 4,5,6,7, and 9 would not have counted.
"But still, this 17% seems really low? Especially given it looks like they are giving Artemis II a very high success rate once it does launch? I’m going to say 22% at Metaculus."
> I am loathe to bet against the market when not getting market prices, and I don’t know what the market will price this at, but I am going to say 50% based on traders expecting this to happen so I don’t get too out of line. With the understanding that I can only sell, I want to be below whatever Manifold will say.
> The Manifold traders are smart. They have it at 45%. Which implies either I was looking at the wrong expectations, or they’re willing to flat out call BS on the traders and instead believe the Fed itself, which I love. So I’m going to sell to 41% and put in 40% at Metaculus.
I think you've conflated a bunch of things when making this forecast. January 2025 Fed funds futures are currently ~96.09 (https://www.cmegroup.com/markets/interest-rates/stirs/30-day-federal-fund.quotes.html#venue=globex) so the number of cuts expected is ~5.7. However, this isn't the same as the market median being below 4%. In fact, the market does give us more fine grained probabilities for the distribution of rates next December. (Unfortunately it's not exactly the market we want - it's on 3m SOFR out of Dec). However, looking at that market, and making some rough approximations, they are putting the odds of fed funds < 4% at ~48%, pretty much bang inline with Manifold.
I would expect a higher chance of unionization in law than you because public defenders are poorly paid and ideologically motivated in favor of unions. At current margins I think they would welcome AI as reducing their workload, but if it gets good enough to lead to job cuts they'd unionize against it in a heartbeat.
I wrote my predictions post up here (same form of blind then looking up you and manifold for adjusting).
The one place I end up strongly disagreeing with you is Netanyahu, who I put at over 80% of staying in power - he doesn't get replaced unless he resigns (which he almost certainly won't) or he gets a no confidence vote by September, which would require 5 defections out of 64 (plus Ganz going with it), and his coalition partners all know they'd lose most of their power in a new election (plus he's done well so far at holding them together).
Quoting myself from the question detail page: "Leaving the question in would reward participants who update on this news. And while it's true that participants are welcome to update until the snapshot, we want to avoid privileging updaters where possible, in line with the spirit of this competition."
I believe you're missing the question: "Will a member of the United States Congress introduce legislation limiting the use of LLMs in 2024?"
There's additional detail in the Resolution Criteria located here:
https://www.metaculus.com/questions/20780/legislation-limiting-llms-introduced-in-2024/
Note: Community Predictions will not be visible until Feb 1st, so you can visit this link without fear of seeing a forecast.
Also Note: You can click 'Filter' and scroll down to 'My Participation' to include, for example, only questions that you haven't predicted.
(Also Also Note: I work for Metaculus.)
Ah, yes! That would be the one. I knew I'd missed something but somehow couldn't figure out what. I'll try to get time to look at the wording and figure it out.
Below, language from the Resolution Criteria and Background Information sections, if helpful:
*Resolution Criteria*
This question resolves as Yes if, before January 1, 2025, a bill is introduced in either the US House or US Senate which would impose one of the following restrictions on large language models or products using large language models.
-Forbids their creation
-Sets limits on how they're trained, for example by limiting access to previously usable training data or by setting limits on the number of parameters they may be trained with.
Prevents their use for certain applications, such as interacting with customers, interfacing with other applications, or performing actions on the web.
-Restricts the ability of US citizens to use foreign LLM-based products or restricts US operated LLM products or businesses from being sold to foreign customers or entities.
- - The export restriction must be specific, and not universally applied to all exports or exports of a broad industry. For example, a blanket ban on exports to a specific country would not qualify, and neither would a blanket ban on allowing the purchase of US businesses by investors in a specific country. Introducing a ban that specifically limits the export of artificial intelligence or machine learning software to a specific country or a broad number of countries would qualify if it was known to apply to LLM products.
- - An introduced bill specifically classifying artificial intelligence software or machine learning models such that they would newly qualify for existing export bans would qualify.
Resolution will be determined according to reporting from credible sources and keyword searches of GovTrack.us.
*Background Info*
Specific legislation limiting LLMs has not been introduced in the U.S. Congress as of now. However, the ongoing discussions and emerging concerns in various sectors, including healthcare, indicate a growing awareness and potential for legislative action in the near future.
Note: There is no formal definition for LLMs, though one article describes them as follows:
Typically, a large language model contains more than 100 billion parameters and is trained using advanced algorithms on a large corpus.
I feel like "introduced to Congress" is a very poor question to ask because of how much garbage gets introduced every year without going anywhere. I wish they'd phrase it as "got to a vote in one chamber" at the very least.
I hear you. However, you might also be interested in the signal being sent by this lower-bar question. I'd be interested in your higher-bar question, however.
If you submit a question with the additional constraint you suggest (or similar), I'll get mods to look at it quickly. (https://www.metaculus.com/questions/create/) You can borrow much of the language & resolution criteria from the looser, existing question if you like. (This new question would not appear in the ACX tournament, however.)
If you decide to submit, just let me know and link to it.
Thank you. I can certainly create a question that's worded more strongly but I was hoping one would appear on the ACX contest instead.
> noting that I briefly saw a number by accident that was something starting with a 1 while navigating other pages, which I did my best to ignore but one is never perfect.
This is admirably scrupulous honesty and more than a little charming.
"Apollo 1 exploded, but the remaining 10 would all have counted including Apollo 13." Maybe I'm being pedantic, but I don't understand what you're trying to say, here. The Apollo mission numbering scheme is confusing and was changed after "Apollo 1" fire, but he only manned missions that "approached" the moon were Apollo 8, 10-17. That's only 9. Apollo 4,5,6,7, and 9 would not have counted.
"But still, this 17% seems really low? Especially given it looks like they are giving Artemis II a very high success rate once it does launch? I’m going to say 22% at Metaculus."
Excellent timing! NASA just today officially delayed Artemis II until September 2025. https://www.nasa.gov/news-release/nasa-shares-progress-toward-early-artemis-moon-missions-with-crew/
Yep! Oh well. That's what happens when you are blind. I've bought out and taken the loss.
Really fun post, I like the blind guess format and I'm really impressed by how often you guessed the same % as the market!
> I am loathe to bet against the market when not getting market prices, and I don’t know what the market will price this at, but I am going to say 50% based on traders expecting this to happen so I don’t get too out of line. With the understanding that I can only sell, I want to be below whatever Manifold will say.
> The Manifold traders are smart. They have it at 45%. Which implies either I was looking at the wrong expectations, or they’re willing to flat out call BS on the traders and instead believe the Fed itself, which I love. So I’m going to sell to 41% and put in 40% at Metaculus.
I think you've conflated a bunch of things when making this forecast. January 2025 Fed funds futures are currently ~96.09 (https://www.cmegroup.com/markets/interest-rates/stirs/30-day-federal-fund.quotes.html#venue=globex) so the number of cuts expected is ~5.7. However, this isn't the same as the market median being below 4%. In fact, the market does give us more fine grained probabilities for the distribution of rates next December. (Unfortunately it's not exactly the market we want - it's on 3m SOFR out of Dec). However, looking at that market, and making some rough approximations, they are putting the odds of fed funds < 4% at ~48%, pretty much bang inline with Manifold.
I would expect a higher chance of unionization in law than you because public defenders are poorly paid and ideologically motivated in favor of unions. At current margins I think they would welcome AI as reducing their workload, but if it gets good enough to lead to job cuts they'd unionize against it in a heartbeat.
I wrote my predictions post up here (same form of blind then looking up you and manifold for adjusting).
The one place I end up strongly disagreeing with you is Netanyahu, who I put at over 80% of staying in power - he doesn't get replaced unless he resigns (which he almost certainly won't) or he gets a no confidence vote by September, which would require 5 defections out of 64 (plus Ganz going with it), and his coalition partners all know they'd lose most of their power in a new election (plus he's done well so far at holding them together).
https://shakeddown.substack.com/p/predictions-for-2024
FYI the Metaculus question 'Will a crewed Artemis II flight approach the moon in 2024?' has been removed from the ACX 2024 contest: https://www.metaculus.com/questions/20760/artemis-ii-lunar-flyby-in-2024/ given NASA's update that they are no longer targeting 2024.
Quoting myself from the question detail page: "Leaving the question in would reward participants who update on this news. And while it's true that participants are welcome to update until the snapshot, we want to avoid privileging updaters where possible, in line with the spirit of this competition."
Yep, very happy to see that so I wasn't forced into a choice of breaking the no-update rule or effectively crippling my chances. Good decision.