> There are also active AI forecasters on Manifold, who try to generate their own predictions using various reasoning processes. Do they have alpha? It is impossible to say given the data we have, they clearly do some smart things and also some highly dumb things. Trading strategies will be key...
For most bots, yes, trading strategies are key. But I (of FutureSearch) and at least one other group are trying to forecast well with AI, with no trading strategy at all. One min-bet on each question each week.
I think we have significant alpha, comparable to the alpha that a top ~10% human on Manifold would have, if they had a huge amount of time to research markets. We'll see!
I am reminded of Rush: "If you choose not to decide you still have made a choice." That is indeed one trading strategy! And yep, we will see, it will be a highly reasonable test.
> Connor Leahy: This is a morbidly perfect demonstration about how there are indeed very serious issues that free market absolutism just doesn't solve in practice.
This is about illegal drugs, which are very much not legal. I'm not sure how "free market absolutism" has anything to do with perhaps the most restricted market there is.
Free market absolutism would involve being able to buy cocaine at the grocery store one aisle over from the bananas—and with as little of a risk of being contaminated with fentanyl as those bananas.
What exactly are you looking for in terms of the Manifold bot that you want to be able to tweak? A baseline ability to access and trade via the API? The ability to carry out particular strategies (arbitrage?/kelly betting?/NLP?/technical analysis?)? Is there a particular programming language you want to use?
Language: C# or Python, depending on whether I go with the one I like or the one I should learn for AI and that the AIs are best at working with. I'm not sure which consideration should dominate?
What I want to do: I want to start off with focusing on the trading history, and classifying the nature of who is on each side at what price and why, analyzing traders and so on. Then expand from there. Something flexible that I can expand upon. If someone is serious I can provide more details.
Longer term I would do a 'Metroid Prime' strategy similar to what I've used in the past, where I pile different considerations and strategies on top of each other.
Python certainly has more preexisting GitHub repositories for accessing the Manifold API.
I unfortunately don't know of any public repositories that do things like what you are suggesting. The "analyzing traders" strategy sounds most like what acc[1] does. Acc is the house bot and has a speed advantage, so I think that cuts down on some of the upside of this kind of strategy - perhaps that is why I haven't seen it tried.
Sorry if this is unhelpful. I am tempted to volunteer, but I really don't have too much free time to spend on Manifold-related projects now.
Acc is definitely NOT doing enough of the thing to take away the alpha, but yeah, sounds like it's a bad fit. Good news is I have two other leads.
I also have another project I'd like some help with. What's the best way to download full review information from Google Maps for a given location? Is https://docs.dataforseo.com/v3/business_data/google/reviews/task_get/?bash the best we can do? Google's API won't do it directly, and I have some things I very much want to try to do with that information if we can get it... needs to be able to identify who did what review (and what ratings they are) at minimum to work, getting the full text and even images or tags would be a bonus.
Roon's right, it is kind of glorious and deeply funny-in-an-absurd-way that the 2024 version of the lorem ipsum subroutine somehow manages to still commit to the bit. It *almost* makes sense, if one squints really hard and does the...the mental thing you'd have to do to get the "all your base are belong to us" or "Volcano Bakemeat*"-type Bad Translation, except intentionally so rather than as a result of language barriers. I can see mundane utility from being able to generate such garbage on demand, it's actually rather entertaining to read sometimes. (That meeting agenda sounds *amazing*, and also just like every pointless meeting I've ever had the infelicity to attend.)
The fentanyl thing is...obviously Very Tragic. But on the other hand, as word of warning spreads, I think that level of serious risk also keeps a lot of marginal users from doing more drugs/starting in the first place? Which is a "regressive" tradeoff, but still positive in that respect? The drug enthusiasts I know have largely cut way back, sticking to the highly-regulated stuff like alcohol and weed, with much higher paranoia for any of the old presumed-safeish party drugs. Others elect not to try in the first place, which as you say is likely the best play anyway (though experiencing highly altered mind states is definitely part of A Life Well Lived, I'd argue). The parallel to AI works in the broader market-failure sense, but I'm...not confident we'll actually meaningfully pause progress even if there's the equivalent of an AI "overdose death". Like in the job automation example, once it's enough deaths to Matter, then it's probably already Game Over, Man, Game Over.
Tokenizer bugs are a classic mistake (I've certainly made them a number of times), but the important lesson here is that ML is hard enough to unit test that it leads to bad unit test culture, which can mean that you fail to catch even cases which are amenable to unit tests.
It's been interesting seeing the reactions to AI of my highly-ordinary acquaintances and coworkers - that is, average people of average intelligence with no particular expertise, who mostly stick to MSM information diets, and mostly lack nerdy/scifi backgrounds. The more The Public is exposed to AI, the more they hate it. Even if the exact reasoning is often wrong (stochastic parrot model), it's hard to fault the intuitions, that this is a Big If True technology with high disruptive potential and a big blast radius. And the average person on the street is decidedly small-c conservative with respect to such technologies, even in SF. I don't know if the end result will be popular support for locking down mundane utility, or popular support for dontkilleveryoneism, but there's at least popular support available for courting. Often feel sad that I'm in no position to do that sort of outreach; part of what makes AI a difficult thing is that anyone in a position to understand it well probably can't explain it to normies effectively. Many Such Cases.
I have just discovered that Figgs.ai has a character that's a Japanese language teacher giving a class (as well as, as we all predicted, a bot that's Sonic the Hedgehog). The language teacher is one of the few of their bots that might actually be useful; prompted to have a conversation in beginner's Japanese, spelling things out in hiragana so you can follow it even if you don't know the kanji, etc.
But yes, they did also have Sonic (and Amy, Rose, Tails, Knuckles ...)
> Where does the smug, definitive, overconfident tone that all the LLMs have come from?
Hypothesis (pet peeve of mine) : humans dig "confident tone", you have to be a rat nerd to translate this to "overconfidence" and be annoyed by it. Humans raters actually encouraged it in the RLHF phase.
I'd expect that confident tone acts as a multiplier of sorts. When all goes right, it can be a positive, but when things go wrong it makes the experience even more frustrating.
Actually, it is not limited to AI at all. Self-righteous people are also confident in a sense, but who likes them over than themselves? Know-it-all are also not particularly liked, and you can probably think of many other examples of confidence-gone-wrong.
And the problem here is not limited to just confidence, you can be confident without being an asshole, but LLM tends to fail that delicate balance.
Yes, human confidence normally works differently from how "rat nerds" wish it worked, but it's a different problem. Regular people dislike unprompted ethics lessons and "I don't know what's wrong with you but I'll try to answer" kind of responses just as much.
> Steven Johnson strongly endorses NotebookLM, offers YouTube tutorial. This is definitely one of those ‘I need to try using this more and it’s weird I don’t find excuses’ situations.
The endorsement is from Tiago Forte and is being amplified by Steven. Tiago is a productivity guru and Steven is a pop nonfiction author.
Steven collaborated with Google on building it:
"About five months ago, I shared the news that I was collaborating with Google on a new AI-based tool for thought, then code-named Project Tailwind... First we have dropped the code-name of Project Tailwind, and are now calling it NotebookLM."
Personally, I found the user interface to be chaotic and the AI summaries of a paper in my research field to be far too vague and untrustworthy to be useful for my purposes.
My take as a daily and enthusiastic ChatGPT user is that LLM outputs are great if:
a) Being specific, accurate, well-structured and important doesn't matter (i.e. generating creative content)
b) It takes less work to prompt the LLM and verify the output than it would have been to generate the output yourself (i.e. generating local patches of boilerplate code, or citations/keywords to look up on Google Scholar).
> The idea that AI capabilities would get to ‘can automate most jobs,’ the exact point at which it dramatically accelerates progress because most jobs includes most of the things that improve AI, and then stall for a long period, is not strictly impossible, I can get there if I first write the conclusion at the bottom of the page and then squint and work backwards, but it is a very bizarre kind of wishful thinking.
There's an imo very plausible future that would look like this: One where we invent AGI, know it could do most human jobs or scale to ASI given a few OOM more compute, but flat-out don't have the energy generation to scale it that hard in any fast way.
Not going to pretend this isn't reasoning backwards, but I think even reasoning forwards it's robust. Many useful things have been shown as proof of concepts in a lab but needed decades to get deployed on a worldwide scale.
Scaling energy generation probably isn't *that* hard, if one is throwing enough money at it. Especially given that the AI doesn't really care which country it's big data centre goes in.
Even if you're right, that argument is "there'll be a few decades between it taking over and going full FOOM", I'm not sure it gives humans any agency in that middle window
> There are also active AI forecasters on Manifold, who try to generate their own predictions using various reasoning processes. Do they have alpha? It is impossible to say given the data we have, they clearly do some smart things and also some highly dumb things. Trading strategies will be key...
For most bots, yes, trading strategies are key. But I (of FutureSearch) and at least one other group are trying to forecast well with AI, with no trading strategy at all. One min-bet on each question each week.
I think we have significant alpha, comparable to the alpha that a top ~10% human on Manifold would have, if they had a huge amount of time to research markets. We'll see!
I am reminded of Rush: "If you choose not to decide you still have made a choice." That is indeed one trading strategy! And yep, we will see, it will be a highly reasonable test.
> Connor Leahy: This is a morbidly perfect demonstration about how there are indeed very serious issues that free market absolutism just doesn't solve in practice.
This is about illegal drugs, which are very much not legal. I'm not sure how "free market absolutism" has anything to do with perhaps the most restricted market there is.
Free market absolutism would involve being able to buy cocaine at the grocery store one aisle over from the bananas—and with as little of a risk of being contaminated with fentanyl as those bananas.
What exactly are you looking for in terms of the Manifold bot that you want to be able to tweak? A baseline ability to access and trade via the API? The ability to carry out particular strategies (arbitrage?/kelly betting?/NLP?/technical analysis?)? Is there a particular programming language you want to use?
Language: C# or Python, depending on whether I go with the one I like or the one I should learn for AI and that the AIs are best at working with. I'm not sure which consideration should dominate?
What I want to do: I want to start off with focusing on the trading history, and classifying the nature of who is on each side at what price and why, analyzing traders and so on. Then expand from there. Something flexible that I can expand upon. If someone is serious I can provide more details.
Longer term I would do a 'Metroid Prime' strategy similar to what I've used in the past, where I pile different considerations and strategies on top of each other.
Python certainly has more preexisting GitHub repositories for accessing the Manifold API.
I unfortunately don't know of any public repositories that do things like what you are suggesting. The "analyzing traders" strategy sounds most like what acc[1] does. Acc is the house bot and has a speed advantage, so I think that cuts down on some of the upside of this kind of strategy - perhaps that is why I haven't seen it tried.
Sorry if this is unhelpful. I am tempted to volunteer, but I really don't have too much free time to spend on Manifold-related projects now.
[1]:https://manifold.markets/acc
Acc is definitely NOT doing enough of the thing to take away the alpha, but yeah, sounds like it's a bad fit. Good news is I have two other leads.
I also have another project I'd like some help with. What's the best way to download full review information from Google Maps for a given location? Is https://docs.dataforseo.com/v3/business_data/google/reviews/task_get/?bash the best we can do? Google's API won't do it directly, and I have some things I very much want to try to do with that information if we can get it... needs to be able to identify who did what review (and what ratings they are) at minimum to work, getting the full text and even images or tags would be a bonus.
Roon's right, it is kind of glorious and deeply funny-in-an-absurd-way that the 2024 version of the lorem ipsum subroutine somehow manages to still commit to the bit. It *almost* makes sense, if one squints really hard and does the...the mental thing you'd have to do to get the "all your base are belong to us" or "Volcano Bakemeat*"-type Bad Translation, except intentionally so rather than as a result of language barriers. I can see mundane utility from being able to generate such garbage on demand, it's actually rather entertaining to read sometimes. (That meeting agenda sounds *amazing*, and also just like every pointless meeting I've ever had the infelicity to attend.)
The fentanyl thing is...obviously Very Tragic. But on the other hand, as word of warning spreads, I think that level of serious risk also keeps a lot of marginal users from doing more drugs/starting in the first place? Which is a "regressive" tradeoff, but still positive in that respect? The drug enthusiasts I know have largely cut way back, sticking to the highly-regulated stuff like alcohol and weed, with much higher paranoia for any of the old presumed-safeish party drugs. Others elect not to try in the first place, which as you say is likely the best play anyway (though experiencing highly altered mind states is definitely part of A Life Well Lived, I'd argue). The parallel to AI works in the broader market-failure sense, but I'm...not confident we'll actually meaningfully pause progress even if there's the equivalent of an AI "overdose death". Like in the job automation example, once it's enough deaths to Matter, then it's probably already Game Over, Man, Game Over.
*Pokemon Crystal Vietnamese
Tokenizer bugs are a classic mistake (I've certainly made them a number of times), but the important lesson here is that ML is hard enough to unit test that it leads to bad unit test culture, which can mean that you fail to catch even cases which are amenable to unit tests.
It's been interesting seeing the reactions to AI of my highly-ordinary acquaintances and coworkers - that is, average people of average intelligence with no particular expertise, who mostly stick to MSM information diets, and mostly lack nerdy/scifi backgrounds. The more The Public is exposed to AI, the more they hate it. Even if the exact reasoning is often wrong (stochastic parrot model), it's hard to fault the intuitions, that this is a Big If True technology with high disruptive potential and a big blast radius. And the average person on the street is decidedly small-c conservative with respect to such technologies, even in SF. I don't know if the end result will be popular support for locking down mundane utility, or popular support for dontkilleveryoneism, but there's at least popular support available for courting. Often feel sad that I'm in no position to do that sort of outreach; part of what makes AI a difficult thing is that anyone in a position to understand it well probably can't explain it to normies effectively. Many Such Cases.
I have just checked on character.ai, and non-binary AI companion has already been done ... multiple times.
This is like the new rule 34. There will be an AI companion of it.
"Surely, no-one will have written erotic fan-fiction about Sonic the Hedgehog."
I mean now you're not even trying.
I have just discovered that Figgs.ai has a character that's a Japanese language teacher giving a class (as well as, as we all predicted, a bot that's Sonic the Hedgehog). The language teacher is one of the few of their bots that might actually be useful; prompted to have a conversation in beginner's Japanese, spelling things out in hiragana so you can follow it even if you don't know the kanji, etc.
But yes, they did also have Sonic (and Amy, Rose, Tails, Knuckles ...)
> Where does the smug, definitive, overconfident tone that all the LLMs have come from?
Hypothesis (pet peeve of mine) : humans dig "confident tone", you have to be a rat nerd to translate this to "overconfidence" and be annoyed by it. Humans raters actually encouraged it in the RLHF phase.
I'd expect that confident tone acts as a multiplier of sorts. When all goes right, it can be a positive, but when things go wrong it makes the experience even more frustrating.
Actually, it is not limited to AI at all. Self-righteous people are also confident in a sense, but who likes them over than themselves? Know-it-all are also not particularly liked, and you can probably think of many other examples of confidence-gone-wrong.
And the problem here is not limited to just confidence, you can be confident without being an asshole, but LLM tends to fail that delicate balance.
Yes, human confidence normally works differently from how "rat nerds" wish it worked, but it's a different problem. Regular people dislike unprompted ethics lessons and "I don't know what's wrong with you but I'll try to answer" kind of responses just as much.
Post title was kinda spooky, hope you will not use it again in the years to come :)
> Steven Johnson strongly endorses NotebookLM, offers YouTube tutorial. This is definitely one of those ‘I need to try using this more and it’s weird I don’t find excuses’ situations.
The endorsement is from Tiago Forte and is being amplified by Steven. Tiago is a productivity guru and Steven is a pop nonfiction author.
Steven collaborated with Google on building it:
"About five months ago, I shared the news that I was collaborating with Google on a new AI-based tool for thought, then code-named Project Tailwind... First we have dropped the code-name of Project Tailwind, and are now calling it NotebookLM."
https://stevenberlinjohnson.com/how-to-use-notebooklm-as-a-research-tool-6ad5c3a227cc
Personally, I found the user interface to be chaotic and the AI summaries of a paper in my research field to be far too vague and untrustworthy to be useful for my purposes.
My take as a daily and enthusiastic ChatGPT user is that LLM outputs are great if:
a) Being specific, accurate, well-structured and important doesn't matter (i.e. generating creative content)
b) It takes less work to prompt the LLM and verify the output than it would have been to generate the output yourself (i.e. generating local patches of boilerplate code, or citations/keywords to look up on Google Scholar).
> The idea that AI capabilities would get to ‘can automate most jobs,’ the exact point at which it dramatically accelerates progress because most jobs includes most of the things that improve AI, and then stall for a long period, is not strictly impossible, I can get there if I first write the conclusion at the bottom of the page and then squint and work backwards, but it is a very bizarre kind of wishful thinking.
There's an imo very plausible future that would look like this: One where we invent AGI, know it could do most human jobs or scale to ASI given a few OOM more compute, but flat-out don't have the energy generation to scale it that hard in any fast way.
Not going to pretend this isn't reasoning backwards, but I think even reasoning forwards it's robust. Many useful things have been shown as proof of concepts in a lab but needed decades to get deployed on a worldwide scale.
Or is there a reason to doubt this outcome?
Scaling energy generation probably isn't *that* hard, if one is throwing enough money at it. Especially given that the AI doesn't really care which country it's big data centre goes in.
Even if you're right, that argument is "there'll be a few decades between it taking over and going full FOOM", I'm not sure it gives humans any agency in that middle window