What is the mysterious impressive new ‘gpt2-chatbot’ from the Arena? Is it GPT-4.5? A refinement of GPT-4? A variation on GPT-2 somehow? A new architecture? Q-star? Someone else’s model? Could be anything. It is so weird that this is how someone chose to present that model.
Thank you for sharing on Pause AI! if interested in helping us make a difference in something almost 80% of Americans agree needs regulation, please join us here!
Hey Zvi! Thanks a lot for mentioning the Pause AI Protests! I am actually using a Partiful Invite for organising the Paris protest rather than the facebook page: https://partiful.com/e/3Tl1xrS6i9NUZxyJGf5G
>“Every college student should learn to train a GPT-2… not the most important thing but I bet in 2 years that’s something every Harvard freshman will have to do”
This used to be called "writing in a diary" back when people did their own thinking.
I used gpt2-chatbot for relatively demanding questions about current regulatory developments in the EU in the domain of tax, it's answer was very good, I am pretty sure there was some orchestration and CoT-style re-prompting going on to get that level and formatting of output. I would not be surprised if it was a beta 4.5.
"Brevity is also how LLMs often do not work. Ask a simple question, get a wall of text. Get all the ‘this is a complex issue’ caveats Churchill warned us to avoid."
I am about 80% that this is RHLF and that in the not too distant future we will all just stop doing that, at least for models that are not totally public facing (eg corporate models), because with GPT 5 or 6 the performance penalties to RHLF will grow to be so significant as to make it untenable.
I don't have a mathematical model for it, but my theory is that 1) RHLF is inherently degrading prediction accuracy, and 2) the better the underlying model's accuracy, the greater one has to degrade it to get to the same RHLF output, because a more accurate ("smarter") model will be less easily convinced to produce tokens that are "uphill"/remote of the (appropriately weighted etc) last N tokens ("lie", although relative to its training corpus not to any objective reality).
To generalize from the point about whether today's models are good or bad, everything we have today is good and bad. It's good compared to what came before, and it's bad compared to what will come after. It seems difficult for many people to zoom out to this view. We should appreciate what we have, and we should also realize that it will look terrible in some ways decades from now. That's progress.
And this is especially true for technology - AI, energy generation, etc.
I made the mistake of dismissing mobile phone cameras around 2005 because they were terrible at the time. I completely failed to see the trajectory in terms of technological development, economies of scale and learning effects, and so on. Always try to think a couple of generations ahead.
But we also need to consider the probability that the next generation of something will never come due to physical constraints, or consensus among engineers and investors on no chance of ROI. And also the probability that the next generation will come too far in the future to be worth thinking about.
For example, have zippers on clothing and bags sold in developed countries gotten better over the past few decades? I don't think so. I imagine a zipper problem annoys the majority of readers here at least once a year. I would have predicted decades ago that improvements would occur, and I would have been wrong.
The zipper example tells us essentially nothing about AI. I was looking for something familiar, and I had much less confidence that cheap printers haven't improved (even though that is my emotional evaluation of them, driven by frustration).
> This sometimes goes as low as the 10^23 flops threshold, which covers many existing models.
I believe that 10^23 is actually a significant threshold. Not now, but in the future.
Several decades ago, Moravec tried to compare:
- the neural mass required run simple, well-understood visual algorithms like edge detection in our visual system
with
- the number of FLOPS needed to run those same algorithms in silicon.
He then extrapolated these data points to the neural mass of the entire brain, and got an estimate that said the human brain might be performing about ~~10^15 FLOPS of actual work. (It would take far, far more than this to simulate the brain, so this estimate assumes rampant inefficiency in translating neurons into FLOPS-equivalents.)
If we assume it takes about 20 years to fully train a human, that would be about 6.3*10^8 seconds.
Combined, this suggests you could train an adult human using the equivalent of 6.3*10^23 operations, assuming your starting point is as good as human DNA's starting point. So if we assume that future algorithmic improvements and better sample efficiency get us into the same range as humans, then 10^23 might actually be a significant threshold.
Now, if we ever _beat_ human sample efficiency, well, good luck trying to have any real control over human extinction.
Thank you for sharing on Pause AI! if interested in helping us make a difference in something almost 80% of Americans agree needs regulation, please join us here!
https://discord.com/invite/5sYaE9aE
Hey Zvi! Thanks a lot for mentioning the Pause AI Protests! I am actually using a Partiful Invite for organising the Paris protest rather than the facebook page: https://partiful.com/e/3Tl1xrS6i9NUZxyJGf5G
>“Every college student should learn to train a GPT-2… not the most important thing but I bet in 2 years that’s something every Harvard freshman will have to do”
This used to be called "writing in a diary" back when people did their own thinking.
I used gpt2-chatbot for relatively demanding questions about current regulatory developments in the EU in the domain of tax, it's answer was very good, I am pretty sure there was some orchestration and CoT-style re-prompting going on to get that level and formatting of output. I would not be surprised if it was a beta 4.5.
"Brevity is also how LLMs often do not work. Ask a simple question, get a wall of text. Get all the ‘this is a complex issue’ caveats Churchill warned us to avoid."
I am about 80% that this is RHLF and that in the not too distant future we will all just stop doing that, at least for models that are not totally public facing (eg corporate models), because with GPT 5 or 6 the performance penalties to RHLF will grow to be so significant as to make it untenable.
Eh, before too long they’ll train LLM’s to do the RHLF for us
Yes but the resulting model will still be worse for it
> with GPT 5 or 6 the performance penalties to RHLF will grow to be so significant as to make it untenable.
Are there some figures somewhere that could help me get a better grasp of this? I hadn't read anything about it before. Thanks.
I don't have a mathematical model for it, but my theory is that 1) RHLF is inherently degrading prediction accuracy, and 2) the better the underlying model's accuracy, the greater one has to degrade it to get to the same RHLF output, because a more accurate ("smarter") model will be less easily convinced to produce tokens that are "uphill"/remote of the (appropriately weighted etc) last N tokens ("lie", although relative to its training corpus not to any objective reality).
To generalize from the point about whether today's models are good or bad, everything we have today is good and bad. It's good compared to what came before, and it's bad compared to what will come after. It seems difficult for many people to zoom out to this view. We should appreciate what we have, and we should also realize that it will look terrible in some ways decades from now. That's progress.
And this is especially true for technology - AI, energy generation, etc.
I made the mistake of dismissing mobile phone cameras around 2005 because they were terrible at the time. I completely failed to see the trajectory in terms of technological development, economies of scale and learning effects, and so on. Always try to think a couple of generations ahead.
But we also need to consider the probability that the next generation of something will never come due to physical constraints, or consensus among engineers and investors on no chance of ROI. And also the probability that the next generation will come too far in the future to be worth thinking about.
For example, have zippers on clothing and bags sold in developed countries gotten better over the past few decades? I don't think so. I imagine a zipper problem annoys the majority of readers here at least once a year. I would have predicted decades ago that improvements would occur, and I would have been wrong.
The zipper example tells us essentially nothing about AI. I was looking for something familiar, and I had much less confidence that cheap printers haven't improved (even though that is my emotional evaluation of them, driven by frustration).
> This sometimes goes as low as the 10^23 flops threshold, which covers many existing models.
I believe that 10^23 is actually a significant threshold. Not now, but in the future.
Several decades ago, Moravec tried to compare:
- the neural mass required run simple, well-understood visual algorithms like edge detection in our visual system
with
- the number of FLOPS needed to run those same algorithms in silicon.
He then extrapolated these data points to the neural mass of the entire brain, and got an estimate that said the human brain might be performing about ~~10^15 FLOPS of actual work. (It would take far, far more than this to simulate the brain, so this estimate assumes rampant inefficiency in translating neurons into FLOPS-equivalents.)
If we assume it takes about 20 years to fully train a human, that would be about 6.3*10^8 seconds.
Combined, this suggests you could train an adult human using the equivalent of 6.3*10^23 operations, assuming your starting point is as good as human DNA's starting point. So if we assume that future algorithmic improvements and better sample efficiency get us into the same range as humans, then 10^23 might actually be a significant threshold.
Now, if we ever _beat_ human sample efficiency, well, good luck trying to have any real control over human extinction.
> The talk delves into a world of very different concerns
> delves
8-|
:-)