Rendered at 10:01:13 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
visarga 7 minutes ago [-]
I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process, and treats intelligence as something that can be ported across domains.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains. Why does intelligence not simply apply across all domains?
I've been feeling some of the things this article puts some data behind. Thanks for sharing
bob1029 20 minutes ago [-]
I think the most promising recursive bootstrapping thing is using the current linear algebra blackboxes to find better ways to construct competitive symbolic models.
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
BatchJob 20 minutes ago [-]
LLMs do not qualify as "frontier AI". They dont even qualify as AI.
lordnacho 2 hours ago [-]
At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
rpozarickij 3 minutes ago [-]
> Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "... Where do you want to go from here?"
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of things going out of control.
kennywinker 1 hours ago [-]
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
skew-aberration 1 hours ago [-]
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
hereonout2 1 hours ago [-]
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
NitpickLawyer 46 minutes ago [-]
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
vasusai 46 minutes ago [-]
I think if an AI developed completely new fields of thought or science.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
skew-aberration 1 hours ago [-]
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
telesilla 1 hours ago [-]
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
spinningslate 53 minutes ago [-]
“To us” is pivotal there. Continuing GP’s thought experiment: what if the model that produced the output perceives that the human it was presented to offers no value in helping it learn further?
onion2k 57 minutes ago [-]
May I counter, what purpose does such a machine have to us, that can think beyond our needs?
We can think of questions we can't answer. It can answer them.
RandomLensman 55 minutes ago [-]
Maybe it can, maybe it cannot- comes down to the question.
ludston 2 hours ago [-]
At that point, the machines correctness doesn't need to be evaluated by humans, it just needs to provide a recipe for how to achieve some process.
andy_ppp 1 hours ago [-]
So predicting the next word given all humanity’s knowledge is surely going to max out at slightly less good (we probably can’t get perfect data) than the best human in any specific field. What test does the AI do to be able to understand it is improving? At some point it becomes impossible to know that the output is actually better right?
whatever1 56 minutes ago [-]
All planners start with executing just the next step. You do too. You might be planning long term but you execute just one set of things at the current moment.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
kmeh 40 minutes ago [-]
Why would deep NNs or LLMs not work based on our theoretical understanding? What theoretical understanding are you referring to?
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
chrisjj 28 minutes ago [-]
> All planners start with executing just the next step.
token != step.
Just you try executing a complex command one word at a time.
tux3 1 hours ago [-]
There's many fields where it's easier to find new problems than to solve them, and it's easier to check the solution once you have it.
They're doing RL on open problems these days, not just next token prediction.
onion2k 59 minutes ago [-]
slightly less good (we probably can’t get perfect data) than the best human in any specific field
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
andy_ppp 56 minutes ago [-]
We’re talking about run away self improvement of these systems, I’m not even convinced we’ve seen these systems invent a single new thing yet.
slopinthebag 1 hours ago [-]
I dunno if it would max out at slightly less good, I imagine it would max out around the distribution of it's data set, which could be significantly worse than top experts.
thelaxiankey 57 minutes ago [-]
Not necessarily; these learning things are super non convex and it's not clear to me "where" they max out
mattlondon 41 minutes ago [-]
These aren't just repeating what they've been taught. The "stochastic parrot" thing is an irritating common misconception I think.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
Tl:Dr - It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code or fix bugs, let alone people using it for actual research or whatever.
chrisjj 21 minutes ago [-]
> The "stochastic parrot" thing is an irritating common misconception I think.
You think wrong ( https://dl.acm.org/doi/10.1145/3442188.3445922 ) ... except about irritating. Yes, its irritating to people conned into believing next-token predictors are intelligent.
tucnak 1 hours ago [-]
> surely
Citation needed
andy_ppp 58 minutes ago [-]
Have you seen anything produced by LLMs that is better than the best humans?
OKRainbowKid 43 minutes ago [-]
How could I objectively judge that?
To me, it is not at all obvious that the "level" of the training set is an upper limit to the capabilities of an LLM.
Sure, the LLM hasn't been exposed to material more advanced than the most capable human domain expert have produced. However, it has seen and learned from a vast amount of information that this domain expert is completely unaware of. Why shouldn't the LLM be able to use that information to produce output that's beyond the capability of the domain expert?
NitpickLawyer 31 minutes ago [-]
Yup, the e2e proof of FLT (estimated effort of 5 years & 1M$ by the best human in the field) and a counter example for a millennium prize (similarly valued at 1m$.
petesergeant 35 minutes ago [-]
I hadn't seen anything three years ago produced by an LLM that looked better than the most mediocre humans. The argument here isn't about today's output, it's about the potential of LLMs to outperform humans. And it's far from clear that the architecture is bound this way, or that it only repeats stuff it's already heard.
fsflover 43 minutes ago [-]
Recent mathematical breakthroughs?
sinuhe69 1 hours ago [-]
What if automating AI R&D triggers an intelligence explosion?
No title editing please
fredoliveira 41 minutes ago [-]
I agree. Some title edits are clarifying, but this one isn't helpful.
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
58 minutes ago [-]
7777777phil 57 minutes ago [-]
Just thought of this piece [1] by Ramez Naam: Where’s the “intelligence explosion”?
Color me skeptical. LLMs seem to make writing code faster, so of course that means that people can iterate on ideas faster, but I have yet to see actual creative output from an LLM that wasn't coached into it or random juxtaposition.
aflinik 2 hours ago [-]
Can you give me some examples of an actual creative output from a human that wasn't coached into it or random juxtaposition?
RandomLensman 2 hours ago [-]
Was the creation of writing as such coached into humans or a random juxtaposition? Could all human inventions just be coached into humans (who coached?) or be random juxtapositions?
kennywinker 2 hours ago [-]
Special relativity?
skew-aberration 1 hours ago [-]
People are always using this example because of the "LLMs can't jump paper". Suffice to say - it's not that simple, and special relativity was definitely an incremental improvement to well-studied theory that was being developed by dozens of the leading physicists of the day.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains. Why does intelligence not simply apply across all domains?
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of things going out of control.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
We can think of questions we can't answer. It can answer them.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
token != step.
Just you try executing a complex command one word at a time.
They're doing RL on open problems these days, not just next token prediction.
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
Tl:Dr - It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code or fix bugs, let alone people using it for actual research or whatever.
You think wrong ( https://dl.acm.org/doi/10.1145/3442188.3445922 ) ... except about irritating. Yes, its irritating to people conned into believing next-token predictors are intelligent.
Citation needed
To me, it is not at all obvious that the "level" of the training set is an upper limit to the capabilities of an LLM.
Sure, the LLM hasn't been exposed to material more advanced than the most capable human domain expert have produced. However, it has seen and learned from a vast amount of information that this domain expert is completely unaware of. Why shouldn't the LLM be able to use that information to produce output that's beyond the capability of the domain expert?
No title editing please
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
[1] https://www.noahpinion.blog/p/wheres-the-intelligence-explos...
It’s like 10 monkeys trying to keep a human being a prisoner.
It’s just not going to work is it.
The entire meaning of being more intelligent is being able to outwit and out-think people less intelligent.
So do you also believe that Kim Jong Un is the most intelligent man in North Korea?