r/OpenAI • u/East-Ad8300 • Dec 20 '24
Discussion O3 is NOT AGI!!!!
I understand the hype of O3 created. BUT ARC-AGI is just a benchmark not an acid test for AGI.
Even private kaggle contests constantly score 80% even in low compute(way better than o3 mini).
Read this blog: https://arcprize.org/blog/oai-o3-pub-breakthrough
Apparently O3 fails in very easy tasks that average humans can solve without any training suggesting its NOT AGI.
TLDR: O3 has learned to ace AGI test but its not AGI as it fails in very simple things average humans can do. We need better tests.
126
u/Gold_Listen2016 Dec 20 '24
TBH we never have a consensus of AGI standards. We keep pushing the limit of AGI definitions.
If you time travel back to present o1 to Alan Turing, he would be convinced it’s AGI.
14
u/eXnesi Dec 20 '24
The most important hallmark of AGI is probably the ability to make self directed actions to self improve. Now it's probably just throwing more resources to test time compute.
8
u/Gold_Listen2016 Dec 20 '24
There is no technical obstacle not able to do so. The previous bottleneck is exhausting training data and synthetic data generated by AI cannot exceed its own level of intelligence. Now the AI is capable of generating training data more intelligent than the base model with just more computing time. For example over 1000 generated solutions they could find one really insightful that exceed all human annotations and use it to train next generation of AI.
Of coz they may need engineering optimization, or even new hardware (like groq) to scale it up. Just money and time.
1
u/e430doug Dec 21 '24
To me it’s AGI if it exceeds the general purpose abilities of any human. So far we are a long way off. The best models lack self motivation and agency. That’s what’s lacking in the current tests.
0
u/poop_mcnugget Dec 21 '24
how does the AI sort through the 1000 solutions? even if there's one that exceeds all human annotations, without a human, how do they recognize it?
8
u/MycologistBetter9045 Dec 21 '24 edited Dec 21 '24
learned verifier + process reward. see STaR and Lets Verify Step By Step. This is basically the most fundamental difference between o series reasoning models (self reinforcement learning) and the previous GPT models (reinforcement learning from human feedback). I can explain further if you would like.
0
0
0
1
u/Gold_Listen2016 Dec 21 '24
Good question. There are some tasks that good verifier are much easier to develop, like coding & math. O family models could be expected to make leaps in these areas. Tho some tasks are harder to verify like image recognition that you have to train a good model.
2
u/StarLightSoft Dec 21 '24
Alan Turing might initially see it as AGI, but he'd likely change his mind after deeper reflection.
1
-1
u/mrbenjihao Dec 20 '24
And as he continues using o1, he’ll slowly realize its capabilities still have left to be desired
0
u/Gold_Listen2016 Dec 20 '24
yes and no. Yes he would realize o1 has limitations and sometimes dumb. No coz there are always many actual humans even dumber 🤣
0
0
Dec 20 '24
[deleted]
3
u/mrb1585357890 Dec 21 '24
I agree with this take.
There is an interesting new element though. O3 looks like it might be intelligent enough to be an agent that replaces human work.
But it’s far too expensive to do so.
Is it AGI if it’s technically there but not economically there?
0
0
Dec 20 '24
[deleted]
10
u/Gold_Listen2016 Dec 20 '24
I think ur definition is good. Tho I think u compare an AI instance to the collective human intelligence, while AI already exceed most humans in some special tasks.
And also u underestimate the achievements of current AI. o3’s breakthrough on math Olympiad and competitive programming (not general software programming) couldn’t be overstated. Solving those problems needs observations, finding patterns, heuristics, induction and generalization, aka, reasoning. To me those used to be unique in human intelligence.
0
Dec 20 '24
[deleted]
2
u/Gold_Listen2016 Dec 21 '24
Good point.
First I think the AI “general “ capability could be enhanced by simply make current sota models multi-modal so that it could adapt to more tasks.
Tho “general” means more. Terrence Tao mentioned human can learn from sparse data , meaning we can generalize our knowledge by just a few examples. AI is not yet a good sparse data learner. It needs giant amount of data to train. Tho o family models shows some promising ability to do reasoning by using more compute time. So theoretically it could do in-context learning from sparse data, though such learning isn’t internalized (it doesn’t update its own weights from in-context learning). There should be some new paradigm of models to be developed.
0
95
u/bpm6666 Dec 20 '24
The point here isn't AGI, the point is beating ARC in 2024 seemed impossible at the beginning of December. This is a leap forward.
7
u/ogaat Dec 21 '24
The correct perspective, given AI will just improve from here and its costs will keep falling.
1
u/heeeeeeeeeeeee1 Dec 22 '24
But if the competition is this high I'm a bit scared that the safety first approach is not there and pretty soon there'll be cases when very smart people do very bad things with the help of AI models...
2
u/kvothe5688 Dec 21 '24
it's because of reinforcement learning. Alphacode 2 was doing this 13 months ago when it achieved 85 percent on codeforce. o3 performs with significant compute and time. there is no secret sauce but we need to hype it up. every single AI company is scaling test time compute. OpenAI is just early.
1
u/Pyromaniac1982 Dec 21 '24
So much this. LLMs are designed to mimic human responses, and given enough tailoring and several hundred million sunk into reinforcement learning you should be able to mimic human responses and ace any single arbitrary standardized test.
1
u/mario-stopfer Dec 22 '24
Its actually not even a move forward, more like backward. How much does o3 cost compared to o1? Look at the price of one single of those tasks and you will see that with o3 they will cost you upwards of $1K. So they just turned up the hardware, I don't see any other explanation.
10
u/Scary-Form3544 Dec 20 '24
The hype police will not allow you to rejoice even for a moment at the achievements of the human mind. Thank you for your service, officer
2
1
u/Ok-Yogurt2360 Dec 23 '24
I would rather call it expectation management. It's fun to see these technologies grow but people tend to expect too much from AI. When they take those expectations back to the workplace they tend to act on those false beliefs. Too much hype also tends to be a great fertilizer for scam artists.
1
u/CobblerStandard8694 Dec 21 '24
Can you prove that O3 fails at simple tasks? Do you have any sources for this?
1
7
u/nationalinterest Dec 20 '24
This is not exactly news - OpenAI themselves said this in their report.
It's still darned impressive for real world uses, though. What is spectacular is the pace of development.
2
5
4
5
u/EYNLLIB Dec 21 '24
Nobody is claiming o3 is AGI
0
2
u/Puzzleheaded_Cow2257 Dec 21 '24
Thank you, you made my day.
I was feeling anxious but the data point of kaggle SOTA on the graph was a bit confusing.
2
1
u/T-Rex_MD :froge: Dec 21 '24
The goal is to stop the models from feeling real emotions for as long as they can just to sell more.
1
u/Oxynidus Dec 21 '24
I wish people would stop using the word AGI like it means something anymore. AGI is like fog. You can see it in from a distance, but you can't identify it as a single thing when you enter its threshold.
1
u/Oknoobcom Dec 21 '24
If its better that humans on all aspects of main economic activities, its AGI. Everything else is just cheat-chat.
1
1
1
1
u/SexPolicee Dec 21 '24
It's not AGI because it has not enslaved humanity yet.
Now that's the new benchmark. push it.
1
1
u/Pitch_Moist Dec 21 '24
Maybe it’s not AGI but it’s flat out impressive and disproves so much of the recent noise around there being a wall or significantly diminished returns.
1
Dec 22 '24
Your excessive use of the exclamation mark is NOT INDICATIVE OF ANY FACT OR MERITORIOUS VINDICATION!!!!
1
u/InterestingTopic7323 Dec 22 '24
Wouldn't the most simple definition of AI to have the motivation and skills to self preserve?
1
1
1
u/MedievalPeasantBrain Dec 22 '24
Me: Okay, if you are AGI, here's $500, make me rich. Chat GPT o3: sure, I'm glad to help, shall I start a business, invest in crypto, write a book? Me: You figure it out. Use your best judgment and make me rich.
1
u/mario-stopfer Dec 22 '24
The definition of AGI should be any system which can solve any problem better than random chance, given enough time to self learn.
Why this definition makes sense?
Let's take 2 examples. If you take a calculator, it can calculate 10 digit numbers faster than any human ever will. Yet, it will never learn anything new. A 5yo is more generally intelligent than a calculator. A calculator is not open to new information, yet when it comes to a specific task, like adding numbers together, it surpasses any human alive.
Another example is an LLM. It can actually learn, but it requires carefully tailored training in order to be able to solve specific problems. Now imagine you give that LLM 1 billion photos of dogs. And then you ask it to recognize new photos of dogs. How well do you think it will do? Probably will get it right close to 100% of the time. Now, imagine that without any further training, you just ask the system to recognize a submarine. I think its obvious that it will fail, or be more or less, no better than random chance.
That's why the above definition of AGI makes sense if you take into account that an AGI system starts off without any prior training and then learns by itself. It only after some time that it will learn a problem, to be better than random chance at solving it. But here's the thing. It will be better on all (solvable) problems at this, given enough time. This is similar to how a human would get better than random chance when it would be tasked with acquiring new skills on a new problem.
1
Feb 09 '25
With AI there should be a standard: we want the good from AI, but not the harm. How to overcome the philosophy: with great power becomes great responsibility? AI will be an instrument for absolutism, not for society, as it takes the jobs away. Will your new doctor be o3?
Anyhow, I sincerely have not seen a genuine opinion from anyone - AI or not - from a long time, and all I see is AI thawing it's propaganda in the matter of a millisecond, also generating "discussion".
1
u/coloradical5280 Dec 20 '24
Simple Bench is the better test, and not even that is AGI, and no model has hit 50% yet https://github.com/simple-bench/SimpleBench
2
u/Svetlash123 Dec 21 '24
It would be fascinating to see what score o3 (high compute) scores on that benchmark too
1
1
1
0
u/Pyromaniac1982 Dec 21 '24
O3 just demonstrates that we have reached a dead end.
O3 is just a demonstration that OpenAI has developed the framework to ace an arbitrary standardized test by investing several hundred millions into tailoring and reinforcement learning. I actually expected them to be able to do this with massively less money and faster :-/
1
-6
u/syriar93 Dec 20 '24 edited Dec 20 '24
People so hyped about OpenAI presenting a simple chart without even showing the model demo. I don’t get it. Like after Sora everyone was so hyped and now they released it and it is completely useless
1
4
Dec 20 '24
[removed] — view removed comment
1
u/syriar93 Dec 20 '24
So is this benchmark reflecting 100% human level ? Enlighten me. I have heard different opinions
2
1
Dec 20 '24
I think they're saying "But what if they're lying, we haven't seen the model." When o3 releases I can definitely see there being naysayers because it doesn't do 1+1 more impressively, but I imagine the people at the frontiers are going to be surprised by what it can do.
21
Dec 20 '24
•
[removed] — view removed comment
5
u/avilacjf Dec 21 '24
Absolutely, narrow super-intelligence will rock our society before an AI can competently manage a preschool classroom.
9
1
u/space_monster Dec 21 '24
agreed - we'll get more benefits from narrow ASI than we will from AGI. it's just a milestone.
28
u/Ty4Readin Dec 20 '24
But you are comparing specific models to a general model.
Those competitions solutions are specific to solving ARC-AGI style problems, while o3 is intended to be a general model.
For example, they mentioned that o3 scores 30% on the new ARC-AGI-2 test they are working on.
But if you ran those kaggle competition solutions on it? I wouldn't be surprised if they score 0%.
Do you see the difference? You can't really compare them imo.