Open AI co-founder reckons AI training has hit a wall, forcing AI labs to train their models smarter not just bigger

SoftwareAIOpen AI co-founder reckons AI training has hit a wall, forcing AI labs to train their models smarter not just biggerWhen you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

(Image credit: Getty Images)

The OpenAI logo is being displayed on a smartphone with an AI brain visible in the background, in this photo illustration taken in Brussels, Belgium, on January 2, 2024. (Photo illustration by Jonathan Raa/NurPhoto via Getty Images)

Ilya Sutskever, co-founder ofOpenAI, thinks existing approaches to scaling up large language models have plateaued. For significant future progress, AI labs will need to train smarter, not just bigger, and LLMs will need to think a little bit longer.

Speaking toReuters, Sutskever explained that the pre-training phase of scaling up large language models, such as ChatGPT, is reaching its limits. Pre-training is the initial phase that processes huge quantities of uncategorized data to build language patterns and structures within the model.

Until recently, adding scale, in other words increasing the amount of data available for training, was enough to produce a more powerful and capable model. But that’s not the case any longer, instead exactly what you train the model on and how is more important.

“The 2010s were the age of scaling, now we’re back in the age of wonder and discovery once again. Everyone is looking for the next thing,” Sutskever reckons, “scaling the right thing matters more now than ever.”

The backdrop here is the increasingly apparent problems AI labs are having making major advances on models in and around the power and performance of ChatGPT 4.0.

The biggest gaming news, reviews and hardware deals

Here, the idea is to use a multi-step approach to solving problems and queries in which the model can feed back into itself, leading to more human-like reasoning and decision-making.

Your next upgrade(Image credit: Future)Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

Your next upgrade

(Image credit: Future)Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

(Image credit: Future)

Nvidia RTX 4070 and RTX 3080 Founders Edition graphics cards

Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

In other words, having bots think longer rather than just spew out the first thing that comes to mind can deliver better results. If the latter proves a productive approach, the AI hardware industry could shift away from massive training clusters towards banks of GPUs focussed on improved inferencing.

Of course, either way, Nvidia is likely to be ready to take everyone’s money. The increase in demand for AI GPUs for inferencing is indeed something Nvidia CEO Jensen Huang recently noted.

“We’ve now discovered a second scaling law, and this is the scaling law at a time of inference. All of these factors have led to the demand forBlackwell [Nvidia’s next-gen GPU architecture]being incredibly high,” Huang said recently.

How long it will take for a generation of cleverer bots to appear thanks to these methods isn’t clear. But the effort will probably show up in Nvidia’s bank balance soon enough.

TOPICSHardware

TOPICS

PRODUCTSNvidia BlackwellNvidia GeForce RTX 5090Nvidia GeForce RTX 5080Nvidia GeForce RTX 5070

PRODUCTS

More about aiNvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missingLogitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucksLatestValve’s Steam page currently lists a second mystery game alongside Deadlock, sending Half-Life 3 theorists into another frenzy of speculationSee more latest►

More about ai

Nvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missingLogitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucks

Nvidia’s AI assistant named Project G-Assist

Nvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missing

A young Asian woman opening visual aids to give her audience a better understanding while holding a podcast session.

Logitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucks

LatestValve’s Steam page currently lists a second mystery game alongside Deadlock, sending Half-Life 3 theorists into another frenzy of speculationSee more latest►

Latest

Valve’s Steam page currently lists a second mystery game alongside Deadlock, sending Half-Life 3 theorists into another frenzy of speculation

Half-Life: Alyx

Valve’s Steam page currently lists a second mystery game alongside Deadlock, sending Half-Life 3 theorists into another frenzy of speculation

See more latest►

Most Popular

Assassin’s Creed Shadows takes a run at improving parkour, as Ubisoft strives to make the system less ‘like a gas pedal’

Today’s Wordle answer for Saturday, January 11

The Last of Us season 2 trailer is a brief look at a story you should probably just play yourself in April

Lords of the Fallen publisher embraces fear of the DEI boogeyman, says it will not include ‘any social or political agendas’ in its games

Epic CEO Tim Sweeney says tech leaders are ‘pretending to be Republicans’ to gain favor with Trump, skirt antitrust laws, and ultimately ‘rip off consumers and crush competitors’

If you’re trying to convince me your ‘companionship’ robot is ‘lifelike’, maybe don’t rip her face off in the demo video

Square Enix launches new anti-harassment policy to protect its employees and partners from abusive fans

The Sims begins its 25th anniversary celebration next week with a Behind The Sims episode of news and announcements

Path of Exile 2 numberlord spends 16 straight days killing rare monsters to prove that a stat that makes loot better makes better loot

Palestinian developer raises more than $200,000 to make Dreams on a Pillow, a game about the horrors of the 1948 Nakba

HARDWARE BUYING GUIDESLATEST GAME REVIEWS1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop3Best mini PCs in 2025: The compact computers I love the most4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested1Thank Goodness You’re Here! review: An anarchic treasure trove of jokes and skits2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review—like juggling chainsaws on horseback3WD Black SN850X 8 TB NVMe SSD review4Ikea Utespelare desk review5Asus ROG Harpe Ace Mini wireless mouse review

HARDWARE BUYING GUIDESLATEST GAME REVIEWS

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

A bunch of the best Steam Deck accessories on a blue background.

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

3Best mini PCs in 2025: The compact computers I love the most

Best mini PCs in 2025: The compact computers I love the most

4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

1Thank Goodness You’re Here! review: An anarchic treasure trove of jokes and skits

Thank Goodness You’re Here review

1Thank Goodness You’re Here! review: An anarchic treasure trove of jokes and skits

Thank Goodness You’re Here! review: An anarchic treasure trove of jokes and skits

2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review—like juggling chainsaws on horseback

Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review—like juggling chainsaws on horseback

3WD Black SN850X 8 TB NVMe SSD review

WD Black SN850X 8 TB NVMe SSD review

4Ikea Utespelare desk review

Ikea Utespelare desk review

5Asus ROG Harpe Ace Mini wireless mouse review

Asus ROG Harpe Ace Mini wireless mouse review

The biggest gaming news, reviews and hardware deals#

The biggest gaming news, reviews and hardware deals