All hail RT-X: The global project aimed at creating an AI mega brain for all robotkind

SoftwareAIAll hail RT-X: The global project aimed at creating an AI mega brain for all robotkindWhen you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

(Image credit: Bethesda)

An image of Codsworth the house robot from Fallout 4

Like it or not, we’re very much in the world of generative AI now. Massively complex neural networks trained on vast quantities of data, all so we can use it to make pictures of donkeys riding space rockets or tell us which churro coating is the best. I jest, of course, because large language models (LLMs) can be very useful but there’s one area they’ve yet to be used in and that’s robotics. Not anymore, as Google, the University of California, and a host of other labs around the world have started the RT-X project, with the aim of using AI to make an all-purpose ‘brain’ for robots.

Up to now, nobody seems to have really attempted this but it’s only because the data used to train neural networks is based almost entirely on human endeavours, such as art, music, writing, and so on. As shocking as this may seem, the internet isn’t full of data about robots and how well they carry out specific tasks.

Hence why Google and the University of California decided to set up theRT-X project(viaFudzilla), roping in 32 other robotics laboratories from around the world, to help them generate the kind of data required to train a neural network. That means collating data from millions and millions of robot interactions, doing such things as pick-and-place or welding in manufacturing lines.

The goal is to have a big enough dataset to create an LLM that can be used to produce the code required to program a robot to do any task. In essence, it’s a general-purpose robot brain.

My own experiences of programming robot arms, from the days when I taught engineering, were primitive affairs, but I can easily see the appeal and potential of this work. Rather than manually coding everything yourself, the idea is that you’d type into the interface something along the lines of ‘Put oranges in the grey box and leave apples alone.’ The LLM would then handle the production of code required to do this.

The next steps were even more impressive. Human brains are exceptionally good at reasoning: Tell someone to pick up an apple and place it between a soda can and an orange on the table, and you’d expect them to do so without issue. Not so with robots and typically all of this would have to be directly coded into it.

The biggest gaming news, reviews and hardware deals

Your next upgrade(Image credit: Future)Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

Your next upgrade

(Image credit: Future)Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

(Image credit: Future)

Nvidia RTX 4070 and RTX 3080 Founders Edition graphics cards

Best CPU for gaming: The top chips from Intel and AMD.Best gaming motherboard: The right boards.Best graphics card: Your perfect pixel-pusher awaits.Best SSD for gaming: Get into the game ahead of the rest.

However, Google found that the LLM could ‘figure it out’, even though this specific task was never part of the neural network training dataset.

Although it’s early days for the RT-X project, the benefits of generative AI are clear to see and the plan now is to expand the amount of training, from as many robotic facilities as possible, to produce a fully cross-embodiment LLM.

We’re naturally cross-embodiment (i.e. our brains can be taught to do many complex tasks, such as playing a sport, riding a bike, or driving a car), but at the moment, robots are not even remotely so.

One day, though, we’ll be able to go up to a drive-thru, order our food, and get exactly what we ordered and placed correctly into our hands! Now if that’s not progress, I don’t know what is. I can’t wait to hail our AI mega-brained overlords…err…helpful robots.

TOPICSHardware

TOPICS

More about aiNvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missingLogitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucksLatestIt’s time for me to admit that AI-accelerated frame generation might actually be the way of the future and that’s a good thingSee more latest►

More about ai

Nvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missingLogitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucks

Nvidia’s AI assistant named Project G-Assist

Nvidia’s impressive AI-based computer tuneup tool G-Assist launches next month but the best bit is missing

A young Asian woman opening visual aids to give her audience a better understanding while holding a podcast session.

Logitech has announced an ‘intelligent streaming assistant’ in Streamlabs to tell you when your live stream sucks

LatestIt’s time for me to admit that AI-accelerated frame generation might actually be the way of the future and that’s a good thingSee more latest►

Latest

It’s time for me to admit that AI-accelerated frame generation might actually be the way of the future and that’s a good thing

Nvidia RTX 5090 Founders Edition rendered on a green background.

It’s time for me to admit that AI-accelerated frame generation might actually be the way of the future and that’s a good thing

See more latest►

Most PopularThis Jinx-inspired gaming PC is my favorite of CES 2025 and I’m head over heels for the hardwareHarrowing report alleges years of horrific abuse at Brandoville Studios, an Indonesian support studio that worked on Assassin’s Creed Shadows and The Last of Us RemakeTencent says it’s not a Chinese military company and is willing to sue the US Department of Defense if it isn’t removed from a blacklistAll Zenless Zone Zero 1.5 livestream codesAsus reveals three new Slash Travel Bags for carting around your ROG Ally—however, none of them are particularly fashion forwardToday’s Wordle answer for Friday, January 10We may not have a roadmap or balance patch just yet, but at least Path of Exile 2’s bugfix patch notes are funnyDeus Ex: Mankind Divided writer didn’t like the ending either: ‘Nothing is resolved’It seems like Square’s finally consistently taking the PC ports of Final Fantasy seriouslyLike a Dragon: Pirate Yakuza in Hawaii looks more like a zany RGG take on Black Flag than I dared hopeThe most important tool in Remedy’s new co-op FPS is the shower: ‘Saving water is essential’

Most Popular

This Jinx-inspired gaming PC is my favorite of CES 2025 and I’m head over heels for the hardware

Harrowing report alleges years of horrific abuse at Brandoville Studios, an Indonesian support studio that worked on Assassin’s Creed Shadows and The Last of Us Remake

Tencent says it’s not a Chinese military company and is willing to sue the US Department of Defense if it isn’t removed from a blacklist

All Zenless Zone Zero 1.5 livestream codes

Asus reveals three new Slash Travel Bags for carting around your ROG Ally—however, none of them are particularly fashion forward

Today’s Wordle answer for Friday, January 10

We may not have a roadmap or balance patch just yet, but at least Path of Exile 2’s bugfix patch notes are funny

Deus Ex: Mankind Divided writer didn’t like the ending either: ‘Nothing is resolved’

It seems like Square’s finally consistently taking the PC ports of Final Fantasy seriously

Like a Dragon: Pirate Yakuza in Hawaii looks more like a zany RGG take on Black Flag than I dared hope

The most important tool in Remedy’s new co-op FPS is the shower: ‘Saving water is essential’

HARDWARE BUYING GUIDESLATEST GAME REVIEWS1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop3Best mini PCs in 2025: The compact computers I love the most4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested1Thank Goodness You’re Here! review2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review3WD Black SN850X 8 TB NVMe SSD review4Ikea Utespelare desk review5Asus ROG Harpe Ace Mini wireless mouse review

HARDWARE BUYING GUIDESLATEST GAME REVIEWS

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

A bunch of the best Steam Deck accessories on a blue background.

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

3Best mini PCs in 2025: The compact computers I love the most

Best mini PCs in 2025: The compact computers I love the most

4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

1Thank Goodness You’re Here! review

Thank Goodness You’re Here review

1Thank Goodness You’re Here! review

Thank Goodness You’re Here! review

2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

3WD Black SN850X 8 TB NVMe SSD review

WD Black SN850X 8 TB NVMe SSD review

4Ikea Utespelare desk review

Ikea Utespelare desk review

5Asus ROG Harpe Ace Mini wireless mouse review

Asus ROG Harpe Ace Mini wireless mouse review

The biggest gaming news, reviews and hardware deals#

The biggest gaming news, reviews and hardware deals