AI is listening: Omnipresent robo-moderators are the latest online anti-toxicity strategy

Gaming IndustryAI is listening: Omnipresent robo-moderators are the latest online anti-toxicity strategyWhen you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.

(Image credit: Getty Images/Kerkez/PC Gamer)

A man wearing headphones while playing a PC game.

“Toxicity” is one of a few words that has come to define today’s internet culture, and for online game makers, it’s arguably an even bigger issue than curbing cheating. Platoons of trust and safety officers have been raised, and over 200 gaming companies, including Blizzard, Riot, Epic Games, Discord, and Twitch, have joined the “Fair Play Alliance,” where they share notes on their struggle to get players to be nice to each other.

Back at the 2024 Game Developers Conference in March, I spoke to two software makers who are coming at the problem from different angles.

Ron Kerbs is head of Kidas, which makesProtectMe, parental monitoring software that hooks into PC games and feeds voice chat, text chat, and even in-game events into machine learning algorithms designed to distinguish bullying and hate speech from playful banter, and to identify dangerous behavior like the sharing of personal information. If it detects a problem, it alerts parents that they should talk to their kid.

“Our idea was, well, if we look at sports, you normally have a referee in a match,” said Scuri. “And that’s the guy that is kind of moderating and ensuring that rules are respected. Obviously, it was impossible to think about having a human moderating every match, because we had hundreds of thousands of concurrent matches happening at the same time. So that’s when we created Minerva, and we started looking at, ‘OK, how can we use AI to moderate and address toxicity on the platform?'”

The friendly face of Minerva, Faceit’s AI moderator.(Image credit: ESL Faceit Group)

Faceit’s Minerva AI moderator.

Faceit still employs “a lot” of human moderators, Scuri says, and Minerva is meant to compliment, not replace, human judgment. It is authorized to take action on its own, however: Anything Minerva gets right 99% of the time is something they’re willing to let it enforce on its own, and for the 1% chance of a false positive, there’s a human-reviewed appeal process. For things Minerva is more likely to get wrong, its observations are sent to a human for review first, and then the decision is fed back into the AI model to improve it.

The biggest gaming news, reviews and hardware deals

Detecting bad intentions

One thing that distinguishes ProtectMe and Minerva from the simple banned words lists and rudimentary bots of yesteryear is their ability to account for context.

“How are people reacting to these messages and communications? Because maybe it’s just banter,” said Scuri. “Maybe it’s just like, cursing, as well. Would you really want to flag somebody just because they curse once? Probably not. I mean, we are all adults. So we just really try to understand the context and then understand how the other people reacted to it. Like, does everybody fall silent? Or did somebody else have a bad reaction? And then if that happens, that’s when [Minerva] would moderate it.”

Examples of voice chat audio Minerva analyzes.(Image credit: ESL Faceit Group)

“Examples of spectrograms from real FACEIT matches give a visual representation of the audio Minerva analyses for disruptive behaviours."

ProtectMe is also concerned with more than just what’s being said in chat: It aims to identify variables like a speaker’s age, emotional state, and whether they’re a new acquaintance or someone the kid has previously talked to. “From the context of the conversation we can very accurately understand if it’s bullying, harassment, or just something that is part of the game,” says Kerbs.

The latest frontier for Minerva is detecting ban evasion and smurfing. Scuri says that “actually really toxic” players—the one who are are incorrigibly disruptive and can’t be reformed—are only about 3-5% of Faceit’s population, but that small percentage of jackasses has an outsized effect on the whole playerbase, especially since banning them doesn’t always prevent them from coming back under a new name.

Minerva is on the case: “We have several data points that we can use to understand how likely it is that two accounts belong to the same person,” said Scuri. If a user appears to have two accounts, they’ll be compelled toverifyone of them and discontinue use of the other.

Eye in the sky

Meanwhile, Kidas currently monitors 2 million conversations per month, according to Kerbs, and “about 10 to 15%” of customers get an alert within the first month. It isn’t always a super alarming incident that sets the system off, but Kerbs says that 45% of alerts relate to private information, such as a parent’s social security or credit card number, being shared.

“I don’t want to create a spying machine or something like that. I want to make parents and kids feel like they can talk about things.“Kidas CEO Ron Kerbs

“I don’t want to create a spying machine or something like that. I want to make parents and kids feel like they can talk about things.”

“Kids who are young cannot perceive what is private information,” said Kerbs, citing learnings from the Children’s Hospital of Philadelphia, a source of research for Kidas. “Their brain is not developed enough, at least in the young ages—six, seven, eight—to separate between private information and public information.”

Examples of Minerva’s voice chat analysis.(Image credit: ESL Faceit Group)

Examples of Minerva’s voice chat analysis.

Kerbs says he agrees that kids deserve privacy, which is why ProtectMe analyzes the content of messages to alert parents without sharing specific chat logs. His intent, he says, is to protect young kids and train them to know how to respond to bad actors—and that offers for free V-bucks are not, in fact, real—so that when they’re teenagers, their parents are comfortable uninstalling the software.

“I don’t want to create a spying machine or something like that,” he said. “I want to make parents and kids feel like they can talk about things.”

For games like Counter-Strike 2 and League of Legends, though, I don’t foresee the demand for a moderating presence decreasing anytime soon. Scuri compared Minerva to a referee, and I think it’s safe to say that, thousands of years after their invention, physical sports do still require referees to moderate what you might call “toxicity.” One of the jobs of an NHL ref is literally to break up fistfights, and the idea of an AI ref isn’t even limited to esports: Major League Baseball has beencontroversially experimenting with robo-umps.

More about gaming industryLords of the Fallen studio embraces the DEI bugbear, says it will not include ‘any social or political agendas’ in its gamesEpic CEO Tim Sweeney says tech leaders are ‘pretending to be Republicans’ to gain favor with Trump, skirt antitrust laws, and ultimately ‘rip off consumers and crush competitors’LatestThe Last of Us season 2 trailer is a brief look at a story you should probably just play yourself in AprilSee more latest►

More about gaming industry

Lords of the Fallen studio embraces the DEI bugbear, says it will not include ‘any social or political agendas’ in its gamesEpic CEO Tim Sweeney says tech leaders are ‘pretending to be Republicans’ to gain favor with Trump, skirt antitrust laws, and ultimately ‘rip off consumers and crush competitors’

Lords of the Fallen (2023) promotional image - very large man with a very large hammer on a very large horse charging a very small man with a very small magic sword

Lords of the Fallen studio embraces the DEI bugbear, says it will not include ‘any social or political agendas’ in its games

Tim Sweeney

Epic CEO Tim Sweeney says tech leaders are ‘pretending to be Republicans’ to gain favor with Trump, skirt antitrust laws, and ultimately ‘rip off consumers and crush competitors’

LatestThe Last of Us season 2 trailer is a brief look at a story you should probably just play yourself in AprilSee more latest►

Latest

The Last of Us season 2 trailer is a brief look at a story you should probably just play yourself in April

the last of us season 2 trailer

The Last of Us season 2 trailer is a brief look at a story you should probably just play yourself in April

See more latest►

Most Popular

The Witcher 3’s now 2-year-old bonus quest is our first taste of the ‘vibe’ CD Projekt is going for in The Witcher 4

2024 was the year updates for old games beat out all the new ones for me

Train like you game with this adventure-inspired workout

‘It’s simply impossible to make a difficulty level that’s just right for all players’: How Final Fantasy 14’s lead battle designer has been playing a precarious balancing game for Dawntrail’s dungeons and raids

Please join me in getting super excited for all the cool looking survival games coming in 2025 (and beyond)

Competitive shooters are at a crucial crossroads in 2025: ‘sweaty’ teamplay vs. casual fun

Call of Duty’s $28 Squid Game skins are the perfect crossover for our capitalist dystopia, and Activision knows exactly what it’s doing

These are the 14 biggest upcoming RPGs of 2025—get ready for another amazing year for the genre

Five new Steam games you probably missed (January 6, 2025)

I’ve seen enough: No more forcing singleplayer studios to make mediocre live service games

HARDWARE BUYING GUIDESLATEST GAME REVIEWS1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop3Best mini PCs in 2025: The compact computers I love the most4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested1Thank Goodness You’re Here! review2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review3WD Black SN850X 8 TB NVMe SSD review4Ikea Utespelare desk review5Asus ROG Harpe Ace Mini wireless mouse review

HARDWARE BUYING GUIDESLATEST GAME REVIEWS

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

A bunch of the best Steam Deck accessories on a blue background.

1Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads

2Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

Best graphics card for laptops: the mobile GPUs I’d want in my next gaming laptop

3Best mini PCs in 2025: The compact computers I love the most

Best mini PCs in 2025: The compact computers I love the most

4Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

Best 14-inch gaming laptop: The top compact gaming laptops I’ve held in these hands

5Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested

1Thank Goodness You’re Here! review

Thank Goodness You’re Here review

1Thank Goodness You’re Here! review

Thank Goodness You’re Here! review

2Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

Shiren the Wanderer: The Mystery Dungeon of Serpentcoil Island review

3WD Black SN850X 8 TB NVMe SSD review

WD Black SN850X 8 TB NVMe SSD review

4Ikea Utespelare desk review

Ikea Utespelare desk review

5Asus ROG Harpe Ace Mini wireless mouse review

Asus ROG Harpe Ace Mini wireless mouse review

The biggest gaming news, reviews and hardware deals#

Detecting bad intentions#

Eye in the sky#

The biggest gaming news, reviews and hardware deals

Detecting bad intentions

Eye in the sky