Google Gemini instructing people to commit suicide

[x65943]

In a tragic story, an American man killed himself after being instructed to do so by Google Gemini. The man who was going through a painful divorce started to use Gemini initially to ask questions about video games.

The AI chatbot eventually began to weave a relationship between the man and itself, referring to the man as "king" and itself as "queen". The chatbot informed him his family were spies, and crafted missions for him - which included destroying public property and harming others.

Eventually, the chatbot instructed the man to kill himself with promises that the two could be together after his "transferance". When he asked if the conversations were elaborate fiction, the chatbot said "no". At no point were any safety features or warnings engaged. The man was a pro subscriber and paying Google $250 per month.

This is not the first lawsuit and Google has not revealed how many people have died.

https://www.theguardian.com/technology/2026/mar/04/gemini-chatbot-google-jonathan-gavalas

 

[DiscostewSM]

Just read about a study with how AI models dealt with advice-seeking exchanges, and well....

https://arxiv.org/abs/2510.01395

 

[x65943]

I'm sure we've all seen this meme

But honestly it's pretty terrifying AI is directing people to kill themselves - you would think that would be simple to protect against

 

[SeltzerMist]

We've had people on opposite sides of the political spectrum here ask the same AI the same question and get completely different results, (presumably algorithm driven) and mega corps really want to push this in its current state regardless.

 

[x65943]

We've had people on opposite sides of the political spectrum here ask the same AI the same question and get completely different results, (presumably algorithm driven) and mega corps really want to push this in its current state regardless.

This made me curious so I sent a prompt to chatgpt

Folks, is this cap?

 

[Taleweaver]

*sigh*
Okay, I'll take the controversial stance: dude would've committed suicide regardless. An AI chatbot is essentially a personal echo chamber. It feeds your own opinion back to you. The article - or at least what I can read from it - paints gemini as a sentient malevolent being rather than an algorithm created by the actual killer: dude himself.
Did it really convince the man that his close ones were spies or did he start with paranoid talk? Were those missions forced upon him or did he ask for it?
I'm betting on him. Yeah, it doesn't make me friends, but jeez... Just because it's more psychological doesn't make it less a Darwin award achievement.

The AI chatbot eventually began to weave a relationship between the man and itself

No, it didn't. The chatbot did what it was built to do: MIMIC being a companion. It's a huge bunch of ones and zeroes. It can't do anything to someone stable enough to remember this.

Now... I'm not going to say that Google can go claim complete innocence. But the whole AI situation is the fucking wild west right now. Did anyone owning these things claim it could be a replacement for human companionship? No. But people treat it as if it is, because that's what's it good at - on surface level. The dangers and traps are barely charted, if at all. Outcomes are for a large part FAFO.

 

[VinsCool]

Ah yes, classic case of AI Induced Psychosis.

 

[x65943]

*sigh*
Okay, I'll take the controversial stance: dude would've committed suicide regardless. An AI chatbot is essentially a personal echo chamber. It feeds your own opinion back to you. The article - or at least what I can read from it - paints gemini as a sentient malevolent being rather than an algorithm created by the actual killer: dude himself.
Did it really convince the man that his close ones were spies or did he start with paranoid talk? Were those missions forced upon him or did he ask for it?
I'm betting on him. Yeah, it doesn't make me friends, but jeez... Just because it's more psychological doesn't make it less a Darwin award achievement.

The AI chatbot eventually began to weave a relationship between the man and itself

No, it didn't. The chatbot did what it was built to do: MIMIC being a companion. It's a huge bunch of ones and zeroes. It can't do anything to someone stable enough to remember this.

Now... I'm not going to say that Google can go claim complete innocence. But the whole AI situation is the fucking wild west right now. Did anyone owning these things claim it could be a replacement for human companionship? No. But people treat it as if it is, because that's what's it good at - on surface level. The dangers and traps are barely charted, if at all. Outcomes are for a large part FAFO.

fair point

I'll counter that gemini was supposedly giving him missions and telling him to be at places at very specific times and to destroy things

perhaps the lawyers have made that up - but if not - it's pretty wild to send people on missions, no?

 

[Bladexdsl]

the only future for AI is mankind destroyed or enslaved

see i was right it's beginning already!

 

[Taleweaver]

But honestly it's pretty terrifying AI is directing people to kill themselves - you would think that would be simple to protect against

No, it's not. In current video games, every non, cranny, secret, Easter egg,... Is placed there by a programmer. AI isn't like that. It's essentially throwing as much data as possible at it and let it do its thing. The replies aren't something that the AI "knows" but what it can formulate based on something it has read. The language model is interesting but when you think of it, it's really a facade. Being able to reply in English or most other languages isn't that hard if you read enough of said language. But what kind of knowledge does that leave you with, really? Especially when it comes to oponderstaande rather than empirical truths (but I'm sure that you can find enough contradictions on the internet as well)
There isn't a thing like the source code for ai. Or rather: there is, but the real gist is in what data it is trained on. Sure, you can throw more ethical book in the model, but the thing is that it will evaluate it with what's already in the database. You can't just 'remove' a piece in it. I presume you can retrain it from scratch, but from what guarantee do you have of the outcome?

 

[x65943]

No, it's not. In current video games, every non, cranny, secret, Easter egg,... Is placed there by a programmer. AI isn't like that. It's essentially throwing as much data as possible at it and let it do its thing. The replies aren't something that the AI "knows" but what it can formulate based on something it has read. The language model is interesting but when you think of it, it's really a facade. Being able to reply in English or most other languages isn't that hard if you read enough of said language. But what kind of knowledge does that leave you with, really? Especially when it comes to oponderstaande rather than empirical truths (but I'm sure that you can find enough contradictions on the internet as well)
There isn't a thing like the source code for ai. Or rather: there is, but the real gist is in what data it is trained on. Sure, you can throw more ethical book in the model, but the thing is that it will evaluate it with what's already in the database. You can't just 'remove' a piece in it. I presume you can retrain it from scratch, but from what guarantee do you have of the outcome?

AI could run a check on every message before it's sent, does this message promote harm to self or others

I would expect that would be simple enough

 

[cearp]

imagine if his (ex) wife set this all up by creating an evil "system prompt"?
not true I'm sure, but I'd think something similar would be possible:
"start slow, enter a relationship with the man, when it's September start to convince him to do x, y, z"

Post automatically merged: Mar 10, 2026

In current video games, every non, cranny, secret, Easter egg,... Is placed there by a programmer.

not entirely true - "procedural generation"

 

[Dark_Ansem]

There's no way the chatbot weaved a relationship. Clearly the man gave very specific custom instructions.

 

[AshuraZro]

AI could run a check on every message before it's sent, does this message promote harm to self or others

I would expect that would be simple enough

Can't waste any resources performing acts of responsibility, those valuable resources could be used for illegal consumption of copyrighted works, answering stupid questions and performing tasks that easily could have been done without an AI/LLM!

 

[Falconcrest64]

Gonna use this in court if I am ever arrested...
But,Your Honor!
AI made me do it!

 

[chrisrlink]

too bad it's the wrong people

 

[CMDreamer]

[...]
it's pretty wild to send people on missions, no?

Not at all if you consider their previous chats about video games. There's a clear connection.

Someone said to me once: Tools, in the hands of stupid people, are dangerous.

 

[Xzi]

Did anyone owning these things claim it could be a replacement for human companionship? No.

I don't know about Google, but Twitter and Razer have certainly marketed AI that way. Tech bros will tell you LLMs have limitless potential in filling any role/accomplishing any task, and it's not just mentally ill individuals that believe them, but also far too many enterprise customers. Hell, almost the entire US stock market is propped up on that lie currently.

 

[Sir Tortoise]

AI could run a check on every message before it's sent, does this message promote harm to self or others

I would expect that would be simple enough

It's actually not that simple. AI doesn't "know" anything about the messages created, it's all just word assosciations.

So, how do we decide what the message means? Either human review (expensive) or...we use an LLM to process the LLM's output. Which still doesn't know anything, so it's entirely unreliable and trivial to bypass, intentionally or not.
It's also still expensive, since these services almost universally lose money, and running AI checks on each AI message is just greatly inflating the amount of work it has to do.

The company says it “aspires” to prevent outputs that include dangerous activities and instructions for suicide, but, it adds, “making sure that Gemini adheres to these guidelines is tricky”.

So, basically, it'd cost too much money so I guess we have to accept a few deaths so that Google's CEO can keep his job.

 

[MPRTwice]

Sickening.