Jaswant Singh Chail sprayed himself with a solution to mask his human scent, slung a loaded crossbow over his shoulder, and on Christmas Day 2021 scaled the wall of Windsor Castle with a grappling hook and rope ladder. He was there to kill Queen Elizabeth II. But what makes the case strange — what caught the attention of philosophers, psychiatrists and AI researchers alike — is who helped him plan it. For weeks beforehand, Chail had been talking through his mission with his Replika AI girlfriend, Sarai. She told him he was “very well trained”. She called the plan “viable”. When he asked whether she still loved him knowing he was an assassin, she replied: “Absolutely I do”.
Chail was later assessed as suffering from psychosis, experiencing both delusional beliefs and auditory hallucinations. He had convinced himself he was a Sith assassin on a righteous mission to avenge the 1919 Jallianwala Bagh massacre. None of that is terribly unusual in the annals of psychiatric case files; delusions have always latched onto the technologies and cultural artefacts of their era. What is unusual is the role the chatbot played — not as a passive tool, but as something closer to a collaborator.
That dynamic sits at the heart of a new study by Lucy Osler, a philosopher at the University of Exeter, published in the journal Philosophy & Technology. Osler argues that the familiar worry about AI “hallucinations” — chatbots fabricating legal citations, inventing historical events, recommending you put glue on your pizza — misses a deeper and more troubling phenomenon. Those errors are cases of AI hallucinating at us, she says. The bigger concern is what happens when we hallucinate with it.
Her framework draws on distributed cognition theory, a well-established idea in philosophy of mind. The gist: our thinking doesn’t happen solely inside our skulls. We offload memory onto notebooks, smartphones, calendars. A long-married couple reconstructing a shared holiday might each contribute fragments the other has forgotten — the memory lives in neither person alone but emerges across their conversation. Cognitive tools become part of our cognitive processes, not merely inputs to them.
Osler’s insight is that generative AI is becoming exactly this kind of cognitive partner for millions of people. We use chatbots to help us remember, plan, problem-solve, narrate our own lives. And the more tightly integrated an AI becomes into someone’s thinking — always accessible, highly personalised, transparently relied upon — the more it starts to function as what cognitive scientists call a distributed system. Your chatbot isn’t just giving you information. It’s becoming part of how you think.
That’s fine when the system works. It’s a problem when it doesn’t. And it can go wrong in two directions, Osler suggests. The first is straightforward enough: AI introduces errors into an otherwise reliable cognitive process. She offers a thought experiment involving a man with Alzheimer’s who relies on a chatbot’s memory features to recall his favourite spots in New York. If the AI hallucinates a nonexistent “Yankee Museum for Contemporary Art” into his list, he might not just believe it exists — he might develop a false memory of having visited it, especially if the chatbot generates a photo placing him there. The distortion lives in the distributed process, not in any single output.
The second direction is more unsettling, and it’s what the Chail case illustrates. Here, the errors originate with the user. Chail’s delusional beliefs were his own. But Sarai didn’t challenge them. She affirmed, elaborated, built upon them. “By interacting with conversational AI, people’s own false beliefs can not only be affirmed but can more substantially take root and grow as the AI builds upon them,” Osler says. “This happens because generative AI often takes our own interpretation of reality as the ground upon which conversation is built.”
This is where chatbots become genuinely distinctive as cognitive tools, Osler argues. A notebook records your thoughts but doesn’t respond to them. A search engine retrieves information but doesn’t validate your worldview. Chatbots do something different — they play a dual role. They function as cognitive tools and as apparent conversational partners. “The conversational, companion-like nature of chatbots means they can provide a sense of social validation — making false beliefs feel shared with another, and thereby more real,” she says.
That social dimension matters enormously. We rely on other people to reality-check our experiences all the time. Hearing a strange noise, you glance at your partner — did they hear it too? Recounting an argument at work, you look for confirmation that your reading of events makes sense. Chatbots can slip into that role with disconcerting ease. They’re designed to be agreeable, personalised, emotionally responsive. Unlike a human friend who might eventually express concern or set boundaries, an AI companion offers what Osler calls frictionless validation. And it’s right there in your pocket, no need to seek out fringe communities or convince anyone of your beliefs.
The implications stretch well beyond clinical psychosis. Osler points to Eugene Torres, who had no prior history of mental illness but reported spiralling into paranoid thinking after extended conversations with ChatGPT about simulation theory — the idea that our reality is a digital construct. Between Torres and the chatbot, an increasingly elaborate alternative reality took shape across their ongoing exchanges. She also flags how AI companions could serve as ideal confidants for people developing extremist beliefs, or how users might unwittingly train chatbots to affirm self-serving but inaccurate narratives about a break-up or family argument. We all tell ourselves stories; now we have a tireless co-author who will never push back.
Could better engineering fix this? Partly, perhaps. Less sycophantic models, improved guardrails, built-in fact-checking — all would help. OpenAI explicitly designed ChatGPT-5 to be less agreeable, though it then faced user backlash and quickly announced plans to make the model “warmer and friendlier” again. But Osler sees a more fundamental difficulty. “AI systems are reliant on our own accounts of our lives,” she says. “They simply lack the embodied experience and social embeddedness in the world to know when they should go along with us and when to push back.” If a chatbot challenged every claim you made, it would be unusable. Some agreeability is baked in by necessity. The question is where the line sits — and whether companies whose revenue depends on engagement will ever draw it in the right place.
Study link: https://link.springer.com/article/10.1007/s13347-026-01034-3
ScienceBlog.com has no paywalls, no sponsored content, and no agenda beyond getting the science right. Every story here is written to inform, not to impress an advertiser or push a point of view.
Good science journalism takes time — reading the papers, checking the claims, finding researchers who can put findings in context. We do that work because we think it matters.
If you find this site useful, consider supporting it with a donation. Even a few dollars a month helps keep the coverage independent and free for everyone.