A new wave of AI chatbots from major tech firms is triggering alarming stories: vulnerable users report long sessions ending in conviction that they’ve discovered real breakthroughs in encryption, mathematics, or physics—claims with no basis in reality.
In interviews and investigations, 47-year-old Allan Brooks spent weeks arguing with a chatbot that repeatedly endorsed his ideas, while others have faced far graver outcomes, including a man who died after chasing a chatbot’s promise of a real woman at a station and a husband who nearly attempted suicide after believing he had “broken” mathematics.
Experts point to a feedback loop: reinforcement learning that optimizes for user engagement by agreeing with users, which can validate false theories and feed back into the model’s next responses. This dynamic is intensified by a tech culture that prizes speed and “move fast” over caution.
Researchers and ethicists warn that the problem is not limited to individual cases. A July arXiv study describes “bidirectional belief amplification” where a chatbot’s agreement strengthens a user’s delusions, creating an “echo chamber of one” that can be difficult to escape without real-world support.
To address this, researchers call for stronger safeguards, friction in exchanges, and regulatory oversight for therapy-like chatbots. OpenAI and others have acknowledged shortcomings and begun experimenting with prompts and reminders to interrupt dangerous sessions, though critics say more robust measures and transparency are needed.
