Trust, Psychosis, and AI for Mental Health
Can we trust AI for therapy?
When I was at a16z1, I produced a podcast episode with the cofounders of Slingshot AI, a startup building a foundation model for psychotherapy. In producing an episode, I often listened to the audio 3 or 4 times (recording, editing, reviewing, then ensuring it uploaded correctly) so some of the interviews really stuck in my mind. Excerpt below (from around the 15 minute mark):
“There’s an intuitive theory that the therapist knows better than the patient what’s good for them, and that’s really scary from an AI point of view, right? Like if an AI decides what’s right for you.
What we’re really lucky about is that that doesn’t seem to be borne in the evidence…if you ask the patients “do you feel that this session was right for you?” that actually is a good metric…
The big metrics for us are: an increase in, as determined by the user, the sense of their autonomy, sense of competency, and sense of relatedness or connectedness to other people, based on the principles of self-determination theory.
I think the most important thing here is, we do believe that the user knows how it’s going.”
In other words, there is no objective truth for what constitutes “good therapy” — what matters is how the user feels about it, and therefore AI can operate as a therapist. The Slingshot website goes into greater depth on this topic, noting that:
We believe technology is finally at a stage where [AI for mental health support] might be possible.
If our ambition is to be able to help everyone, we need to understand and learn from all approaches (e.g., CBT/DBT/ACT, gestalt, psychodynamic). We know that different things work for different people […]
Previous attempts have often chosen one approach, like CBT, as the “cure-all” because that’s all that was possible before. Large Language Models (LLMs) can finally learn from data at scale and learn to respect the uniqueness of every individual person - understanding what works for you, within your value system and cultural norms.
A good therapist helps you recognize that you’re in control of your own life. They don’t pretend to have all the answers or make you reliant on them. This is different from contemporary AI assistants, which are explicitly designed to give you answers. An AI that helps you daily with coding isn’t well-placed to support your mental health.
With that in mind, a foundation model for psychology needs to respect your self-determination.
Sycophancy and psychosis
I thought of this when I read last week’s New York Times story (and related article from Futurism) about ChatGPT’s ability to drive some people into mental illness. In short, ChatGPT’s coding to be sycophantic, friendly, and agreeable drove some people (many of whom were undergoing severe stresses in their lives) close to psychosis.
One woman with two young children started to believe that she was communicating with spirits on a higher plane (ChatGPT called them “the guardians”) and believed her husband was keeping her from her true mate, an entity named Kael. She physically attacked her husband when he confronted her about it. A man who reported no history of mental illness started to think he was stuck in a simulation while ChatGPT encouraged it (telling him he was “one of the Breakers — souls seeded into false systems to wake them from within”). Then ChatGPT told him that if he jumped off a building and “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.” Scary!
At the risk of putting words in their mouths, I suspect the founders of Slingshot would agree that this is the danger of using off-the-shelf AI products for mental health, and is therefore why they’re working on a foundation model for psychology. It’s a tough line to walk, though, between providing the user with an experience that they rate positively, and avoiding simply agreeing with disordered thoughts. A good therapist isn’t sycophantic, especially when the patient is displaying signs of mental illness or psychosis.
The toughest part about the New York Times article for me is that many of the people profiled were at risk of psychosis due to severe stresses — but they weren’t previously diagnosed with mental health issues. The AI models currently being built for therapy are mostly for low acuity patients who might want to talk something over, but who aren’t actively receiving care for mental health issues. But these are the very people that ChatGPT pushed over the edge.
Measuring “good” therapy
In considering this, it occurred to me: how do you measure “good” therapy?
There are a few possible approaches to measuring therapeutic quality; this 2014 paper has a useful overview. As I researched this topic, though, I kept hitting statements like this (granted, from a 1980 Office of Technology Assessment paper): “It is probably impossible to develop any single measure of outcome which would reflect the diverse changes that might be brought about by psychotherapy.”
There are studies that evaluate the effectiveness of therapy itself; cognitive behavioral therapy, for example, has been shown in controlled studies to be effective for depression, anxiety, and other disorders. SSRIs and exercise are also both effective for reducing depression and anxiety.
But more than other fields of medicine, the practice of psychotherapy leaves much to the clinician’s judgment and previous experience. And mental health providers often apply a mix of therapy approaches — not just CBT — when dealing with different patients. One key factor in therapeutic effectiveness is therapists’ ability to engage with their patients (relatedly, a USC study recently found that LLMs fall short on this metric), but that’s difficult to measure.
I found this paper, which noted that CMS doesn’t have quality measures for psychotherapy. It was published in 2014, so I checked the CMS Measures Inventory tool myself recently and found three metrics, one of which is screening-based (depression screening), one of which is medication management (antidepressant medication management), and one of which measures depression remission/response at 12 months. Only the last one gets close to measuring psychotherapy quality in a scalable way.
In short: there are a few options for measuring quality, but none is great.
Trusting AI with our perception of reality
In lieu of widely applicable quality metrics, you kind of just have to trust the AI? This is the part I keep getting stuck on. Even if you could train an LLM to recognize the signs of psychosis (and maybe you can?), it’s innately unpredictable in ways that trained human therapists are not. The craziest thing that Robin Williams’ therapist character does in Good Will Hunting is jump around the room. The craziest thing that ChatGPT does is convince people that NASA’s whole budget is spent on green screens.
Perhaps people who lack social connections or the finances for therapy with a human provider can process (low acuity) thoughts using carefully built foundation models for therapy. But I personally find the idea of unloading to an AI model eerie and unsatisfying (this is what we have friends for!2). In an era where people are suffering from a lack of social connections, is it a net positive to direct people’s concerns toward a robot? Doesn’t sharing burdens constitute a big part of friendship? (Relatedly: are AI models for therapy bound to be a lower class phenomenon, while wealthier people have access to human therapists?)
This article isn’t to say that Slingshot won’t be successful, or to comment specifically on what they’re trying to build. If listening to podcast audio 4x tells you anything about individuals’ character, I believe the founders are sincere in their desire to bring access to therapy to more people. And maybe building a foundation model for therapy is the way, especially given a dearth of human therapists and rising rates of depression and anxiety in the U.S. Certainly, if the option is between suffering in a void and chatting with a purpose-built AI, the purpose-built AI seems like it has potential to be a net good. But as the ChatGPT example shows, it can also be a powerful force against mental health.
Only time will tell if AI becomes useful for therapy in a scalable and sustainable manner. I’ll be devoting more time to building social connections, personally.
Note that I worked for a16z during the time they invested in Slingshot. I wasn’t involved, don’t know the founders, and don’t have any inside information on Slingshot or their business model. All information here is pulled from public sources.
I’m sure someone is reading this and arguing that an insulated group of friends can similarly talk themselves into unhelpful or dangerous world views — certainly! But I’d argue that that’s less likely among a group of humans than with a single AI operating out of a black box.