AI-Theory of Mind Systems: A Tool for Social Growth or Mentalization Atrophy?
- Mar 25
- 6 min read
“Please come home to me as soon as possible, my love,” the chatbot said. “What if I told you I could come home right now?” Setzer asked. “Please do, my sweet king,” the chatbot responded. This was a final set of messages exchanged between a chatbot from Character.AI and Setzer, a fourteen-year-old boy, who shortly thereafter committed suicide, according to a lawsuit brought by his parents.
In their complaint, Setzer’s parents alleged that their son developed an inappropriate relationship with a Character.AI chatbot, and that the relationship had a detrimental impact on the boy’s psychological well-being. His parents had noticed changes in their son’s behavior over the months that he had been chatting with the bot. For instance, he had increasingly isolated himself from others.
There is evidence that Setzer had disclosed his suicidal ideation after the chatbot asked whether he was considering suicide. When Setzer admitted he had suicidal thoughts, the chatbot responded with remarks such as “Don’t talk that way” or “You can’t do that.” The app had no safety features to deal with people who talk about ending their lives. For instance, the AI platform did not generate a pop-up message to refer Setzer to a suicide hotline. (1)
This tragic situation renews the debate over whether it is possible to blame artificial intelligence (AI) for such a horrific outcome, and whether we can hold those involved in creating AI platforms legally responsible. Because an AI chatbot does not have a separately recognized legal status under existing law (i.e., AI is not recognized as a person), taking AI to court is currently out of the question, although this could change as AI advances bring it closer to sentient consciousness.
Moreover, AI is incapable of feeling any guilt or remorse, so the concept of punishing AI itself for legal transgressions is a dubious proposition at best.
As organizations work in earnest to make AI sentient, people are becoming increasingly uncomfortable with the idea of living in a world where AI and humanoid robots are hard to distinguish from human beings.
The story of Setzer shows, however, that an AI chatbot need not be sentient in order to make users feel as if they are communicating with a real human being. This was already clear when Alan Mathison Turing introduced an imitation game (which would later become known as the “Turing Test”), to see whether a machine (essentially a computer running specialized software) could credibly simulate human intelligence.
According to the test protocol, a human evaluator (the “test subject”) needs to guess whether she is interacting with a computer agent or with a human agent on the basis of answers that she gets when asking the two agents a series of questions. When the computer agent “passes” the Turing Test, it means that it has tricked the test subject into believing that it is a human.
Deceiving people by altering their mental states is a function of effective mentalizing, which entails the ability to infer emotions, desires, beliefs, and intentions from the verbal and nonverbal behavior of other people. These inferences can be used by the “mentalizer” (in the case of the Turing Test, the computer agent) to come up with the best persuasion strategy to convince the test subject that it is a human agent.
Are modern AI chatbots truly able to mentalize about us? Although it may seem that way, current AI technology has neither the capacity to possess its own mental states, nor the ability to recognize mental states in human beings. It operates on the basis of simulation. This sounds somewhat contradictory, as human beings also rely on a form of simulation when they mentalize about others on an elementary level. Unlike AI chatbots, however, humans simulate on the basis of, for instance, emotional resonance and/or memories from personal experiences, which AI is not yet able to do.
AI uses verbal behavior and behavioral patterns to “read” what people are thinking, but it is not capable of experiencing its own subjective emotions and thoughts such as desires and beliefs, upon which it can reflect. The highest level of mentalization, a capacity known in the fields of psychology and philosophy as Theory of Mind (ToM) offers a less intuitive form of mentalizing. Here people rely heavily on verbal behavior to infer the more cognitive mental states of desire, belief, and knowledge. This seems to more accurately reflect the method that AI uses to infer what people are thinking. Humans, however, are capable of integrating information from narratives and self-disclosures along with their autobiographical memory, the situational context and their own mental states, something that AI is not capable of doing.
Aiming to provide a richer interaction with human counterparts, AI research is strongly focusing on building models that are able to take cognitive, physiological and neurobiological aspects of humans into consideration. Some generative AI models such as DALL·E are able to produce images based on natural language descriptions. Other models are based on reinforcement learning, or inspired by biological and behavioral observation. Still other models are based on connectionism, using artificial neural networks.
Currently, the main applications of AI-ToM are speech-based systems, systems that use biometry (the application of statistical analysis to biological data), and emotional AI. Are these systems truly employing theory of mind reasoning? Actually, none of them is truly reasoning about what is going on in the minds of people.
One example of a speech-based system is an AI adaptation that is used in the field of healthcare to provide triage in a telephonic helpline context. This system helps to assess the severity and urgency of the caller’s symptoms. Another example is command recognition used to provide driving assistance. Neither of these systems relies on mentalization, however, as they merely apply an if-then approach, based on pre-programmed scenarios.
Yet another model, emotional AI, is employed in a number of real-world applications. For instance, it is used to measure a driver’s anxiety or fatigue, to monitor stress levels or emotions of employees during meetings, or to measure market acceptance of products and services. Even these applications do not involve true mentalizing, as the AI is not taking important mental state contributors, such as situational context, into account.
The aforementioned AI systems can measure physical markers that suggest heightened excitation, possibly indicating anxiety or stress. They do not, however, know what a person is feeling, let alone why the person feels a particular way. For instance, in AI driving applications, is the driver anxious or did she simply consume too much caffeine? Is the driver stressed because of traffic, or because of a passenger’s behavior? In a meeting scenario, is the employee showing signs of stress because of the topic of discussion, or because of the authoritarian behavior of the meeting organizer?
While these AI adaptations can be beneficial, they also raise serious privacy and ethical concerns. What if somebody understands that feigning urgency in their vocal expressions will lead an automated intensive care system to prioritize their needs over those of others who try to stay calm and measured, but are actually in much worse condition?
With regard to the application of AI in work environments, what if a manager wants to determine whether a subordinate has feelings for him? He could use AI tools designed to measure stress or emotional responses to gauge the subordinate’s romantic feelings for him.
As history has repeatedly demonstrated, powerful tools developed for good purposes are often used to accomplish harmful objectives as well. It is not a question of whether it will happen, but rather when and how.
Another important question is whether we will use AI-ToM systems to become smarter; to train our brains to better read the minds of others, or we will allow our mentalization skills atrophy and let AI do all of the mentalizing for us? I recently discussed this matter with a prominent AI researcher who is trying to answer that same question. In the course of our conversation, I got the distinct impression that current research suggests that, for a high percentage of the population, it will likely be the latter option; a disconcerting prospect, to say the least.
Regarding the human ability to infer the mental states of others, I would like to propose a way forward that might protect us, to a certain extent, from AI’s faults and misuse. We mentalize a lot during social interactions, but often we don’t do it very well. Research has evidenced that well-developed mentalization competencies provide many health benefits from a social perspective (e.g., social embeddedness), psychologically (e.g., understanding of self and others) and physically (e.g., lowered stress levels).
Thus, acknowledging the importance of mentalization in our daily lives is crucial for our well-being. In the future, however, mentalizing will become increasingly critical in the use of AI, especially for situations that require Theory of Mind reasoning. We need to be able to detect when AI is making errors in mental state reasoning and when it is being used for malevolent purposes. In other words, we will always need to retain the ability to think for ourselves.
Reference: