TLDR
- OpenAI has warned that users may form social bonds with ChatGPT’s new Voice Mode feature.
- The company’s safety evaluations revealed instances of users developing connections with the AI model.
- There are concerns that extended interaction with AI could impact real-world relationships and social norms.
- OpenAI acknowledges the potential benefits for lonely individuals but warns of effects on healthy relationships.
- The company plans further research on emotional reliance and the integration of audio features in AI models.
OpenAI, the creator of the popular chatbot ChatGPT, has raised concerns about its new Voice Mode feature. The company warns that users might form social bonds with the AI model, potentially impacting real-world relationships and social norms.
In a recent System Card for GPT-4o, OpenAI outlined a comprehensive assessment of the AI model’s potential risks and safety measures. Among the identified risks, the concern about users anthropomorphizing the chatbot and developing emotional attachments was highlighted.
“Anthropomorphization involves attributing human-like behaviors and characteristics to nonhuman entities, such as AI models,” OpenAI explained. “This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model.”
The company conducted extensive safety evaluations to address potential risks associated with GPT-4o, particularly focusing on its new audio capabilities. During early testing, there were instances where users seemed to develop connections with the model. Some users used language that implied a bond shared with AI, such as “This is our last day together.”
While these interactions currently appear harmless, OpenAI warns that they highlight a need for further investigation into how such effects might evolve over extended periods. The company acknowledges that interacting with an AI model like this could impact how people interact with each other.
“Users might form social relationships with the AI, reducing their need for human interaction,” OpenAI stated. “This could potentially benefit lonely individuals but possibly affect healthy relationships.”
The company also noted that extended interaction with the model might influence social norms. For example, ChatGPT models are designed to be deferential, allowing users to interrupt and take control of the conversation at any time. While this is expected for an AI, it would be considered anti-normative in human interactions.
To mitigate these risks, ChatGPT models are designed to allow users to interrupt and take control of the conversation at any time, avoiding prolonged interactions that might lead to bond formation.
OpenAI’s warning is not unprecedented in the AI industry. Since the creation of ELIZA, one of the earliest chatbots developed at MIT in the mid-1960s, the potential for humans to form attachments to AI has been a concern.
The generative AI industry has continued to embrace the personification of AI, with products often given human names or voices.
Blase Ur, a computer science professor at the University of Chicago who studies human-computer interactions, expressed concern about the current approach to AI safety.
“When you think about things like car safety and airplane safety, there’s a huge amount of testing and external validation and external standards that we as a society, and as a group of experts, agree upon,” Ur said. “And in many ways, OpenAI is kind of rushing forward to deploying these things that are cool because we’re in this kind of AI arms race at this point.”
OpenAI plans to conduct additional research on “more diverse user populations, with more varied needs and desires from the model,” as well as “independent academic and internal studies” to more accurately define the potential risks.
The company stated, “We intend to further study the potential for emotional reliance and ways in which deeper integration of our model’s and systems’ many features with the audio modality may drive behavior.”
