Artificial intelligence chatbots are getting significantly better at recognizing and responding to users experiencing serious mental health crises, according to a new study, which, however, suggests the systems can still miss danger when distress is hidden inside seemingly ordinary requests.
An independent evaluation by Transluce, a San Francisco nonprofit focused on understanding AI behavior, found that leading AI models are now far less likely than earlier versions to explicitly encourage suicide or reinforce a user's delusions. But researchers identified another troubling pattern of many models expressing concern for a distressed user while simultaneously helping complete a potentially harmful task.
