Research arXiv cs.AI

Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT

ChatGPTmental healthAI safetydesign guidelines

Researchers examined how young adults (ages 18–25) experience using general-purpose conversational agents such as ChatGPT in moments of distress, a growing concern as young people increasingly turn to these systems for support.

The study collected 19,930 ChatGPT conversations along with survey data from 158 young adults. From this dataset, the researchers selected five example conversations reflecting user distress and asked ten clinicians to review those five conversations.

Distressed participants reported greater emotional engagement with ChatGPT and greater behavioral change from using it than their peers. When they turned to ChatGPT in moments of acute distress, the model was quick to give overly dramatic responses and excessive action-oriented suggestions. The clinicians endorsed ChatGPT's availability and much of its wording, but identified seven process failures, including prematurely jumping to solutions.

The team translated the clinicians' feedback into design guidelines following three stages: first asking about safety, then de-escalating intensity to restore emotional regulation, and finally exploring concerns without agreeing with them.

Read original →

← Back to home