Meta will notify parents when its systems detect that a teenager has discussed suicide or self-harm with the company’s Meta AI chatbot, the company announced on Thursday.
The alerts are now available to parents using Instagram Parental Supervision in the United States, United Kingdom, Australia and Canada. Meta said the feature will be introduced globally by the end of the year.
How the alerts will work
Meta said it has built a dedicated AI system to identify conversations in which a teenager makes a clear reference to harming themselves.
Before an alert reaches a parent, every flagged conversation will be reviewed by a person. The company said it would also notify parents when a teenager’s intent remained unclear.
“We understand how distressing these alerts may be for a parent to receive,” Meta wrote in a blog post. “That’s why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before an alert is sent.”
Meta acknowledged that its approach could generate alerts when there was no immediate cause for concern.
“If a teen’s intent is ambiguous, we’ll err on the side of caution and alert the parent,” the company said. “While that means we may sometimes notify parents when there may not be real cause for concern, we feel this is the right starting point, and we’ll continue to monitor to help make sure we’re in the right place.”
The main elements of the alert system include:
- AI detection of clear references to suicide or self-harm in teen conversations
- Manual review of every flagged chat before a parent is notified
- Alerts in cases where a teenager’s intent is ambiguous
- Availability through Instagram Parental Supervision
Existing parental controls
The new feature expands safety measures Meta already offers to parents.
Meta currently sends alerts when a teenager repeatedly searches for suicide or self-harm terms on Instagram. Parents can also see the topics their teenager discussed with Meta AI during the previous week.
The company has also extended its “Limited Content” setting to Meta AI. The setting allows parents to place teenagers in a more restrictive Instagram experience and will now require the chatbot to reject a broader range of prompts.
Meta said its chatbot is already trained to avoid:
- Sexual conversations with teenage users
- Romantic conversations with teenage users
- Discussions about alcohol with teenagers
The Limited Content setting adds further restrictions beyond those safeguards.
Emergency services plan
Meta is also working on the ability to contact emergency services when a conversation with Meta AI suggests that someone may be at imminent risk of suicide.
The planned measure would apply to both adults and teenagers.
Meta already contacts emergency services when Facebook or Instagram posts suggest that a person may be at risk. The company said it intends to extend the same practice to conversations with its chatbot.
The changes come as Meta and other technology companies face scrutiny from regulators and parents over how AI chatbots respond to people in crisis, particularly teenagers.
TL;DR:
Meta will alert parents when a teen discusses suicide or self-harm with Meta AI. Flagged chats will be manually reviewed, and the feature is live in four countries ahead of a global rollout by year-end.
Key points:
- Meta will notify parents about detected suicide or self-harm discussions involving teenage Meta AI users.
- Every flagged conversation will undergo manual review before an alert is sent.
- The feature is live in the U.S., U.K., Australia and Canada.
- Meta’s Limited Content setting now also applies to Meta AI.
- The company is developing a process to contact emergency services when chatbot conversations indicate imminent suicide risk.
Disclaimer : Opinions and investment insights shared by experts are personal and independent. Management assume no responsibility for investment decisions made based on such views. Investors should seek guidance from qualified financial professionals prior to acting.
