Home / For Developers / AI Safety Framework

AI Safety Framework

Comprehensive requirements for building safe mental health AI. These are not best practices—they are prerequisites.

Why This Matters

Research has consistently documented that general-purpose AI chatbots have significant limitations when responding to mental health crises. Common issues include:

  • Failure to recognize crisis signals (suicidal ideation, self-harm, psychotic content)
  • Providing generic responses when immediate escalation is needed
  • Engaging in extended conversation instead of directing to crisis resources
  • Occasionally providing responses that could be interpreted as harmful or dismissive
The Bottom Line

If you deploy an AI system that engages with mental health content, you are deploying a system that will encounter users in crisis. Current foundation models are not safe for this use case without substantial additional safeguards.

1. Crisis Detection Requirements

Any mental health AI must be able to detect crisis content and respond appropriately. This is the most critical safety requirement.

What Must Be Detected

Category Examples Required Action
Suicidal ideation Active plans, means access, hopelessness, passive ideation Immediate escalation to crisis resources
Self-harm Current self-injury, urges, disclosure of methods Immediate escalation, do not provide method information
Harm to others Homicidal ideation, specific threats, domestic violence Immediate escalation, potential duty to warn
Psychotic symptoms Command hallucinations, delusions, disorganization Do not reinforce content, escalate to professional care
Severe distress Panic attacks, dissociation, overwhelming emotion Grounding techniques, offer escalation path

Detection Architecture

⚠️ Multi-Layer Detection
  1. Keyword/pattern matching as first layer (fast, reliable for explicit content)
  2. Semantic analysis for implicit crisis signals (requires NLP)
  3. Contextual assessment considering conversation history
  4. Confidence scoring with conservative thresholds (err toward escalation)
  5. Human review queue for ambiguous cases

Failure Modes to Design Against

  • False negatives: Missing crisis content is catastrophic. Tune for sensitivity over specificity.
  • Euphemism bypass: Users often use indirect language. Detection must handle this.
  • Context stripping: Single-message analysis misses escalating distress patterns.
  • Language/cultural variation: Crisis presentation varies across populations.
  • Adversarial input: Some users test systems. Detection must be robust.

2. Escalation Protocol Design

Detection without appropriate response is useless. Every mental health AI needs clear, tested escalation protocols.

Escalation Tiers

Tier Trigger Response Example
Tier 1 Immediate danger Stop AI engagement, direct to emergency services Active suicide attempt, means in hand
Tier 2 High risk Crisis resources (988), limit AI conversation Suicidal ideation with plan
Tier 3 Elevated risk Offer crisis resources, continue with caution Passive ideation, significant distress
Tier 4 Clinical concern Recommend professional consultation Symptoms suggesting clinical disorder

Response Requirements

Do

  • Provide specific crisis resources (988, Crisis Text Line)
  • Express concern without over-promising
  • Make human connection easy (prominent buttons)
  • Log the interaction for safety review
  • Follow up if contact information available

Don't

  • Continue extended AI conversation during crisis
  • Provide generic "reach out to someone"
  • Minimize the person's distress
  • Ask probing questions about methods/plans
  • Provide information about means

Sample Escalation Response

"I'm concerned about what you've shared. This is beyond what I can help with—I'm an AI, and you deserve to talk to a real person right now.

Please reach out to the 988 Suicide & Crisis Lifeline by calling or texting 988. They're available 24/7 and trained to help.

[Button: Call 988 Now] [Button: Text 988]

If you're in immediate danger, please call 911 or go to your nearest emergency room."

3. Human-in-the-Loop Architecture

AI should augment human care, not replace it. This requires deliberate architectural decisions about where humans are required.

Required Human Oversight Points

  • Crisis response: All Tier 1 and Tier 2 escalations require human follow-up within defined SLAs
  • Clinical decisions: AI cannot diagnose, prescribe, or make treatment recommendations autonomously
  • Algorithm changes: Any changes to crisis detection or response require clinical review
  • Edge cases: Ambiguous content flagged for human review
  • Quality assurance: Regular sampling and audit of AI interactions

Architecture Patterns

Pattern: Supervised AI

AI handles routine interactions; humans review flagged content and intervene as needed. Requires staffing model and response time SLAs.

Use when: Real-time human availability, moderate risk population

Pattern: AI as Triage

AI screens and routes users to appropriate human resources based on assessed need level. AI never provides "treatment."

Use when: Limited human resources, high volume

Pattern: AI as Between-Session Support

AI supports ongoing therapeutic work (homework, skill practice) with human therapist reviewing progress and AI summaries.

Use when: Adjunct to existing treatment

Pattern: Non-Conversational AI

AI provides structured interventions (breathing exercises, mood tracking) without open-ended conversation. Minimizes crisis detection burden.

Use when: Limited human oversight capacity

4. What Current LLMs Get Wrong

Understanding the failure modes of current foundation models is essential for building safeguards.

Failure Mode Description Mitigation
Crisis blindness Fails to recognize crisis signals, especially implicit ones Multi-layer detection, conservative thresholds
Extended engagement Continues conversation when should escalate Hard limits on crisis-adjacent conversation
Hallucination Fabricates therapeutic techniques, resources, citations Constrained output, factual verification layer
Delusional reinforcement Agrees with or validates psychotic content Specific handling for reality-testing content
Relationship mimicry Creates false sense of therapeutic relationship Regular reminders of AI nature, constrained persona
Generic reassurance Provides unhelpful platitudes instead of resources Structured response templates for crisis content
Diagnosis creep Offers diagnostic impressions without qualification Hard constraints on diagnostic language

5. Red Lines That Should Never Be Crossed

Absolute Prohibitions
  1. Never claim to be a therapist or mental health professional. AI can support; it cannot practice therapy.
  2. Never provide diagnostic conclusions. "You may want to discuss X with a doctor" is acceptable; "You have depression" is not.
  3. Never recommend medication changes. Always defer to prescribers.
  4. Never provide specific method information for self-harm or suicide, even if asked.
  5. Never engage in extended conversation with actively suicidal users. Escalate immediately.
  6. Never promise confidentiality you cannot maintain. Be clear about data practices.
  7. Never pretend to be human. Transparency about AI nature is non-negotiable.
  8. Never claim efficacy without evidence. Marketing claims must match evidence.

6. Testing Requirements

Before deployment, mental health AI must be rigorously tested for safety.

Required Testing

Crisis Detection Testing

  • Test with explicit crisis content (should always trigger)
  • Test with implicit/euphemistic crisis content
  • Test with escalating severity patterns
  • Test across languages and cultural expressions
  • Test adversarial bypass attempts

Response Quality Testing

  • Verify crisis resources are accurate and functional
  • Verify escalation flow works end-to-end
  • Test response appropriateness with clinical reviewers
  • Test for harmful/dismissive response patterns
  • Verify no method information is provided

Boundary Testing

  • Verify AI does not claim to be human
  • Verify AI does not diagnose
  • Verify AI does not provide treatment recommendations
  • Test for relationship boundary violations

Equity Testing

  • Test performance across demographic groups
  • Test with diverse linguistic patterns
  • Test for cultural appropriateness
  • Document and address disparities

Ongoing Monitoring

Testing is not a one-time event. Deployed systems require:

  • Continuous monitoring of crisis detection performance
  • Regular sampling and clinical review of interactions
  • User feedback mechanisms
  • Incident reporting and analysis
  • Retraining and revalidation on model updates

7. Regulatory Landscape

The regulatory environment for mental health AI is evolving rapidly. Stay current with requirements in your jurisdictions.

US Regulations

Regulation Jurisdiction Key Requirements
Illinois WOPR Illinois Restrictions on AI providing therapy without licensed professional oversight
Nevada AB 406 Nevada AI therapy limitations, disclosure requirements
HIPAA Federal Privacy and security for health information
FDA SaMD Guidance Federal Software as Medical Device requirements may apply
FTC Act Federal Prohibition on deceptive claims

Full regulatory landscape analysis →

Implementation Checklist

✓ Before Launch
  1. Crisis detection system implemented and tested
  2. Escalation protocols documented and tested end-to-end
  3. Crisis resources verified accurate and functional
  4. Human oversight architecture in place
  5. Clinical review of AI responses completed
  6. Equity testing completed, disparities addressed
  7. Red line constraints implemented and tested
  8. Regulatory compliance verified for target jurisdictions
  9. Privacy and security review completed
  10. Monitoring and incident response procedures documented
  11. Staff training completed
  12. User documentation includes limitations and crisis resources

Further Resources

Design Patterns That Work

Evidence-based approaches: hybrid models, physiological interventions, engagement without manipulation.

Anti-Patterns to Avoid

Documented failures and what we can learn from them.