ReadyBed / SmartER

ReadyBed is a multi-agent EMS handoff + routing system. It combines live audio transcription, frame-by-frame visual analysis, and hospital resource simulation to deliver a pre-arrival summary and destination recommendation in seconds.

Highlights

• Multimodal intake: streaming audio + visual frames into a unified patient context.
• Pre-arrival handoff: concise clinical snapshot, timeline, and pathway signals.
• Routing engine: fastest time-to-treatment based on travel time + resource queues.
• Synthesized patient chart: ABCDE exam findings and assessment from live observations.
• Voice escalation: optional Twilio call + SMS updates with live context.
• Admin control: real-time hospital resource adjustments.

Live Views

• /: routing UI (travel time + wait time) with symptom input and hospital recommendations.
• /vid: redirects to /vid/live.
• /vid/live: ambulance-video intake with continuous frame analysis, transcript updates, synthesized patient chart, nurse call trigger, and route-patient action.
• /cam: webcam intake using the same multimodal pipeline.
• /summary: live summary dashboard (polls /api/vid every second).
• /admit: admin resource manager to change inUse/queue per hospital resource.
• /analyze: legacy placeholder page.

How It Works

1. Ingestion
   • Video/webcam frames are sent to POST /api/video/frame-analyze.
   • Audio chunks are sent to POST /api/audio/transcribe.
2. Agent state updates
   • Transcript segments and visual insights are stored in in-memory runtime state (lib/agents/state.ts).
   • Visual analysis also updates a synthesized patient chart (lib/agents/chart.ts).
3. Decision support
   • Route recommendations come from POST /api/triage or POST /api/agents/route-patient.
   • Routing combines travel time + simulated hospital wait states.
4. Handoff outputs
   • /summary consumes GET /api/vid, which merges trip store data, runtime context, and vision_analysis.json.
   • Twilio voice/SMS endpoints can speak or send context updates to receiving teams.

State Model

• Runtime context: lib/agents/state.ts
  • Transcript/live summary, visual report, routing result, patient chart, escalation state.
  • Stored in memory only (resets on server restart).
• Trip summary store: lib/trip/store.ts
  • Handoff summary, triage tags, timeline events, vitals trend.
  • Includes simulated vitals drift when queried.

API Surface

Routing + Resource State

• POST /api/triage
• GET /api/admin/update-resource
• POST /api/admin/update-resource
• POST /api/agents/route-patient
• GET /api/agents/routing-status

Live Multimodal Pipeline

• POST /api/audio/transcribe
• POST /api/video/frame-analyze
• GET /api/agents/live-summary
• POST /api/agents/live-summary/clear
• GET /api/agents/visual-status
• GET /api/agents/patient-chart
• DELETE /api/agents/patient-chart

Background Agent

• POST /api/agents/background/start
• POST /api/agents/background/stop
• GET /api/agents/background/status
• GET /api/agents/report

Trip Summary + Timeline

• GET /api/trip/summary
• POST /api/trip/vitals
• POST /api/trip/event
• POST /api/trip/finalize
• GET /api/vid

Voice + Messaging

• POST /api/voice/call
• GET /api/voice/twiml
• POST /api/voice/twiml
• POST /api/voice/handle
• GET /api/voice/status?callSid=...
• GET /api/voice/context
• POST /api/voice/respond
• POST /api/voice/send-update
• POST /api/voice/push

Utilities

• POST /api/media/split (split uploaded video into audio + no-audio video via ffmpeg)
• GET /api/analyze (deprecated compatibility endpoint)

Setup

1. Prerequisites

• Node.js 20+
• npm
• ffmpeg on PATH

macOS:

brew install ffmpeg

Ubuntu/Debian:

sudo apt-get update && sudo apt-get install -y ffmpeg

2. Install

npm install

3. Configure .env.local

Start from .env.example and set values based on what you want to run.

Minimum for multimodal AI pipeline

DEDALUS_API_KEY=your_key_here

Without DEDALUS_API_KEY, audio transcription and frame analysis endpoints fail.

Minimum for real Twilio voice/SMS

TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
TWILIO_FROM_NUMBER=+1...
APP_BASE_URL=https://your-public-url

If Twilio vars are missing, voice/SMS endpoints return simulation-mode responses.

ConversationRelay mode (optional)

ConversationRelay is only used when explicitly enabled.

TWILIO_USE_CONVERSATION_RELAY=true
TWILIO_CONVERSATION_RELAY_WS_URL=wss://your-relay-host/conversation-relay

If not enabled, Twilio uses a standard <Say> + <Gather> call flow.

Optional env vars

• DEDALUS_REPORT_MODEL (default: openai/gpt-5.2)
• DEDALUS_VOICE_MODEL (default: openai/gpt-5.2)
• GOOGLE_MAPS_API_KEY (if unset, travel time falls back to Haversine estimate)
• FFMPEG_PATH (default: ffmpeg)
• RELAY_PORT (default: 8081)
• TWILIO_CONVERSATION_RELAY_PATH (default: /conversation-relay)
• RELAY_BASE_URL (default: http://localhost:8081)
• NEXT_PUBLIC_RECIEVER_PHONE_NUMBER (note: key is spelled RECIEVER in code)

4. Run

Terminal 1:

npm run dev

Terminal 2 (only needed for ConversationRelay /api/voice/push flow):

npm run dev:relay

Open:

• http://localhost:3000/

Data Notes

• /vid/live expects public/processed/ambulance.mp4.
  • If you do not have that file, use /cam instead, or place your own clip at that path.
• /api/vid also loads vision_analysis.json for the summary dashboard’s vision panel.

Fallback Behavior

• Missing Twilio credentials: call/send-update endpoints return simulation payloads.
• Missing Google Maps key: travel times are estimated locally.
• Missing Dedalus key: voice responses can still return context-based fallback text, but multimodal analysis endpoints do not run.

Security Reminder

Do not commit real .env.local secrets (Twilio tokens, API keys, etc.) to source control.

Clinical Note

ReadyBed is for prototypes and demos only. It is not a medical device and does not provide clinical advice.