OpenCV × Alexa+ × Fire TV

A care system that confirms
before it escalates.

OpenCV notices something wrong - stillness, a covered camera, a fall signature. An Alexa+ agent checks in by voice, waits, and only then pages the family on Alexa mobile - while the Fire TV in the living room mirrors the whole trail, quoted conversation and all.

See the ladder pamu512/care-ladder-amazon
9 MCP tools, agent-callable Silhouette-only privacy Emergency dial: fail-closed off
Care Ladder Fire TV timeline showing the all-clear state: resident silhouette presence panel, reassurance headline and the six-rung escalation ladder standing by
production dashboard capture · 1280×720 · /firetv/
The escalation ladder

Six rungs. Every step confirmed, every step logged.

The ladder is the product: a care plan climbs rung by rung, and each climb is an auditable event. Voice check-ins carry the resident's real words. A covered camera is treated as privacy, never distress. And the last rung stays dark.

1 OpenCV cue Stillness past the plan window, resident not visible, or a fall signature - from on-device vision. CueDetector Demo plan: living-room stillness 4 minutes. An adaptive schedule can tighten the window as the system learns the household routine.
2 Re-check the room Re-perceive before acting - the cue is re-read against the zone before anything is said. reperceive Occlusion holds here and restarts on recovery: no distress waits while the camera cannot read the room.
3 Alexa+ voice check-in Two spoken attempts with real prompt text, classified fail-closed into reply intents. alexa_checkin ×2 "Are you okay?" - then a second, named attempt. A clear OK stands the ladder down. Mixed or concerning answers never do.
4 45-second listen window A bounded wait. Late answers still count; silence moves the ladder up. wait_window The window is honest about being a window: it closes, it is logged, and the family sees exactly how long it waited.
5 Page the family on Alexa mobile An inform card with a countdown and three actions - the caregiver pager. The Fire TV mirrors the same moment. notify_caretaker Channels: alexa_mobile. The caregiver answers from their phone; the TV keeps showing the trail for everyone in the room.
6 Request a call Simulated in the demo build, reserved fictional number only. request_call (555) 010-2276 - reserved NANP fiction. No real dial exists in this build.
- Emergency dial Off by default. Hard-locked behind a hold-to-review gate that explains, and never enables. emergency · off Care Ladder never calls emergency services on its own. A human would have to turn this on; in this build, no one can.
Resident reply intents

"I'm okay but I think I'm hurt" never clears the ladder.

Every voice reply is classified into one of three buckets before the ladder decides anything. Concern always wins over reassurance, and silence is never read as consent. Try it - this is the real classification logic from the repo, running in your browser.

What did the resident say?
Unclear Type a reply to see how the ladder would read it.

Clear OK clear_ok

Genuine reassurance, unmixed: "fine", "just resting", "don't worry". May stand the ladder down - resolve at the check-in rung, no page, no call.

Needs human needs_human

Any concern token - hurt, fell, help, can't get up - or an explicit call request. The ladder does not clear; the family is paged with the quoted reply.

Unclear unclear

Groans, empty transcription, garbage ASR. Never invents OK: treated as no answer, and the ladder keeps climbing calmly.

Caregiver ack intents

The pager listens too - on Alexa mobile.

When the family is paged, the inform card runs a conversation: a countdown, three actions, and free-text follow-up. The caregiver's answer is classified with the same fail-closed discipline - also the real logic from the repo, ported to this page.

What did the caregiver say?
Unclear Type an ack to see how the pager would read it.

I'm on it im_on_it

"On my way", "got it", "I'll call her myself" - a real human took the page. The ladder stops pressuring; the conversation waits for an outcome.

Call Mom now call_mom_now

An explicit ask for the call rung: the request-call step fires for the primary contact, simulated in this build.

Pass to next pass_to_next

"Can't take it - go to Secondary". Escalation pressure moves to the next contact, first ack still wins.

Outcome outcome

"Called Mom, she's fine", "checked on her" - a free-text close. The incident resolves with the outcome recorded in the trail.

Unclear unclear

Anything that doesn't read as a real ack. Never invents one: the countdown and escalation continue exactly as before.

Surfaces

Alexa mobile is the pager. Fire TV is the room's timeline.

The Alexa+ agent drives the ladder and speaks in the home; the caregiver's Alexa mobile app receives the page and closes the loop; the Fire TV mirrors it all for everyone on the couch - quoted conversation, rung rail, and full audit trail.

Fire TV timeline, all-clear state: presence silhouette with soft glow, All clear headline, six-rung ladder standing by

Fire TV - the living-room timeline supporting

Designed for the couch, driven by the remote: D-pad spatial focus, TV-scale type, and a presence panel that reads calm from across the room. Silhouettes only - no live feed, ever.

Fire TV timeline, escalated state: Needs human verdict, quoted voice check-in, ladder lit up to the notify rung

When it escalates needs human

The family is paged on Alexa mobile; this TV mirrors the moment - headline in plain words, both attempts quoted, and the ladder showing exactly which rung fired. The trail stays for the record.

The Alexa+ track gate

A real MCP server, agent-callable at runtime.

Self-hosted inside the same FastAPI app at /mcp - Streamable HTTP per the MCP spec 2025-11-25+ - exposing the care flow as nine tools. An in-repo MCP client plays the Alexa+ agent and drives the ladder over real HTTP: initialize, tool calls, caregiver ack, outcome, done.

start_or_resume_incidentopen or resume a care incident
check_in_promptspeak a check-in attempt
advance_rungclimb to the next rung
resolve_incidentstand the ladder down
get_incident_statusfull status + audit trail
notify_caretakerAlexa mobile inform card, simulated
request_callreserved fictional number
caregiver_ackfirst ack wins, stops pressure
caregiver_outcomerecord outcome, close incident
alexa_sim - MCP client over HTTP
→ initializeMCP Streamable HTTP · spec 2025-11-25+
← serverCare Ladder · tools: 9
→ tools/callstart_or_resume_incident {"cue_kind":"no_movement"}
← resultincident opened · rungs: reperceive → alexa_checkin → wait_window → …
→ tools/callnotify_caretaker
← resultfamily paged · channels: ["alexa_mobile"] · inform card + countdown
→ tools/callcaregiver_ack {"utterance":"I'm on it, calling her myself"}
← resultacked: true · action: im_on_it · escalation stopped
→ tools/callcaregiver_outcome {"text":"Called Mom, she's fine"}
← resultresolved · reason: caregiver_outcome
Safety posture

Built fail-closed, by default.

Emergency: off, and hard-locked

The emergency rung is enabled: false in the plan and fail-closed in code. The TV's hold-to-review gate explains what a live build would do - and never enables it.

Silhouette-only privacy

Frames are privacy-transformed before they leave the device path - blur or filled silhouette. No live video exists on any surface, and no raw identifiable frames ship in demos.

A complete audit trail

Every cue, attempt, wait, jump, ack and outcome is a timestamped event - the same trail the Alexa+ agent, the caregiver app and the Fire TV read from.

Reserved fictional numbers

Demo contacts use NANP reserved fiction only - (555) 010-2276. The dialer is a stub; no PSTN session can open from this build.