System Health Monitoring

Proactive infrastructure monitoring for every layer of the smart home — from network radios to ceiling fans to the hot tub.

Overview

A smart home with dozens of devices across multiple protocols will inevitably have things go wrong: a Zigbee coordinator loses its connection, a ceiling fan drops off WiFi, or the hot tub controller enters a zombie state where it looks connected but ignores commands. The health monitoring system catches these problems early and, where possible, fixes them automatically.

The system is built around two ideas:

1. Decoupled checkers — each checker app monitors one slice of infrastructure and reports status via Home Assistant events. The controller never needs to know how a check works.
2. A single aggregated sensor — the controller combines all checker reports into sensor.health_check_status, which custom Lovelace cards read to render the dashboard.

This architecture means adding a new health check is often just a YAML config change — no code required.

What Gets Monitored

Category	Examples	Checker Type
Network protocols	Zigbee bridge, Z-Wave controller, coordinator ping, web UI	NetworkProtocolChecker (Z-Wave uses the repair-capable subclass)
MQTT infrastructure	Broker publish/subscribe round-trip	MqttBrokerChecker
MQTT devices	Zigbee2MQTT device availability + linkquality	MqttDeviceChecker
Environmental sensors	Temperature and humidity with threshold alerts	TempHumidityChecker
Smart devices	Vestaboard, any entity + optional ping	BasicDeviceChecker
Powered devices	The printer and the Movie Room Sonos Port — repaired by power-cycling their smart plug or PDU outlet	RepairableDeviceChecker
Device groups	Cielo AC controllers, TP-Link plugs — related devices as one unit	DeviceGroupChecker
Ceiling fans	Modern Forms fans with per-fan repair; the outdoor Pavilion fan has no relay, so it is watched but never power-cycled (it is powered off and left out of the check over the winter)	FanHealthChecker
Hot tub / spa	Gecko integration health, staleness detection, power-cycle repair	SpaHealthChecker
Protect health	UniFi Protect — silent-freeze detection, hard-outage fast path, entry-sensor group, config-entry reload auto-heal	ProtectHealthChecker
Image generation	ComfyUI API reachability and queue progress — a GPU watchdog	ImageGenHealthChecker
Voice assistant	Speech-to-text and text-to-speech servers (Wyoming handshake) and the assistant's language model — what the phone's Assist card shows	VoiceHealthChecker
Batteries	Z-Wave, shade, lock, Airthings, Protect, Zigbee levels — with per-model thresholds for batteries that fall off a cliff (Airthings Wave Mini), a stale-reading check for cloud devices that went quiet, and an opt-in guard that tells a real low battery from a gateway-disconnect 0%	BatteryChecker
Motorized shades	PowerView G3 gateway RF-disconnect detection, with PoE port power-cycle repair	ShadeGatewayChecker
Internet	HTTPS reachability of Google, Cloudflare and Apple, which the cloud integrations depend on	CloudChecker
Power	Battery charge and load of each UPS	UpsChecker
AppDaemon itself	Heartbeat timestamp — the card detects staleness client-side	Controller heartbeat
The checkers themselves	Every configured checker has registered; a missing one is restarted, then reported as a red tile	Controller registration watchdog

How It Works

Event-Driven Architecture

Checker apps and the controller exchange all status and commands through Home Assistant events — never through direct Python references. This means:

• The controller can run in production Kubernetes while a new checker is being developed on a laptop
• Checkers can restart independently without affecting others
• Adding a new checker type requires no changes to the controller

The one exception is the controller's watchdog for checkers that never register. It reads AppDaemon's own list of apps to know which checkers should be present, and asks AppDaemon to restart one that is missing (see the health checks themselves). That only covers checker apps running in the same AppDaemon as the controller; a checker on a laptop is never restarted.

Checker Apps (Zigbee, Z-Wave, MQTT, Spa, Fans, ...)
  │
  │  register_checker / report_status events
  ▼
HealthCheckController ──▶ AppDaemon admin namespace (app list, app/restart)
  │                          — only for checkers that never register
  │  set_state()
  ▼
sensor.health_check_status
  │
  ├──▶ health-check-card (compact bar on wall displays)
  │         │
  │         │ tap to expand
  │         ▼
  └──▶ health-check-detail-card (full breakdown + repair controls)
              │
              │ callService("script", "health_check_relay", ...)
              ▼
         HA event bus → Controller routes commands back to checkers

Status Levels

Each individual check reports one of five statuses. The controller rolls these up to a per-checker status (worst wins) and then to an overall system status.

Status	Meaning	Card Color
ok	Everything healthy	Green
warning	Degraded but functional	Yellow
degraded	Significant issues	Orange
critical	Service down or unreachable	Red
unknown	Not yet checked or dependency unavailable	Grey

Dependency System

Checkers can declare dependencies on other checkers. When a dependency is unhealthy, or has not registered at all, the controller automatically marks affected checks as unknown instead of reporting misleading failures.

For example, MQTT device checks depend on the MQTT broker checker. If the broker itself is down, individual device checks show as unknown rather than critical — because the real problem is the broker, not the devices.

# basement_lights_checker (MqttDeviceChecker)
protocol_dependency_id: zigbee   # the "<device> MQTT" checks depend on Zigbee
health_dependencies:
  - checker_id: mqtt_broker      # every check depends on the broker

The controller resolves these dependencies at publish time without modifying the underlying check data, so when the broker recovers, device checks immediately resume reporting their true status.

A dependency that has not registered at all is named that way in the detail, for example dependency unavailable: Z-Wave (not registered). The controller restarts a missing checker and, if it stays missing, reports it as critical so it pages. See the health checks themselves below.

Heartbeat

The controller updates an input_datetime helper every 60 seconds. The dashboard card compares this timestamp against the current time client-side. If the heartbeat is stale by more than 3 minutes, the card shows AppDaemon as offline — no HA automation or template sensor needed.

Alert History

Every status transition is recorded with timestamps and duration tracking. The detail card shows a scrollable history of recent alerts, making it easy to see patterns like "the Zigbee bridge drops every night at 2 AM" without digging through logs.

Alerts are pruned automatically — both by age (default 36 hours) and by count (default 50 per checker).

Auto-Repair

Some checkers support automatic repair, typically via smart switch power cycling. The repair system follows strict safety rules:

• Only CRITICAL failures trigger repair — repair fires only when all checks are down; partial failures (warnings or degraded) do not cause a power cycle. The one deliberate exception is the Z-Wave bridge, where the partial failure is the fingerprint of the fault (see below)
• Only sustained failures trigger repair — a brief blip does not cause a power cycle
• Configurable delay — the problem must persist for a configurable number of minutes before repair begins
• Auto-clear on recovery — after a failed repair, the failed state automatically resets to idle when all checks recover. No auto-retry while checks are still unhealthy — except where a checker declares its own retry budget: the fans climb a CrashLoopBackOff ladder, and the Z-Wave bridge retries up to 3 times in 24 hours, 15 minutes apart, before giving up and paging
• Unknown does not trigger repair — if AppDaemon itself is restarting, repair actions are suppressed
• Cancellable — a pending repair can be cancelled via the detail popup before the power cycle executes
• Power comes back on, or it says so — a repair that switches a smart plug or power-strip outlet off and on (the printer, the Movie Room Sonos Port, the hot tub) checks that the switch really reports on again before it starts waiting for the device, tries the turn-on a second time if it doesn't, and if the outlet still won't come back the repair fails straight away with "did not turn back on — check the outlet" (or, if Home Assistant has lost track of the outlet altogether, a pointer to reload the integration that owns it). A repair must never quietly leave a device switched off

The repair state machine:

idle → pending → in_progress → success → idle (checks stay healthy)
         │            │           └──▶ failed (relapsed before a fully healthy cycle — no second repair this outage)
         │            └──▶ failed → idle (checks recover naturally)
         └──▶ idle (cancel_repair before the deadline — the dwell restarts, so it defers rather than dismisses)

Repair-capable checkers provision their own HA helpers for configuration:

• Auto-repair toggle — input_boolean to enable/disable
• Delay setting — input_number for minutes before repair triggers

These settings persist across AppDaemon restarts because they live in Home Assistant.

Per-device repair

The fan checker and repairable device group checker support per-device repair. Each device tracks its own repair state independently, and devices are repaired sequentially — one at a time — to avoid overwhelming the electrical system.

Paging: Alertmanager Integration

The dashboard is great when someone is looking at it. For failures that need attention now, the controller mirrors checker health into the Kubernetes cluster's Prometheus Alertmanager — the same Alertmanager that already routes critical infrastructure alerts to Pushover on the phone. No new notification plumbing; health checks just join the existing pipeline.

• Critical pages the phone. A checker going critical raises a severity=critical alert, which the cluster's routing delivers as a Pushover notification.
• Warnings stay quiet. warning and degraded map to severity=warning — visible in the Alertmanager and Grafana UIs, never a page.
• Recovery resolves once it sticks. When a checker reports healthy again the controller resolves the alert and the phone gets the matching [RESOLVED] notification — after a short hold that proves the recovery is real (see One page per incident below), so no lingering stale alerts and no resolve-then-refire ping-pong.

Each alert carries the failing check details in its description, so the notification usually tells the whole story on its own ("Event stream frozen for 3.2h — no genuine event since detection"). While an alert is firing, the controller re-posts it every two minutes to keep it alive (Alertmanager expires silent alerts after five), and if Alertmanager itself is down, health checking carries on unaffected — posts are simply retried until it returns.

The Cigars, battery, and UPS tiles remain single tiles that show every check. Their alerts distinguish an owner-action value such as low humidity, a low battery, or UPS charge/load outside its threshold from broken sensor telemetry. Owner-action alerts are marked for the host and skipped by the automated responder. Telemetry alerts remain eligible for the responder, so a failed sensor is not hidden behind an owner-action skip. The two alert conditions can fire and recover independently while the tile continues to show their combined health. Missing or malformed ordinary battery readings still show as unknown and do not page; UPS telemetry failures remain critical.

Checkers can customize their alert name in config (e.g. ProtectEventStreamFrozen, ImageGenQueueStuck) or opt out of alerting entirely.

Smarter Than a Tripwire

A page that fires on every momentary blip trains you to ignore it, so the paging path is deliberately patient:

• Debounce. A checker has to stay unhealthy for a sustained window — not just a single poll — before it pages. A transient blip that clears on the next check never reaches the phone.
• Escalation is gated too. When something that's already warning gets worse and goes critical, the critical page is held for that same sustained window instead of firing instantly. A checker that flaps between warning and critical keeps its quiet warning and never pages, while a real, sustained escalation still promotes to a page.
• Repair gets first crack. If a checker supports auto-repair and a repair is scheduled or running, the critical page is withheld while the repair does its thing — capped at 30 minutes so a stuck repair can never permanently mask a real outage. Most self-healing failures resolve inside that window with no page at all; only a repair that fails or times out escalates to the phone.

One Page Per Incident

The debounce keeps a blip from starting a page. Its mirror image keeps a blip from ending one — because the Pushover receiver sends resolves too, every recovery is a notification, and a condition that oscillates would otherwise page on every swing in both directions. One flapping Wi-Fi ceiling fan produced eighteen notifications in a single night that way.

So an improvement hold (alert_improve_hold_s, 15 minutes in production) applies to everything that makes a firing alert better:

• Recovery has to hold. A checker returning to ok doesn't resolve its alert on the spot — the alert keeps firing until the recovery has been continuous for the whole window. Drop back into failure inside that window and the improvement is discarded silently; nothing was ever posted, so nothing pages.
• De-escalations wait too. Critical easing back to warning is held the same way, with the critical alert staying up (annotations still refreshed each cycle, so the UI shows what is failing right now) until the calmer state proves it holds.
• Only improvements are held. Anything at or above the firing severity — a re-failure, an escalation — cancels the hold instantly. Getting worse is never delayed.

The result is one [FIRING] page when a real incident starts and one [RESOLVED] page when it genuinely ends, no matter how much the underlying condition thrashes in between. The cost is that the resolve lands up to 15 minutes after the dashboard turns green — deliberate, and worth it.

Reaching Every Alertmanager Replica

Alertmanager runs as an HA pair, and the pair gossips silences and the notification log — but not the alert set itself. Prometheus handles this by sending every alert to every replica; a direct poster has to do the same. Posting through the load-balanced Service left each replica with a different, partial view of which alerts existed, their notification logs never lined up, and both replicas paged for the same incident.

The controller now posts to the headless Service and fans each batch out to every address it resolves to, concurrently — one POST per replica, per cycle. A post counts as delivered if at least one replica accepts it, so a replica being down degrades to the old single-replica behaviour instead of losing the page. A hostname that resolves to a single address (or an IP outright) takes exactly the original code path.

Muting a Checker

Sometimes you already know about a problem and don't want to be paged about it — a camera is unplugged for the season, or you're mid-maintenance on the spa. Every checker in the detail popup has an Alerting row with Mute 1d, Mute 7d, and indefinite Mute buttons (and Unmute to lift it). A muted checker keeps running and still shows its true status on the dashboard — with a MUTED badge in its header — but it won't page while muted. A mute resolves all of that checker's alert conditions immediately, including an alert awaiting its normal recovery hold. Timed mutes lift themselves automatically when they expire, and a mute survives an AppDaemon restart, so a 7-day mute really lasts 7 days.

Auto-Heal First, Page If That Fails

The checkers that drove this integration span the whole spectrum, from "heals itself before you wake up" to "no automation can help":

UniFi Protect has a known failure mode where its websocket dies silently: camera entities keep updating, but every motion and smart-detection sensor stops changing state — with zero log errors. The checker detects the freeze (no events for hours of active daytime, so a quiet night never false-positives), reloads the Protect config entry automatically, and then waits for a genuine new event to prove the stream is really back — the reload itself re-registers every sensor with fresh timestamps, which would otherwise look like a recovery. Most freezes heal themselves this way; the page only goes out if the reload didn't work, and the alert keeps firing (annotated "auto-repair FAILED") until events actually resume.

A second, harder failure mode gets its own fast path: when the UNVR connection is lost outright (a reboot, an auth failure), every Protect entity flips to unavailable at once. That transition refreshes the sensors' timestamps, so the staleness detector alone would wait hours — instead, a Sensor Availability check pages within ~15 minutes when essentially everything is unavailable, and the same config-entry reload auto-heal kicks in. Warnings are reserved for devices that recently went offline after being seen alive — disabled-by-default Protect channels and the wireless cameras that live unplugged most of the year never produce standing warnings. The USL entry sensors (door contact/motion) are broken out as their own Entry Sensors line item: they warn when offline, but a door that nobody opens for hours never pages.

ComfyUI is the opposite. A queue that stops moving — or an API that stops answering — is the classic symptom of the GPU falling off the PCI bus on the virtualization host, and only a host reboot fixes that. No automation can help, so this checker is deliberately page-only: it raises the alarm and stays out of the way. Its thresholds are tuned to the real workload — the first generation after a ComfyUI restart takes ~10.5 minutes of model loading, still under the 30-minute stuck threshold, so cold starts never page. The generator side had to learn the same lesson: its render budget used to be a flat five minutes, so the first job after every restart timed out before the model had finished loading. Each workflow now carries its own budget — fifteen minutes for most of them, twenty for the one whose extra reference frames really do cost several times more.

PowerView shade gateway is the pattern in its purest form, and it exists because of a real overnight incident. When a Hunter Douglas G3 gateway loses contact with a shade, the shade doesn't report "offline" — it reports 0% battery, and a flapping gateway makes it bounce 100% ↔ 0% for hours. The old battery checker read every 0% as a dying battery and paged four times before dawn for something no human could fix by charging. Now two checkers cooperate: BatteryChecker recognizes the physically-impossible drop (a healthy battery can't lose 40+ points between readings) and downgrades it to a silent warning instead of paging, while a dedicated ShadeGatewayChecker owns the disconnect itself. It tracks a gateway-wide "disconnect episode" that survives the flapping — a momentary bounce back to 100% doesn't count as recovery — and stays silent through a grace period (default two hours), because these outages usually self-heal. If the shades are still flapping when the grace expires, it power-cycles the one thing that helps — the PoE port feeding the primary gateway — waits for the shades to come back, and only pages if that single restart didn't hold. One overnight page storm becomes zero pages when it self-heals, or one actionable "gateway restarted" page when it doesn't.

Ceiling fans show what happens when auto-heal is too eager. The fans are Wi-Fi devices; when one drops off the network the repair is a power cycle through its ZEN32 relay — a blunt instrument that reboots the fan to its hardware default. The retry schedule is borrowed from Kubernetes' CrashLoopBackOff: each failed repair doubles the wait before the next one, capped at six hours. What was missing was the reset rule. A fan that came back for ninety seconds counted as recovered, which zeroed the ladder, so the next drop earned another immediate power cycle — one flapping fan collected about eleven power cycles in five hours. Now a recovery has to hold for thirty minutes before the ladder resets, and a relapse inside that window counts the false recovery as another failed attempt and resumes climbing. The ladder is also written to a Home Assistant helper, because an AppDaemon reload mid-incident used to wipe it clean.

The other half of that incident was blame. Each fan declares the UniFi access point it usually holds — fans roam, so the mapping is a best-current value rather than a fixed binding, and one of them moved to a different access point within hours of being recorded — and when that AP is down the fan being unreachable is the network's problem, not the fan's: repairs are held (along with the backoff clock, so the ladder doesn't climb through an outage the fan didn't cause) and the alert text names the access point instead of the fan. It also says "Wi-Fi fan" out loud — during triage the ZEN32 relay in the repair path got mistaken for the fan's own radio, and half an hour went into the Z-Wave stack for a Wi-Fi problem.

The Z-Wave bridge is the case where the partial failure is the whole diagnosis. Z-Wave doesn't reach its radio over USB — it reaches it over TCP, across the house, to a TubesZB board running ESPHome. That board's serial-to-network server accepts exactly one client and never notices when that client goes away. So when a switch reboot dropped the TCP session at 3:24 one morning, nothing looked broken: the board answered every ping, its own "serial connected" sensor still read ON, the web UI still loaded. Only the controller was gone, and with it every lock, every Z-Wave motion sensor, and the door automations that depend on them — silently, for four hours and twenty minutes. Restarting the Z-Wave server doesn't help; the dead socket is on the board's side, and our end sits in FIN_WAIT2 waiting for a close that never comes. The one thing that works is telling the board to reboot itself, which drops the stale client so the waiting reconnect succeeds — it recovered in seventeen seconds when we finally did it by hand.

Two things had to change. The checker now presses that ESPHome restart button on its own, and the outage stops being invisible. The old behaviour is the reason nobody was paged: with the radio still pinging and the web UI still up, the cross-check that exists to suppress noise looked at "two of three checks fine" and downgraded a total Z-Wave outage to a warning — and warnings don't page. Now, once auto-repair has spent its budget, that downgrade is reversed and the alert goes critical.

The budget is the point. This board is fragile — its predecessor was killed by repeated power cycling — so the repair is a software restart, never a power cut, and it is fenced in on every side: five minutes of sustained failure before the first attempt, fifteen minutes between attempts, and at most three in any rolling day, after which it stops trying and pages a human instead. It only acts on the exact fingerprint above — controller down while the radio still answers — because if the board has genuinely dropped off the network, a software restart is not a thing that can help. That case now pages instead, which it previously didn't: a dead board still leaves the web UI answering, so the same "two of three checks are fine" arithmetic was quietly hiding it too. And the counter that enforces the daily cap is written to its own Home Assistant helper, so an AppDaemon deploy landing in the middle of an outage resumes the ladder where it left off instead of starting over with a fresh three restarts.

The health checks themselves can fail too, and for a while that was the quietest failure of all. Each checker announces itself to the controller when it starts. On 3 October 2026, Home Assistant moved to another node, AppDaemon restarted every app, and the Z-Wave checker hit a start-up bug and died before it announced itself. Nothing paged. Its tile simply disappeared, and because the Z-Wave battery and cigar humidor checks depend on it, 36 of them turned grey with "dependency unavailable: zwave". That rule exists so one Z-Wave outage doesn't page once per battery, but it also meant a leak sensor's battery could have gone flat that night without anyone hearing about it. It stayed that way for a little over three hours, until Tom noticed the grey tiles himself.

Now the controller knows which checkers should be there: every checker app in the AppDaemon config, plus any checker that another one depends on. A checker that hasn't announced itself two minutes after AppDaemon started it is restarted, then restarted again four minutes later and once more eight minutes after that; three tries at most. The clock doesn't start until AppDaemon has finished starting that app, so a slow restart of everything is never interrupted halfway. Once it has been missing for five minutes it appears as a red tile with a single Registration check that says what is wrong, and its alert waits the same five minutes as any other critical alert, so a restart that works inside that window never pages. The grey dependent checks now name the problem, for example "dependency unavailable: Z-Wave (not registered)". If the broken checker is switched off in the config instead, its page resolves straight away. Normal start-ups don't trip any of this, because every checker announces itself within seconds, and the lights' half-hour warm-up happens after they have announced themselves.

The Movie Room Sonos Port is the simplest case, and a good example of choosing the signal carefully. After a network switch hiccup one day the Port kept its cable link but stopped answering anything at all, and the Movie Room silently dropped out of the music system for six days — until a twelve-second power cycle of its outlet on the UniFi power strip brought it straight back. Now a checker pings it every three minutes; after ten minutes of silence it power-cycles that outlet and waits up to five minutes for the Port to return, paging only if it doesn't. What it deliberately does not watch is the Movie Room player itself: that disappears every time the music server restarts, and a routine restart must never cut power to the Port.

The power strip taught a second lesson. Switching one of its outlets makes the whole strip reconfigure itself for about forty seconds, Home Assistant can take well over ten seconds to notice, and once the integration lost track of every outlet on it until it was reloaded. A power cycle that simply says "on" and moves on could leave the Port switched off while it waits for it to recover. So every repair that switches an outlet now waits to see it actually report on — not trusting an "on" left over from before the cycle unless nothing else ever comes back, in which case it goes ahead but says so in the repair detail if the device then doesn't recover — tries once more if it doesn't, and fails loudly with "check the outlet" if it still won't come back. (The ceiling fans' repair works through a scene-controller relay and the shade gateway's through a PoE port, so they have no outlet to watch.)

Metrics and History

The controller publishes the same health data as Prometheus metrics, which the cluster's Prometheus collects for Grafana: every checker's and check's status, how long each has been in that state, repair outcomes and recovery times, and values some checkers report themselves, such as humidity, battery level and the image-generation queue. That gives history and graphs behind the alerts. See appdaemon/providers/metrics/README.md for the interface and the Prometheus Metrics section of appdaemon/apps/health_checks/README.md for every metric.

Dashboard Experience

Compact Health Bar

The compact card is designed for wall-mounted displays where screen space is limited. It shows a single row of colored status indicators — one per checker — with the overall system status.

• Tap the bar to open the detail popup
• Green bar with no indicators means everything is healthy
• Warning and critical states are immediately visible through color

Detail Popup

The detail card provides a full breakdown:

• Per-checker sections with individual check results and status icons
• Last check timestamps showing when each checker last ran
• Alert history with transition details (e.g., "Bridge Connection: ok -> critical")
• Force Re-check button to trigger all checkers immediately
• Clear History button to dismiss resolved alerts
• Repair controls for repair-capable checkers (manual trigger, cancel pending repair, auto-repair toggle, delay setting)
• Mute controls on every checker's Alerting row — silence its paging for a day, a week, or indefinitely (a MUTED badge and one-tap Unmute show while active)
• Triage notes that automations leave through the relay script appear inline in the alert history
• Missing checkers appear as their own red section with a single Registration check and no last-check time. It explains what is wrong; mute still works on it and triage notes are kept with it (see the health checks themselves)

Extending the System

Adding a New Protocol Check (Config Only)

The NetworkProtocolChecker supports any combination of entity state, ICMP ping, and HTTP checks. Adding a new protocol is a single YAML entry:

thread_health_checker:
  module: health_checks.checker_apps.network_protocol_checker.network_protocol_checker
  class: NetworkProtocolChecker
  checker_id: thread
  checker_name: Thread
  entity_id: binary_sensor.thread_border_router_state
  entity_healthy_state: "on"
  entity_check_name: Border Router
  radio_host: thread-br.local
  radio_check_name: Radio Ping
  check_interval_s: 180

Adding a New Device Check (Config Only)

For any device that has an HA entity and optionally responds to ping:

nas_health_checker:
  module: health_checks.checker_apps.device_checker.device_checker
  class: BasicDeviceChecker
  checker_id: nas
  checker_name: NAS
  ping_host: "192.168.0.10"
  ping_check_name: Ping
  check_interval_s: 300
  entities:
    - entity_id: sensor.synology_status
      healthy_state: normal
      name: Status

A Wi-Fi device that sleeps its radio will drop the odd ping while working perfectly, and one dropped ping turns its card yellow for a whole cycle. Add ping_attempts: 3 and the checker pings up to three times, accepting the first reply — the Vestaboard and the Cielo units run this way. A device that is really down still fails all three.

A device pinged by a DNS name can also be given its fixed IP as ping_fallback_host. Then a DNS problem can't masquerade as a dead device: if the name won't resolve, the checker pings the IP instead. A device that answers there shows a warning — "the name is broken, the device is fine" — which never pages and never power-cycles anything; a device that doesn't answer there is down for real, and repairs and pages as usual. The Movie Room Sonos Port is set up this way.

Adding Environmental Monitoring (Config Only)

The TempHumidityChecker supports configurable warning and critical thresholds:

server_room_temp_checker:
  module: health_checks.checker_apps.temp_humidity_checker.temp_humidity_checker
  class: TempHumidityChecker
  checker_id: server_room_temp
  checker_name: Server Room
  check_interval_s: 120
  temp_low_warning: 60
  temp_high_warning: 80
  temp_low_critical: 55
  temp_high_critical: 85
  sensors:
    - entity_id: sensor.server_room_temperature
      name: Rack Temperature
      type: temperature

Writing a Custom Checker App

For monitoring that goes beyond entity state and ping — such as the MQTT round-trip test or the spa staleness detector — you can write a custom checker app. The minimum contract is:

1. Listen for health_check_controller_ready and register with check names
2. Periodically run checks and report results via health_check_command events
3. Listen for health_check_recheck to support on-demand re-checks

The shared check_utils module provides reusable building blocks like ping_check() and http_check(), and switch_power_cycle provides power_cycle_switch() for a repair that turns an outlet off and back on and confirms it came back.

Current Checkers

Checker	Type	What It Monitors	Repair
Zigbee	NetworkProtocolChecker	Bridge connection, coordinator ping, web UI	No
Z-Wave	RepairableNetworkProtocolChecker	Controller state, radio ping, web UI	Yes — ESPHome software restart of the TubesZB bridge
MQTT Broker	MqttBrokerChecker	Publish/subscribe round-trip latency	No
Basement Lights	MqttDeviceChecker	Zigbee2MQTT device HA state + linkquality	No
Cigar Room Humidity	TempHumidityChecker	Humidity sensors with threshold alerts	No
Spa	SpaHealthChecker	Gateway ping, connections, multi-entity staleness (OR logic across thermostat/lights/pumps)	Yes — power cycle
Fans	FanHealthChecker	Entity state + IP ping per Wi-Fi fan; repair held while the fan's UniFi AP is down	Yes — per-fan zen32 power cycle on a CrashLoopBackOff ladder (not the Pavilion fan, which has no zen32 and is monitor-only)
Printer	RepairableDeviceChecker	Entity state + IP ping	Yes — power cycle
Movie Room Sonos	RepairableDeviceChecker	Ping only — the music player itself is deliberately not watched	Yes — power cycle of its UniFi power-strip outlet, on by default
Vestaboard	BasicDeviceChecker	Controller + configuration status	No
Cielo Home	DeviceGroupChecker	AC controller status + IP per room	No
UniFi Protect	ProtectHealthChecker	Sensor discovery + availability fast path + camera-event freshness in active hours + entry-sensor group	Yes — config entry reload
Image Gen	ImageGenHealthChecker	ComfyUI API reachability + queue progress	No — page only
Voice	VoiceHealthChecker	Wyoming describe handshake to the speech-to-text and text-to-speech servers; the phone's conversation agent loaded and its local LLM server healthy	No — page only
Shade Batteries	BatteryChecker	PowerView shade battery levels; downgrades implausible drops to warning (disconnect-aware) so a gateway blip isn't a low-battery page	No
Shade Gateway	ShadeGatewayChecker	Gateway RF-disconnect detection across all PowerView shades (implausible 0% flaps), gateway-wide episode tracking	Yes — PoE port 32 power-cycle
Cloud	CloudChecker	HTTPS reachability of Google, Cloudflare and Apple — the internet connection the cloud integrations need	No
UPS	UpsChecker	Battery charge and load of each UPS; pages straight away, never debounced	No
Z-Wave Batteries	BatteryChecker	Z-Wave device battery levels, leak sensors included; shown grey while Z-Wave itself is down	No
Zigbee Batteries	BatteryChecker	Zigbee sensor battery levels; shown grey while Zigbee is down	No
Schlage	BatteryChecker	Door-lock battery levels; shown grey while the Cloud check is failing	No
Airthings	BatteryChecker	Air-quality monitor batteries, with a higher threshold for the Wave Minis so they warn earlier (their batteries fall off a cliff), and a check for readings that stopped	No
UniFi Batteries	BatteryChecker	UniFi Protect entry-sensor batteries; shown grey while UniFi Protect is down	No
Downstairs, Upstairs and Exterior Lights	MqttDeviceChecker	Same as Basement Lights, for the rest of the house	No

Related

• App README: appdaemon/apps/health_checks/README.md
• Architecture: Overview
• Custom cards: appdaemon/apps/health_checks/cards/