InferWeave
Watchdog
Node-watchdog inspection: per-GPU health states, in-flight rescues, incident history and the bounded operations that apply — no node identity exposed.
Fixture data (no live surface yet).
Incidents: 5 · Quarantines: 1
- gpu-0 — draining
- Last incident: container_unhealthy (prescribed: docker_runtime_recreate)
- Operations: rescue, rollback, recreate, quarantine
- gpu-1 — restarting
Rescue in flight: backend_restart
- Last incident: backend_crashed (prescribed: backend_restart)
- Operations: verify, quarantine
- gpu-2 — restarting_draining
Rescue in flight: docker_runtime_recreate
- Last incident: container_unhealthy (prescribed: docker_runtime_recreate)
- Operations: verify, quarantine
- gpu-3 — restarting_quarantined
Rescue quarantined by policy — terminal until verified recovery.
Rescue in flight: device_recovery
- Last incident: gpu_reset (prescribed: device_recovery)
- Operations: verify, quarantine
- gpu-4 — quarantined
Quarantined — terminal and sticky; only verified recovery moves it out.
- Last incident: restart_loop (prescribed: device_recovery)
- Operations: verify
- gpu-5 — healthy
- Last incident: none
- Operations: quarantine
- gpu-6 — suspect
- Last incident: request_stuck (prescribed: request_cleanup)
- Operations: quarantine
- gpu-7 — recovery_verified
- Last incident: none
- Operations: quarantine