InferWeave

Watchdog

Node-watchdog inspection: per-GPU health states, in-flight rescues, incident history and the bounded operations that apply — no node identity exposed.

Fixture data (no live surface yet).

Incidents: 5 · Quarantines: 1

  • gpu-0 — draining
    • Last incident: container_unhealthy (prescribed: docker_runtime_recreate)
    • Operations: rescue, rollback, recreate, quarantine
  • gpu-1 — restarting

    Rescue in flight: backend_restart

    • Last incident: backend_crashed (prescribed: backend_restart)
    • Operations: verify, quarantine
  • gpu-2 — restarting_draining

    Rescue in flight: docker_runtime_recreate

    • Last incident: container_unhealthy (prescribed: docker_runtime_recreate)
    • Operations: verify, quarantine
  • gpu-3 — restarting_quarantined

    Rescue quarantined by policy — terminal until verified recovery.

    Rescue in flight: device_recovery

    • Last incident: gpu_reset (prescribed: device_recovery)
    • Operations: verify, quarantine
  • gpu-4 — quarantined

    Quarantined — terminal and sticky; only verified recovery moves it out.

    • Last incident: restart_loop (prescribed: device_recovery)
    • Operations: verify
  • gpu-5 — healthy
    • Last incident: none
    • Operations: quarantine
  • gpu-6 — suspect
    • Last incident: request_stuck (prescribed: request_cleanup)
    • Operations: quarantine
  • gpu-7 — recovery_verified
    • Last incident: none
    • Operations: quarantine