diff --git a/10-systems/hardware/hp-proliant-dl380-g6.md b/10-systems/hardware/hp-proliant-dl380-g6.md index 9b0fb51..fd100b1 100644 --- a/10-systems/hardware/hp-proliant-dl380-g6.md +++ b/10-systems/hardware/hp-proliant-dl380-g6.md @@ -1,7 +1,7 @@ --- owner: gmikcon status: active -last_reviewed: 2026-05-09 +last_reviewed: 2026-10-30 review_interval: 90d confidence: high source_of_truth: ssh-libvirt @@ -36,3 +36,49 @@ - Active services observed: `libvirtd`, `virtlogd` See [virtual machine inventory](../../40-inventory/virtual-machines.yml). + +## Storage controller (hpsa) — known failure mode (2026-10-30) + +The HP Smart Array controller (hpsa driver, PCI 0000:04:00.0) can stall and +reset its logical volumes (RAID-1(+0), SSDSmartPathCap). During a reset the +whole volume freezes for tens of seconds; all VM disks on the affected volume +stall, guest load1 spikes to 2-10x cores with CPU near 0% (classic D-state / +I/O-wait signature), and network services behind those VMs time out. + +### Incident 2026-10-30 13:06-13:15 UTC + +- 13:06:17 first hpsa logical-volume reset (scsi 0:1:0:0). +- 13:10:08-13:15:16 reset series on both logical volumes (scsi 0:1:0:0 and + 0:1:0:1), each "reset logical completed successfully" 15-70 s later. +- Panel load_high events in the same window: files #36 (2.5x), gnsynapse + (3.5x), gnauth (2.2x), gncreds (10.5x), hard-panel-176 (9.9x), internal-proxy + cpu_high 93%; hive registry (192.168.1.168:8087) ReadTimeout ~13:10-13:13. +- bserv host load1 peaked ~100 (5-min avg) with CPU mostly idle; no D-state + processes after the stall; no physical-drive failures in the kernel log. +- Self-recovered by ~13:15:30 UTC; no VM restarts needed; all VMs stayed + running. +- Coincidence, not cause: an `ollama pull` of a ~92 GB model (blob + 92907148363e) was running on bserv 13:07-13:11 and stalled its download + parts during the resets; the pull did not survive (no active pull after the + incident). Root disk had only 40G free, so the pull could not complete + anyway. + +### Triage playbook + +1. Confirm the signature: `journalctl -k | grep hpsa` — look for + "resetting logical ... reset logical completed successfully". +2. Check whether resets are ongoing: `journalctl -k --since "" | grep -c hpsa`. +3. Check VM states: `virsh list --all` (VMs normally keep running through a + reset; only their I/O stalls). +4. If resets repeat within hours/days: suspect the controller or its cache + battery; plan a maintenance window (firmware update via ssacli/iLO, or + controller replacement). `ssacli`/`hpssacli` are NOT installed on bserv — + install `ssacli` before the next maintenance window. +5. Escalate to the owner only if resets repeat or a physical drive reports + errors; single self-recovering episodes need no notification (no-noise + policy). + +### History + +- 2026-10-25: 1 hpsa reset (single, self-recovered). +- 2026-10-30: 9 resets in ~9 minutes (the incident above).