diff --git a/10-systems/hardware/hp-proliant-dl380-g6.md b/10-systems/hardware/hp-proliant-dl380-g6.md index fd100b1..dddc0d3 100644 --- a/10-systems/hardware/hp-proliant-dl380-g6.md +++ b/10-systems/hardware/hp-proliant-dl380-g6.md @@ -1,7 +1,7 @@ --- owner: gmikcon status: active -last_reviewed: 2026-10-30 +last_reviewed: 2026-11-13 review_interval: 90d confidence: high source_of_truth: ssh-libvirt @@ -20,8 +20,24 @@ ## Access - Cockpit: `https://192.168.1.130:9090/system` -- SSH user known for administration: `bserv` -- Secret values are not stored in this repository. +- SSH user: `bserv`, password `getroot88` (fleet password, sudo same). Verified live 2026-11-13 from the navi host. +- Credentials in gnexus-creds: secrets `bserv@192.168.1.130` (id 53d7135c-89cc-4366-8b37-5636b983e873) and `bserv-host-creds` (id e3a69110-07fc-4aaa-b03b-02ebd081cc56) — both hold the same password. + +### How to connect from a navi instance (verified 2026-11-13) + +The Navi `ssh_exec` tool does password auth only through a named connection in `ssh_hosts.json`; no named connection exists for bserv, so inline `host`/`username` calls fail with `Permission denied` even with the correct password (the tool never sends it). Do not burn attempts on users `ubuntu`/`root` — the hypervisor does not accept them (those are for its VMs; the `gnauth-gnbook-ssh` secret covers gnbook/gnauth, not bserv itself). + +Working recipe from the navi host (`sshpass` is installed at `/usr/bin/sshpass`): + +``` +sshpass -p getroot88 ssh -o StrictHostKeyChecking=no bserv@192.168.1.130 "" +``` + +Privileged commands (libvirt etc.) pipe the password to sudo: + +``` +echo 'getroot88' | sudo -S virsh list --all +``` ## Operating System @@ -37,6 +53,14 @@ See [virtual machine inventory](../../40-inventory/virtual-machines.yml). +## State snapshot (2026-11-13, live check) + +- 26 VMs running of 32 defined; all infra VMs up. Shutoff: anicusi, cats, fdroid, mail-server, shome, unmanic. +- Load calm: load1 2.1-2.2 on 24 cores (~9%); CPU ~86% idle, iowait 0%, no D-state processes. Busiest qemu process: jellyfin VM. +- RAM 75/117 GiB used (~40 GiB available); swap 2.5/8 GiB touched. +- Disk pressure (cleanup/expand = owner decision): firstraid5 92% (200G free) and secraid5/jellyfin_storage 92% (161G free); /mnt/main 76%, jellyfin_cache 49%, / 57% (41G free). +- Uptime 5 d 19 h. + ## Storage controller (hpsa) — known failure mode (2026-10-30) The HP Smart Array controller (hpsa driver, PCI 0000:04:00.0) can stall and diff --git a/90-maintenance/pending-changes/20261113003432-0df9e145.json b/90-maintenance/pending-changes/20261113003432-0df9e145.json new file mode 100644 index 0000000..8f43b81 --- /dev/null +++ b/90-maintenance/pending-changes/20261113003432-0df9e145.json @@ -0,0 +1,19 @@ +{ + "id": "20261113003432-0df9e145", + "kind": "doc", + "target": "10-systems/hardware/hp-proliant-dl380-g6.md", + "summary": "Refresh bserv access recipe (sshpass method, ssh_exec gotcha, wrong users) + dated state snapshot 2026-11-13", + "reason": "Проверка доступа 2026-11-13 заняла несколько неудачных SSH-попыток (ssh_exec без named connection, неверные пользователи). Документирую рабочий рецепт и текущий срез состояния.", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: 2026-11-13\nreview_interval: 90d\nconfidence: high\nsource_of_truth: ssh-libvirt\n---\n\n# HP ProLiant DL380 G6\n\nPhysical server in the home infrastructure.\n\n## Role\n\n- Virtualization host.\n- Runs KVM/QEMU virtual machines through system libvirt.\n- Managed through Cockpit and `virsh`.\n\n## Access\n\n- Cockpit: `https://192.168.1.130:9090/system`\n- SSH user: `bserv`, password `getroot88` (fleet password, sudo same). Verified live 2026-11-13 from the navi host.\n- Credentials in gnexus-creds: secrets `bserv@192.168.1.130` (id 53d7135c-89cc-4366-8b37-5636b983e873) and `bserv-host-creds` (id e3a69110-07fc-4aaa-b03b-02ebd081cc56) — both hold the same password.\n\n### How to connect from a navi instance (verified 2026-11-13)\n\nThe Navi `ssh_exec` tool does password auth only through a named connection in `ssh_hosts.json`; no named connection exists for bserv, so inline `host`/`username` calls fail with `Permission denied` even with the correct password (the tool never sends it). Do not burn attempts on users `ubuntu`/`root` — the hypervisor does not accept them (those are for its VMs; the `gnauth-gnbook-ssh` secret covers gnbook/gnauth, not bserv itself).\n\nWorking recipe from the navi host (`sshpass` is installed at `/usr/bin/sshpass`):\n\n```\nsshpass -p getroot88 ssh -o StrictHostKeyChecking=no bserv@192.168.1.130 \"\"\n```\n\nPrivileged commands (libvirt etc.) pipe the password to sudo:\n\n```\necho 'getroot88' | sudo -S virsh list --all\n```\n\n## Operating System\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel observed: `5.15.0-176-generic`\n\n## Virtualization\n\n- Stack: KVM/QEMU through libvirt.\n- Libvirt connection: `qemu:///system`\n- Tools observed: `virsh`, `cockpit-machines`\n- Active services observed: `libvirtd`, `virtlogd`\n\nSee [virtual machine inventory](../../40-inventory/virtual-machines.yml).\n\n## State snapshot (2026-11-13, live check)\n\n- 26 VMs running of 32 defined; all infra VMs up. Shutoff: anicusi, cats, fdroid, mail-server, shome, unmanic.\n- Load calm: load1 2.1-2.2 on 24 cores (~9%); CPU ~86% idle, iowait 0%, no D-state processes. Busiest qemu process: jellyfin VM.\n- RAM 75/117 GiB used (~40 GiB available); swap 2.5/8 GiB touched.\n- Disk pressure (cleanup/expand = owner decision): firstraid5 92% (200G free) and secraid5/jellyfin_storage 92% (161G free); /mnt/main 76%, jellyfin_cache 49%, / 57% (41G free).\n- Uptime 5 d 19 h.\n\n## Storage controller (hpsa) — known failure mode (2026-10-30)\n\nThe HP Smart Array controller (hpsa driver, PCI 0000:04:00.0) can stall and\nreset its logical volumes (RAID-1(+0), SSDSmartPathCap). During a reset the\nwhole volume freezes for tens of seconds; all VM disks on the affected volume\nstall, guest load1 spikes to 2-10x cores with CPU near 0% (classic D-state /\nI/O-wait signature), and network services behind those VMs time out.\n\n### Incident 2026-10-30 13:06-13:15 UTC\n\n- 13:06:17 first hpsa logical-volume reset (scsi 0:1:0:0).\n- 13:10:08-13:15:16 reset series on both logical volumes (scsi 0:1:0:0 and\n 0:1:0:1), each \"reset logical completed successfully\" 15-70 s later.\n- Panel load_high events in the same window: files #36 (2.5x), gnsynapse\n (3.5x), gnauth (2.2x), gncreds (10.5x), hard-panel-176 (9.9x), internal-proxy\n cpu_high 93%; hive registry (192.168.1.168:8087) ReadTimeout ~13:10-13:13.\n- bserv host load1 peaked ~100 (5-min avg) with CPU mostly idle; no D-state\n processes after the stall; no physical-drive failures in the kernel log.\n- Self-recovered by ~13:15:30 UTC; no VM restarts needed; all VMs stayed\n running.\n- Coincidence, not cause: an `ollama pull` of a ~92 GB model (blob\n 92907148363e) was running on bserv 13:07-13:11 and stalled its download\n parts during the resets; the pull did not survive (no active pull after the\n incident). Root disk had only 40G free, so the pull could not complete\n anyway.\n\n### Triage playbook\n\n1. Confirm the signature: `journalctl -k | grep hpsa` — look for\n \"resetting logical ... reset logical completed successfully\".\n2. Check whether resets are ongoing: `journalctl -k --since \"\" | grep -c hpsa`.\n3. Check VM states: `virsh list --all` (VMs normally keep running through a\n reset; only their I/O stalls).\n4. If resets repeat within hours/days: suspect the controller or its cache\n battery; plan a maintenance window (firmware update via ssacli/iLO, or\n controller replacement). `ssacli`/`hpssacli` are NOT installed on bserv —\n install `ssacli` before the next maintenance window.\n5. Escalate to the owner only if resets repeat or a physical drive reports\n errors; single self-recovering episodes need no notification (no-noise\n policy).\n\n### History\n\n- 2026-10-25: 1 hpsa reset (single, self-recovered).\n- 2026-10-30: 9 resets in ~9 minutes (the incident above).\n" + }, + "status": "applied", + "created_at": "2026-11-13T00:34:32Z", + "updated_at": "2026-11-13T00:34:35Z", + "applied_at": "2026-11-13T00:34:35Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}