diff --git a/10-systems/workstations/gstation-tts.md b/10-systems/workstations/gstation-tts.md new file mode 100644 index 0000000..5b848cf --- /dev/null +++ b/10-systems/workstations/gstation-tts.md @@ -0,0 +1,40 @@ +--- +owner: gmikcon +status: active +last_reviewed: 2026-11-03 +review_interval: 90d +confidence: high +source_of_truth: ssh-host-verified-2026-11-03 +--- + +# gstation-tts (192.168.1.129) — LFM3 TTS training node + +## Role +Owner workstation used as the GPU training node for the ru/uk TTS SFT project (LFM3-Omni-2B + LoRA). Runs the yocpt harvest→clean→encode→train→eval pipeline. Experiences nightly power cuts (блэкауты) — host may reboot; the machine is sometimes on backup power, so do not start heavy GPU work without owner command at night. + +## Access +`ssh gmikcon@192.168.1.129` (password `getroot`, sudo same — owner-approved). Connection may drop mid-command if host reboots: treat ssh timeout / rc=255 as "host state unknown", re-ping first. + +## Project layout +- `P=/home/gmikcon/Projects/lfm3-ru-uk-tts-sft` — training repo: `train_v8.py`, `train_v8_launch.sh`, `.venv/bin/python`, `logs/train_v8.log`, `out/lora_v*` adapters, `out/ckpt_v8` checkpoints, `hf_home/` (HF cache). +- `E=/home/gmikcon/Projects/yocpt_encode` — pipeline scripts: harvest/clean/texts/encode, `cache_v8.npz` (disk-backed codes), eval/matrix scripts (`gen_v8_eval.py`, `cer_v8.py`, `gen_matrix_eval.py`, `cer_matrix_v9.py`, `eval_matrix_v9.json`), `auto_eval_v8.sh` watcher, `v8_recovery_remote.sh` recovery. +- NFS `/mnt/secraid5-cache` ← `192.168.1.130:/mnt/secraid5/jellyfin_cache` — dataset `yocpt_dataset_v8` (24130 clips), `backups/` (e.g. `lora_v7_backup_20261102.tar.gz`). + +## Traps (verified 2026-11-03) +1. **HF_HOME must be explicit.** `HF_HUB_OFFLINE=1` without `HF_HOME` makes transformers look in `~/.cache/huggingface` (only configs there) → `OSError: ... does not appear to have a file named pytorch_model.bin or model.safetensors`. Every script that loads the model/processor must start with `os.environ.setdefault("HF_HOME","/home/gmikcon/Projects/lfm3-ru-uk-tts-sft/hf_home")`. Already inlined into `train_v8.py` (line 2), `gen_v8_eval.py`, `cer_v8.py`, encode scripts. Check any NEW script. +2. **Model blob is a sparse file.** `hf_home/hub/models--LiquidAI--LFM3-Omni-2B/blobs/` shows `du -sh` ≈ 516K but `stat` size 4.33 GB — weights are fine. Do not "restore" the model based on `du`. Verify with `ls -laL snapshots/*/model.safetensors` (real size) or safetensors open. +3. **train_v8.log contains NUL bytes and \r progress frames.** Reading progress: `tr -d '\000' < logs/train_v8.log | tr '\r' '\n' | grep -oE '[0-9]+/18586' | tail`. Plain grep falls into binary mode and shows a stale tail of an old run — looks like progress but is not. +4. **NFS mount is NOT in fstab.** After reboot it silently disappears (dataset and backups unreadable). Remount: `mkdir -p /mnt/secraid5-cache && sudo mount -t nfs4 192.168.1.130:/mnt/secraid5/jellyfin_cache /mnt/secraid5-cache`. Check `df -h /mnt/secraid5-cache` FIRST after any reboot. +5. **/tmp is wiped on reboot.** Never keep patchers/launchers in /tmp; store in `E=` (persistent) and copy to /tmp only transiently. +6. **pgrep -f self-matches ssh shells.** A pattern literal in the ssh command line counts as a process. Use `pgrep -af 'pattern'` and filter, or bracket-trick `'[t]rain_v8.py'`. +7. **Eval manifest / markers:** train launch waits for `CACHE_V8_DONE` marker in `E/encode_run_v8.out`; auto-eval watcher v3 triggers on `out/lora_v8/adapter_model.safetensors` + absence of train process (marker TRAIN_V8_EXIT is polluted by old runs — do not use). + +## Training run book (v8, 2026-11-03) +- Chain: `train_v8_launch.sh` → digitdups → eval build → `train_v8.py --cache cache_v8.npz --merge-adapter out/lora_v7 --lr 1e-4 --epochs 2` → 18586 steps, ~1.05 it/s ≈ 5 h on 24 GB GPU (bf16, batch 4, grad-accum 2). +- Checkpoints: `save_steps=1000, save_total_limit=2` into `out/ckpt_v8/checkpoint-*`; resume enabled (`resume_from_checkpoint=latest`). A crash loses ≤1000 steps. +- Recovery: `bash E/v8_recovery_remote.sh` — idempotent, branches on live train process, relaunches launch chain + watcher. Recovery runs log to `E/recovery_v8.log`. +- Baseline (lora_v7): CER mean 0.4364 (n=10, faster-whisper small int8 + jiwer, min over ref variants). Weak spots: dates digits 0.767, dates words 0.696, float 0.648, money words 0.564. +- v8 additions vs v7: marked+plain+spellout mix, new harvest clips, word→digit text dups (v4 mechanics). + +## Generation params (eval scripts) +max_new_tokens=1500, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096; system "Perform TTS.", assistant prefix `<|audio_start|>`, codec decode 24000 Hz. diff --git a/90-maintenance/pending-changes/20261103103003-7d945292.json b/90-maintenance/pending-changes/20261103103003-7d945292.json new file mode 100644 index 0000000..1adc449 --- /dev/null +++ b/90-maintenance/pending-changes/20261103103003-7d945292.json @@ -0,0 +1,19 @@ +{ + "id": "20261103103003-7d945292", + "kind": "doc", + "target": "10-systems/workstations/gstation-tts.md", + "summary": "Add gstation-tts node doc: pipeline layout, 7 verified traps, v8 run book and CER baseline.", + "reason": "Session 2026-11-03 verified live: HF_HOME trap broke two training launches; sparse blob, NUL logs, NFS-after-reboot and /tmp-wipe traps all hit in one night. Persisting the run book to gnexus-book so future sessions don't rediscover them.", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: 2026-11-03\nreview_interval: 90d\nconfidence: high\nsource_of_truth: ssh-host-verified-2026-11-03\n---\n\n# gstation-tts (192.168.1.129) — LFM3 TTS training node\n\n## Role\nOwner workstation used as the GPU training node for the ru/uk TTS SFT project (LFM3-Omni-2B + LoRA). Runs the yocpt harvest→clean→encode→train→eval pipeline. Experiences nightly power cuts (блэкауты) — host may reboot; the machine is sometimes on backup power, so do not start heavy GPU work without owner command at night.\n\n## Access\n`ssh gmikcon@192.168.1.129` (password `getroot`, sudo same — owner-approved). Connection may drop mid-command if host reboots: treat ssh timeout / rc=255 as \"host state unknown\", re-ping first.\n\n## Project layout\n- `P=/home/gmikcon/Projects/lfm3-ru-uk-tts-sft` — training repo: `train_v8.py`, `train_v8_launch.sh`, `.venv/bin/python`, `logs/train_v8.log`, `out/lora_v*` adapters, `out/ckpt_v8` checkpoints, `hf_home/` (HF cache).\n- `E=/home/gmikcon/Projects/yocpt_encode` — pipeline scripts: harvest/clean/texts/encode, `cache_v8.npz` (disk-backed codes), eval/matrix scripts (`gen_v8_eval.py`, `cer_v8.py`, `gen_matrix_eval.py`, `cer_matrix_v9.py`, `eval_matrix_v9.json`), `auto_eval_v8.sh` watcher, `v8_recovery_remote.sh` recovery.\n- NFS `/mnt/secraid5-cache` ← `192.168.1.130:/mnt/secraid5/jellyfin_cache` — dataset `yocpt_dataset_v8` (24130 clips), `backups/` (e.g. `lora_v7_backup_20261102.tar.gz`).\n\n## Traps (verified 2026-11-03)\n1. **HF_HOME must be explicit.** `HF_HUB_OFFLINE=1` without `HF_HOME` makes transformers look in `~/.cache/huggingface` (only configs there) → `OSError: ... does not appear to have a file named pytorch_model.bin or model.safetensors`. Every script that loads the model/processor must start with `os.environ.setdefault(\"HF_HOME\",\"/home/gmikcon/Projects/lfm3-ru-uk-tts-sft/hf_home\")`. Already inlined into `train_v8.py` (line 2), `gen_v8_eval.py`, `cer_v8.py`, encode scripts. Check any NEW script.\n2. **Model blob is a sparse file.** `hf_home/hub/models--LiquidAI--LFM3-Omni-2B/blobs/` shows `du -sh` ≈ 516K but `stat` size 4.33 GB — weights are fine. Do not \"restore\" the model based on `du`. Verify with `ls -laL snapshots/*/model.safetensors` (real size) or safetensors open.\n3. **train_v8.log contains NUL bytes and \\r progress frames.** Reading progress: `tr -d '\\000' < logs/train_v8.log | tr '\\r' '\\n' | grep -oE '[0-9]+/18586' | tail`. Plain grep falls into binary mode and shows a stale tail of an old run — looks like progress but is not.\n4. **NFS mount is NOT in fstab.** After reboot it silently disappears (dataset and backups unreadable). Remount: `mkdir -p /mnt/secraid5-cache && sudo mount -t nfs4 192.168.1.130:/mnt/secraid5/jellyfin_cache /mnt/secraid5-cache`. Check `df -h /mnt/secraid5-cache` FIRST after any reboot.\n5. **/tmp is wiped on reboot.** Never keep patchers/launchers in /tmp; store in `E=` (persistent) and copy to /tmp only transiently.\n6. **pgrep -f self-matches ssh shells.** A pattern literal in the ssh command line counts as a process. Use `pgrep -af 'pattern'` and filter, or bracket-trick `'[t]rain_v8.py'`.\n7. **Eval manifest / markers:** train launch waits for `CACHE_V8_DONE` marker in `E/encode_run_v8.out`; auto-eval watcher v3 triggers on `out/lora_v8/adapter_model.safetensors` + absence of train process (marker TRAIN_V8_EXIT is polluted by old runs — do not use).\n\n## Training run book (v8, 2026-11-03)\n- Chain: `train_v8_launch.sh` → digitdups → eval build → `train_v8.py --cache cache_v8.npz --merge-adapter out/lora_v7 --lr 1e-4 --epochs 2` → 18586 steps, ~1.05 it/s ≈ 5 h on 24 GB GPU (bf16, batch 4, grad-accum 2).\n- Checkpoints: `save_steps=1000, save_total_limit=2` into `out/ckpt_v8/checkpoint-*`; resume enabled (`resume_from_checkpoint=latest`). A crash loses ≤1000 steps.\n- Recovery: `bash E/v8_recovery_remote.sh` — idempotent, branches on live train process, relaunches launch chain + watcher. Recovery runs log to `E/recovery_v8.log`.\n- Baseline (lora_v7): CER mean 0.4364 (n=10, faster-whisper small int8 + jiwer, min over ref variants). Weak spots: dates digits 0.767, dates words 0.696, float 0.648, money words 0.564.\n- v8 additions vs v7: marked+plain+spellout mix, new harvest clips, word→digit text dups (v4 mechanics).\n\n## Generation params (eval scripts)\nmax_new_tokens=1500, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096; system \"Perform TTS.\", assistant prefix `<|audio_start|>`, codec decode 24000 Hz." + }, + "status": "applied", + "created_at": "2026-11-03T10:30:03Z", + "updated_at": "2026-11-03T10:30:05Z", + "applied_at": "2026-11-03T10:30:05Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}