---
owner: gmikcon
status: active
last_reviewed: 2026-11-03
review_interval: 90d
confidence: high
source_of_truth: ssh-host-verified-2026-11-03
---

# gstation-tts (192.168.1.129) — LFM3 TTS training node

## Role
Owner workstation used as the GPU training node for the ru/uk TTS SFT project (LFM3-Omni-2B + LoRA). Runs the yocpt harvest→clean→encode→train→eval pipeline. Experiences nightly power cuts (блэкауты) — host may reboot; the machine is sometimes on backup power, so do not start heavy GPU work without owner command at night.

## Access
`ssh gmikcon@192.168.1.129` (password `getroot`, sudo same — owner-approved). Connection may drop mid-command if host reboots: treat ssh timeout / rc=255 as "host state unknown", re-ping first.

## Project layout
- `P=/home/gmikcon/Projects/lfm3-ru-uk-tts-sft` — training repo: `train_v8.py`, `train_v8_launch.sh`, `.venv/bin/python`, `logs/train_v8.log`, `out/lora_v*` adapters, `out/ckpt_v8` checkpoints, `hf_home/` (HF cache).
- `E=/home/gmikcon/Projects/yocpt_encode` — pipeline scripts: harvest/clean/texts/encode, `cache_v8.npz` (disk-backed codes), eval/matrix scripts (`gen_v8_eval.py`, `cer_v8.py`, `gen_matrix_eval.py`, `cer_matrix_v9.py`, `eval_matrix_v9.json`), `auto_eval_v8.sh` watcher, `v8_recovery_remote.sh` recovery.
- NFS `/mnt/secraid5-cache` ← `192.168.1.130:/mnt/secraid5/jellyfin_cache` — dataset `yocpt_dataset_v8` (24130 clips), `backups/` (e.g. `lora_v7_backup_20261102.tar.gz`).

## Traps (verified 2026-11-03)
1. **HF_HOME must be explicit.** `HF_HUB_OFFLINE=1` without `HF_HOME` makes transformers look in `~/.cache/huggingface` (only configs there) → `OSError: ... does not appear to have a file named pytorch_model.bin or model.safetensors`. Every script that loads the model/processor must start with `os.environ.setdefault("HF_HOME","/home/gmikcon/Projects/lfm3-ru-uk-tts-sft/hf_home")`. Already inlined into `train_v8.py` (line 2), `gen_v8_eval.py`, `cer_v8.py`, encode scripts. Check any NEW script.
2. **Model blob is a sparse file.** `hf_home/hub/models--LiquidAI--LFM3-Omni-2B/blobs/` shows `du -sh` ≈ 516K but `stat` size 4.33 GB — weights are fine. Do not "restore" the model based on `du`. Verify with `ls -laL snapshots/*/model.safetensors` (real size) or safetensors open.
3. **train_v8.log contains NUL bytes and \r progress frames.** Reading progress: `tr -d '\000' < logs/train_v8.log | tr '\r' '\n' | grep -oE '[0-9]+/18586' | tail`. Plain grep falls into binary mode and shows a stale tail of an old run — looks like progress but is not.
4. **NFS mount is NOT in fstab.** After reboot it silently disappears (dataset and backups unreadable). Remount: `mkdir -p /mnt/secraid5-cache && sudo mount -t nfs4 192.168.1.130:/mnt/secraid5/jellyfin_cache /mnt/secraid5-cache`. Check `df -h /mnt/secraid5-cache` FIRST after any reboot.
5. **/tmp is wiped on reboot.** Never keep patchers/launchers in /tmp; store in `E=` (persistent) and copy to /tmp only transiently.
6. **pgrep -f self-matches ssh shells.** A pattern literal in the ssh command line counts as a process. Use `pgrep -af 'pattern'` and filter, or bracket-trick `'[t]rain_v8.py'`.
7. **Eval manifest / markers:** train launch waits for `CACHE_V8_DONE` marker in `E/encode_run_v8.out`; auto-eval watcher v3 triggers on `out/lora_v8/adapter_model.safetensors` + absence of train process (marker TRAIN_V8_EXIT is polluted by old runs — do not use).

## Training run book (v8, 2026-11-03)
- Chain: `train_v8_launch.sh` → digitdups → eval build → `train_v8.py --cache cache_v8.npz --merge-adapter out/lora_v7 --lr 1e-4 --epochs 2` → 18586 steps, ~1.05 it/s ≈ 5 h on 24 GB GPU (bf16, batch 4, grad-accum 2).
- Checkpoints: `save_steps=1000, save_total_limit=2` into `out/ckpt_v8/checkpoint-*`; resume enabled (`resume_from_checkpoint=latest`). A crash loses ≤1000 steps.
- Recovery: `bash E/v8_recovery_remote.sh` — idempotent, branches on live train process, relaunches launch chain + watcher. Recovery runs log to `E/recovery_v8.log`.
- Baseline (lora_v7): CER mean 0.4364 (n=10, faster-whisper small int8 + jiwer, min over ref variants). Weak spots: dates digits 0.767, dates words 0.696, float 0.648, money words 0.564.
- v8 additions vs v7: marked+plain+spellout mix, new harvest clips, word→digit text dups (v4 mechanics).

## Generation params (eval scripts)
max_new_tokens=1500, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096; system "Perform TTS.", assistant prefix `<|audio_start|>`, codec decode 24000 Hz.
