Newer
Older
gnexus-book / 10-systems / workstations / gstation-tts.md

owner: gmikcon status: active last_reviewed: 2026-11-03 review_interval: 90d confidence: high

source_of_truth: ssh-host-verified-2026-11-03

gstation-tts (192.168.1.129) — LFM3 TTS training node

Role

Owner workstation used as the GPU training node for the ru/uk TTS SFT project (LFM3-Omni-2B + LoRA). Runs the yocpt harvest→clean→encode→train→eval pipeline. Experiences nightly power cuts (блэкауты) — host may reboot; the machine is sometimes on backup power, so do not start heavy GPU work without owner command at night.

Access

ssh gmikcon@192.168.1.129 (password getroot, sudo same — owner-approved). Connection may drop mid-command if host reboots: treat ssh timeout / rc=255 as "host state unknown", re-ping first.

Project layout

  • P=/home/gmikcon/Projects/lfm3-ru-uk-tts-sft — training repo: train_v8.py, train_v8_launch.sh, .venv/bin/python, logs/train_v8.log, out/lora_v* adapters, out/ckpt_v8 checkpoints, hf_home/ (HF cache).
  • E=/home/gmikcon/Projects/yocpt_encode — pipeline scripts: harvest/clean/texts/encode, cache_v8.npz (disk-backed codes), eval/matrix scripts (gen_v8_eval.py, cer_v8.py, gen_matrix_eval.py, cer_matrix_v9.py, eval_matrix_v9.json), auto_eval_v8.sh watcher, v8_recovery_remote.sh recovery.
  • NFS /mnt/secraid5-cache ← 192.168.1.130:/mnt/secraid5/jellyfin_cache — dataset yocpt_dataset_v8 (24130 clips), backups/ (e.g. lora_v7_backup_20261102.tar.gz).

Traps (verified 2026-11-03)

  1. HF_HOME must be explicit. HF_HUB_OFFLINE=1 without HF_HOME makes transformers look in ~/.cache/huggingface (only configs there) → OSError: ... does not appear to have a file named pytorch_model.bin or model.safetensors. Every script that loads the model/processor must start with os.environ.setdefault("HF_HOME","/home/gmikcon/Projects/lfm3-ru-uk-tts-sft/hf_home"). Already inlined into train_v8.py (line 2), gen_v8_eval.py, cer_v8.py, encode scripts. Check any NEW script.
  2. Model blob is a sparse file. hf_home/hub/models--LiquidAI--LFM3-Omni-2B/blobs/ shows du -sh ≈ 516K but stat size 4.33 GB — weights are fine. Do not "restore" the model based on du. Verify with ls -laL snapshots/*/model.safetensors (real size) or safetensors open.
  3. train_v8.log contains NUL bytes and \r progress frames. Reading progress: tr -d '\000' < logs/train_v8.log | tr '\r' '\n' | grep -oE '[0-9]+/18586' | tail. Plain grep falls into binary mode and shows a stale tail of an old run — looks like progress but is not.
  4. NFS mount is NOT in fstab. After reboot it silently disappears (dataset and backups unreadable). Remount: mkdir -p /mnt/secraid5-cache && sudo mount -t nfs4 192.168.1.130:/mnt/secraid5/jellyfin_cache /mnt/secraid5-cache. Check df -h /mnt/secraid5-cache FIRST after any reboot.
  5. /tmp is wiped on reboot. Never keep patchers/launchers in /tmp; store in E= (persistent) and copy to /tmp only transiently.
  6. pgrep -f self-matches ssh shells. A pattern literal in the ssh command line counts as a process. Use pgrep -af 'pattern' and filter, or bracket-trick '[t]rain_v8.py'.
  7. Eval manifest / markers: train launch waits for CACHE_V8_DONE marker in E/encode_run_v8.out; auto-eval watcher v3 triggers on out/lora_v8/adapter_model.safetensors + absence of train process (marker TRAIN_V8_EXIT is polluted by old runs — do not use).

Training run book (v8, 2026-11-03)

  • Chain: train_v8_launch.sh → digitdups → eval build → train_v8.py --cache cache_v8.npz --merge-adapter out/lora_v7 --lr 1e-4 --epochs 2 → 18586 steps, ~1.05 it/s ≈ 5 h on 24 GB GPU (bf16, batch 4, grad-accum 2).
  • Checkpoints: save_steps=1000, save_total_limit=2 into out/ckpt_v8/checkpoint-*; resume enabled (resume_from_checkpoint=latest). A crash loses ≤1000 steps.
  • Recovery: bash E/v8_recovery_remote.sh — idempotent, branches on live train process, relaunches launch chain + watcher. Recovery runs log to E/recovery_v8.log.
  • Baseline (lora_v7): CER mean 0.4364 (n=10, faster-whisper small int8 + jiwer, min over ref variants). Weak spots: dates digits 0.767, dates words 0.696, float 0.648, money words 0.564.
  • v8 additions vs v7: marked+plain+spellout mix, new harvest clips, word→digit text dups (v4 mechanics).

Generation params (eval scripts)

max_new_tokens=1500, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096; system "Perform TTS.", assistant prefix <|audio_start|>, codec decode 24000 Hz.