owner: gmikcon status: active last_reviewed: 2026-11-03 review_interval: 90d confidence: high
Owner workstation used as the GPU training node for the ru/uk TTS SFT project (LFM3-Omni-2B + LoRA). Runs the yocpt harvest→clean→encode→train→eval pipeline. Experiences nightly power cuts (блэкауты) — host may reboot; the machine is sometimes on backup power, so do not start heavy GPU work without owner command at night.
ssh gmikcon@192.168.1.129 (password getroot, sudo same — owner-approved). Connection may drop mid-command if host reboots: treat ssh timeout / rc=255 as "host state unknown", re-ping first.
P=/home/gmikcon/Projects/lfm3-ru-uk-tts-sft — training repo: train_v8.py, train_v8_launch.sh, .venv/bin/python, logs/train_v8.log, out/lora_v* adapters, out/ckpt_v8 checkpoints, hf_home/ (HF cache).E=/home/gmikcon/Projects/yocpt_encode — pipeline scripts: harvest/clean/texts/encode, cache_v8.npz (disk-backed codes), eval/matrix scripts (gen_v8_eval.py, cer_v8.py, gen_matrix_eval.py, cer_matrix_v9.py, eval_matrix_v9.json), auto_eval_v8.sh watcher, v8_recovery_remote.sh recovery./mnt/secraid5-cache ← 192.168.1.130:/mnt/secraid5/jellyfin_cache — dataset yocpt_dataset_v8 (24130 clips), backups/ (e.g. lora_v7_backup_20261102.tar.gz).HF_HUB_OFFLINE=1 without HF_HOME makes transformers look in ~/.cache/huggingface (only configs there) → OSError: ... does not appear to have a file named pytorch_model.bin or model.safetensors. Every script that loads the model/processor must start with os.environ.setdefault("HF_HOME","/home/gmikcon/Projects/lfm3-ru-uk-tts-sft/hf_home"). Already inlined into train_v8.py (line 2), gen_v8_eval.py, cer_v8.py, encode scripts. Check any NEW script.hf_home/hub/models--LiquidAI--LFM3-Omni-2B/blobs/ shows du -sh ≈ 516K but stat size 4.33 GB — weights are fine. Do not "restore" the model based on du. Verify with ls -laL snapshots/*/model.safetensors (real size) or safetensors open.tr -d '\000' < logs/train_v8.log | tr '\r' '\n' | grep -oE '[0-9]+/18586' | tail. Plain grep falls into binary mode and shows a stale tail of an old run — looks like progress but is not.mkdir -p /mnt/secraid5-cache && sudo mount -t nfs4 192.168.1.130:/mnt/secraid5/jellyfin_cache /mnt/secraid5-cache. Check df -h /mnt/secraid5-cache FIRST after any reboot.E= (persistent) and copy to /tmp only transiently.pgrep -af 'pattern' and filter, or bracket-trick '[t]rain_v8.py'.CACHE_V8_DONE marker in E/encode_run_v8.out; auto-eval watcher v3 triggers on out/lora_v8/adapter_model.safetensors + absence of train process (marker TRAIN_V8_EXIT is polluted by old runs — do not use).train_v8_launch.sh → digitdups → eval build → train_v8.py --cache cache_v8.npz --merge-adapter out/lora_v7 --lr 1e-4 --epochs 2 → 18586 steps, ~1.05 it/s ≈ 5 h on 24 GB GPU (bf16, batch 4, grad-accum 2).save_steps=1000, save_total_limit=2 into out/ckpt_v8/checkpoint-*; resume enabled (resume_from_checkpoint=latest). A crash loses ≤1000 steps.bash E/v8_recovery_remote.sh — idempotent, branches on live train process, relaunches launch chain + watcher. Recovery runs log to E/recovery_v8.log.max_new_tokens=1500, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096; system "Perform TTS.", assistant prefix <|audio_start|>, codec decode 24000 Hz.