diff --git a/10-systems/bots/gntranscribe.md b/10-systems/bots/gntranscribe.md index 2c03a60..29abdca 100644 --- a/10-systems/bots/gntranscribe.md +++ b/10-systems/bots/gntranscribe.md @@ -84,7 +84,7 @@ - Root cause: python-telegram-bot's `TimedOut` exception from a **Telegram Bot API call** (PTB default read timeout 5 s) — most likely `send_chat_action`/`send_message` around the job. The generic - `except Exception` in `process_job` sends the misleading + `except Exception` in `process_job` sent the misleading "сервис транскрибации недоступен" message for ANY job exception, including Telegram API timeouts. - Service self-recovered; no restart needed; later jobs succeeded. @@ -95,9 +95,27 @@ - Hang up to 600 s with `whisper attempt N/3 failed` warnings → actual whisper-server problem (check bserv-host GPU/Ollama load). -**Proposed (not yet applied) fix:** catch `telegram.error.TimedOut` -separately in `process_job` and send a "попробуй ещё раз" message instead of -the "сервис недоступен" text; optionally raise PTB timeouts. +## Fix applied 2026-09-22 (navi rei, owner-approved) + +Patch to `bot.py` (backup: `bot.py.bak-20260922`, original 16413 B): + +1. `from telegram.error import TimedOut` added to imports. +2. `Application.builder()` now sets all PTB timeouts to 30 s: + `read_timeout`, `write_timeout`, `connect_timeout`, `pool_timeout`, + `get_updates_read_timeout`, `get_updates_write_timeout`, + `get_updates_connect_timeout`, `get_updates_pool_timeout` + (PTB default was 5 s — the source of the false "service unavailable"). +3. `process_job` catches `telegram.error.TimedOut` separately BEFORE the + generic `except Exception`: logs + `Telegram API timed out ... transcription service is fine, user asked to + retry` and sends the user "⏳ Telegram не ответил вовремя — сервис + транскрибации в порядке. Перешли сообщение ещё раз." instead of the + misleading "сервис транскрибации недоступен". + +Verification: `py_compile` OK; `systemctl restart gntranscribe.service` → +active (running), worker started, `GNtranscribe started as @gntranscribe_bot`, +`Application started` (10:34:34 UTC, PID 257995). Rollback: +`cp bot.py.bak-20260922 bot.py && sudo systemctl restart gntranscribe.service`. ## Operations diff --git a/90-maintenance/pending-changes/20260922103500-722ba7f7.json b/90-maintenance/pending-changes/20260922103500-722ba7f7.json new file mode 100644 index 0000000..75b7191 --- /dev/null +++ b/90-maintenance/pending-changes/20260922103500-722ba7f7.json @@ -0,0 +1,19 @@ +{ + "id": "20260922103500-722ba7f7", + "kind": "doc", + "target": "10-systems/bots/gntranscribe.md", + "summary": "gntranscribe: фикс применён — except TimedOut + PTB таймауты 30s, бэкап bot.py.bak-20260922, сервис перезапущен и верифицирован", + "reason": "Фикс применён на tgbots-vm 2026-09-22 (owner-approved): отдельный except TimedOut, PTB таймауты 30s, бэкап bot.py.bak-20260922, сервис перезапущен и верифицирован.", + "payload": { + "content": "---\ntitle: GNtranscribe\ndescription: Telegram bot — audio transcription via whisper-server\nhost: tgbots-vm\nstatus: active\nlast_reviewed: 2026-09-22\nowner: gmikcon\nconfidence: 100\nreview_interval: 90\nsource_of_truth: live_system\n---\n\n# GNtranscribe\n\n## Overview\n\nTelegram bot `@gntranscribe_bot` on VM tg_bots (192.168.1.254). Users forward\nor send voice messages, audio files, video circles or videos; the bot queues\nthem and returns the transcript from whisper-server on bserv-host\n(192.168.1.130:8008, model ggml-large-v3-turbo-q5_0, auto language detect).\n\nDeployed 2026-09-20 by navi instance rei on owner request.\n\n## Design\n\n- Multi-user: incoming media jobs go into an asyncio queue (max 20).\n- Single worker processes jobs one at a time — whisper-server shares a 4 GB\n GPU with Ollama on bserv-host, parallel inference is not safe there.\n- Pipeline per job: download from Telegram CDN via aiohttp (PTB's default\n 5 s read timeout is too slow for the CDN) → convert to 16 kHz mono WAV via\n ffmpeg → POST to whisper → send transcript (chunked at 4000 chars).\n- **whisper-server rejects ogg/opus with HTTP 400 \"Invalid request\"** —\n Telegram voice messages are ogg/opus, hence the mandatory WAV conversion\n (verified 2026-09-20: ogg → 400, wav → exact transcript).\n- Optional chat_id whitelist via `ALLOWED_CHAT_IDS` in `.env`\n (comma-separated; empty = open to everyone). Non-whitelisted users get\n their chat_id back with a deny message.\n- Commands: `/start`, `/help`, `/id` (shows chat_id/user_id for whitelist\n setup), `/status` (queue size, worker state, counters, uptime).\n- Media accepted: voice, audio, video_note (circle), video, audio/video\n documents. Max file size 24 MB (configurable `MAX_FILE_MB`).\n- Whisper calls retried up to 3 attempts on network errors.\n\n## Technical Details\n\n- **Host:** VM tg_bots (192.168.1.254), user `ubuntu`\n- **Path:** `/home/ubuntu/bots/GNtranscribe` — `bot.py`, `.env`,\n `requirements.txt`, `gntranscribe.service`, `venv/`\n- **Stack:** Python 3.10, python-telegram-bot 21.10, aiohttp, ffmpeg 4.4.2\n (system package, required for WAV conversion)\n- **systemd:** `gntranscribe.service` (User=ubuntu, Restart=always,\n RestartSec=30), enabled (multi-user.target.wants)\n- **Config (`.env`):** `BOT_TOKEN` (stored in gnexus-creds secret\n \"GNtranscribe token\", not in docs), `WHISPER_URL=http://192.168.1.130:8008/inference`,\n `MAX_FILE_MB=24`, `MAX_QUEUE=20`, `WHISPER_TIMEOUT=600`,\n `ALLOWED_CHAT_IDS=` (empty)\n- **Logs:** `/home/ubuntu/bots/GNtranscribe/logs/bot.log` (rotating, 2 MB x 3)\n and `journalctl -u gntranscribe.service`\n- **Upstream:** whisper-server on bserv-host — see inventory item\n `bserv-host` (POST /inference, multipart `file=@`)\n\n## Verification 2026-09-20\n\n- `getMe` → ok, `@gntranscribe_bot`.\n- Direct whisper test: TTS-generated Russian audio POSTed to\n http://192.168.1.130:8008/inference returned the exact expected text.\n- Service active after `enable --now`; two consecutive restarts passed.\n- Live-message debugging: first failure was PTB 5 s download timeout\n (fixed with aiohttp download), second was ogg/opus HTTP 400 from\n whisper-server (fixed with ffmpeg WAV conversion; ogg→400, wav→ok verified).\n\n## Incident 2026-09-22 — \"Timed out\" is Telegram API, not whisper\n\nUser reported the bot replying \"не удалось расшифровать, сервис транскрибации\nнедоступен\". Diagnosis (navi rei):\n\n- Two voice jobs failed at 10:03:50 and 10:03:55 UTC:\n `job failed chat=221397150: Timed out` after only 7.2 s / 5.5 s.\n- **Not a whisper outage.** whisper-server was reachable the whole time\n (GET / → HTTP 200 from both navi host and tgbots-vm; POST /inference with\n invalid body → HTTP 400 in 0.009 s). No `whisper attempt N/3 failed`\n retry warnings in the log (whisper retries log them), and the whisper\n timeout is 600 s — a 5–7 s failure cannot be a whisper timeout.\n- Root cause: python-telegram-bot's `TimedOut` exception from a **Telegram\n Bot API call** (PTB default read timeout 5 s) — most likely\n `send_chat_action`/`send_message` around the job. The generic\n `except Exception` in `process_job` sent the misleading\n \"сервис транскрибации недоступен\" message for ANY job exception,\n including Telegram API timeouts.\n- Service self-recovered; no restart needed; later jobs succeeded.\n\n**Known failure mode (triage rule):**\n- `job failed ... Timed out` in ~5–10 s with NO `whisper attempt` warnings\n → Telegram Bot API timeout, transcription service is fine.\n- Hang up to 600 s with `whisper attempt N/3 failed` warnings\n → actual whisper-server problem (check bserv-host GPU/Ollama load).\n\n## Fix applied 2026-09-22 (navi rei, owner-approved)\n\nPatch to `bot.py` (backup: `bot.py.bak-20260922`, original 16413 B):\n\n1. `from telegram.error import TimedOut` added to imports.\n2. `Application.builder()` now sets all PTB timeouts to 30 s:\n `read_timeout`, `write_timeout`, `connect_timeout`, `pool_timeout`,\n `get_updates_read_timeout`, `get_updates_write_timeout`,\n `get_updates_connect_timeout`, `get_updates_pool_timeout`\n (PTB default was 5 s — the source of the false \"service unavailable\").\n3. `process_job` catches `telegram.error.TimedOut` separately BEFORE the\n generic `except Exception`: logs\n `Telegram API timed out ... transcription service is fine, user asked to\n retry` and sends the user \"⏳ Telegram не ответил вовремя — сервис\n транскрибации в порядке. Перешли сообщение ещё раз.\" instead of the\n misleading \"сервис транскрибации недоступен\".\n\nVerification: `py_compile` OK; `systemctl restart gntranscribe.service` →\nactive (running), worker started, `GNtranscribe started as @gntranscribe_bot`,\n`Application started` (10:34:34 UTC, PID 257995). Rollback:\n`cp bot.py.bak-20260922 bot.py && sudo systemctl restart gntranscribe.service`.\n\n## Operations\n\n- Restart: `sudo systemctl restart gntranscribe.service`\n- Enable whitelist: edit `ALLOWED_CHAT_IDS` in `.env`, then restart.\n- Token rotation: update `BOT_TOKEN` in `.env` and the gnexus-creds secret,\n restart." + }, + "status": "applied", + "created_at": "2026-09-22T10:35:00Z", + "updated_at": "2026-09-22T10:35:01Z", + "applied_at": "2026-09-22T10:35:01Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}