diff --git a/10-systems/bots/gntranscribe.md b/10-systems/bots/gntranscribe.md index fd5e308..2c03a60 100644 --- a/10-systems/bots/gntranscribe.md +++ b/10-systems/bots/gntranscribe.md @@ -3,7 +3,7 @@ description: Telegram bot — audio transcription via whisper-server host: tgbots-vm status: active -last_reviewed: 2026-09-20 +last_reviewed: 2026-09-22 owner: gmikcon confidence: 100 review_interval: 90 @@ -69,6 +69,36 @@ (fixed with aiohttp download), second was ogg/opus HTTP 400 from whisper-server (fixed with ffmpeg WAV conversion; ogg→400, wav→ok verified). +## Incident 2026-09-22 — "Timed out" is Telegram API, not whisper + +User reported the bot replying "не удалось расшифровать, сервис транскрибации +недоступен". Diagnosis (navi rei): + +- Two voice jobs failed at 10:03:50 and 10:03:55 UTC: + `job failed chat=221397150: Timed out` after only 7.2 s / 5.5 s. +- **Not a whisper outage.** whisper-server was reachable the whole time + (GET / → HTTP 200 from both navi host and tgbots-vm; POST /inference with + invalid body → HTTP 400 in 0.009 s). No `whisper attempt N/3 failed` + retry warnings in the log (whisper retries log them), and the whisper + timeout is 600 s — a 5–7 s failure cannot be a whisper timeout. +- Root cause: python-telegram-bot's `TimedOut` exception from a **Telegram + Bot API call** (PTB default read timeout 5 s) — most likely + `send_chat_action`/`send_message` around the job. The generic + `except Exception` in `process_job` sends the misleading + "сервис транскрибации недоступен" message for ANY job exception, + including Telegram API timeouts. +- Service self-recovered; no restart needed; later jobs succeeded. + +**Known failure mode (triage rule):** +- `job failed ... Timed out` in ~5–10 s with NO `whisper attempt` warnings + → Telegram Bot API timeout, transcription service is fine. +- Hang up to 600 s with `whisper attempt N/3 failed` warnings + → actual whisper-server problem (check bserv-host GPU/Ollama load). + +**Proposed (not yet applied) fix:** catch `telegram.error.TimedOut` +separately in `process_job` and send a "попробуй ещё раз" message instead of +the "сервис недоступен" text; optionally raise PTB timeouts. + ## Operations - Restart: `sudo systemctl restart gntranscribe.service` diff --git a/90-maintenance/pending-changes/20260922102949-064c401d.json b/90-maintenance/pending-changes/20260922102949-064c401d.json new file mode 100644 index 0000000..cda785e --- /dev/null +++ b/90-maintenance/pending-changes/20260922102949-064c401d.json @@ -0,0 +1,19 @@ +{ + "id": "20260922102949-064c401d", + "kind": "doc", + "target": "10-systems/bots/gntranscribe.md", + "summary": "gntranscribe: задокументирован инцидент 2026-09-22 — «Timed out» был таймаутом Telegram API, не whisper; добавлено правило триажа", + "reason": "Инцидент 2026-09-22: ошибка «сервис транскрибации недоступен» была вызвана таймаутом Telegram Bot API (PTB TimedOut), а не whisper-server. Зафиксирован known failure mode и правило триажа по логам.", + "payload": { + "content": "---\ntitle: GNtranscribe\ndescription: Telegram bot — audio transcription via whisper-server\nhost: tgbots-vm\nstatus: active\nlast_reviewed: 2026-09-22\nowner: gmikcon\nconfidence: 100\nreview_interval: 90\nsource_of_truth: live_system\n---\n\n# GNtranscribe\n\n## Overview\n\nTelegram bot `@gntranscribe_bot` on VM tg_bots (192.168.1.254). Users forward\nor send voice messages, audio files, video circles or videos; the bot queues\nthem and returns the transcript from whisper-server on bserv-host\n(192.168.1.130:8008, model ggml-large-v3-turbo-q5_0, auto language detect).\n\nDeployed 2026-09-20 by navi instance rei on owner request.\n\n## Design\n\n- Multi-user: incoming media jobs go into an asyncio queue (max 20).\n- Single worker processes jobs one at a time — whisper-server shares a 4 GB\n GPU with Ollama on bserv-host, parallel inference is not safe there.\n- Pipeline per job: download from Telegram CDN via aiohttp (PTB's default\n 5 s read timeout is too slow for the CDN) → convert to 16 kHz mono WAV via\n ffmpeg → POST to whisper → send transcript (chunked at 4000 chars).\n- **whisper-server rejects ogg/opus with HTTP 400 \"Invalid request\"** —\n Telegram voice messages are ogg/opus, hence the mandatory WAV conversion\n (verified 2026-09-20: ogg → 400, wav → exact transcript).\n- Optional chat_id whitelist via `ALLOWED_CHAT_IDS` in `.env`\n (comma-separated; empty = open to everyone). Non-whitelisted users get\n their chat_id back with a deny message.\n- Commands: `/start`, `/help`, `/id` (shows chat_id/user_id for whitelist\n setup), `/status` (queue size, worker state, counters, uptime).\n- Media accepted: voice, audio, video_note (circle), video, audio/video\n documents. Max file size 24 MB (configurable `MAX_FILE_MB`).\n- Whisper calls retried up to 3 attempts on network errors.\n\n## Technical Details\n\n- **Host:** VM tg_bots (192.168.1.254), user `ubuntu`\n- **Path:** `/home/ubuntu/bots/GNtranscribe` — `bot.py`, `.env`,\n `requirements.txt`, `gntranscribe.service`, `venv/`\n- **Stack:** Python 3.10, python-telegram-bot 21.10, aiohttp, ffmpeg 4.4.2\n (system package, required for WAV conversion)\n- **systemd:** `gntranscribe.service` (User=ubuntu, Restart=always,\n RestartSec=30), enabled (multi-user.target.wants)\n- **Config (`.env`):** `BOT_TOKEN` (stored in gnexus-creds secret\n \"GNtranscribe token\", not in docs), `WHISPER_URL=http://192.168.1.130:8008/inference`,\n `MAX_FILE_MB=24`, `MAX_QUEUE=20`, `WHISPER_TIMEOUT=600`,\n `ALLOWED_CHAT_IDS=` (empty)\n- **Logs:** `/home/ubuntu/bots/GNtranscribe/logs/bot.log` (rotating, 2 MB x 3)\n and `journalctl -u gntranscribe.service`\n- **Upstream:** whisper-server on bserv-host — see inventory item\n `bserv-host` (POST /inference, multipart `file=@`)\n\n## Verification 2026-09-20\n\n- `getMe` → ok, `@gntranscribe_bot`.\n- Direct whisper test: TTS-generated Russian audio POSTed to\n http://192.168.1.130:8008/inference returned the exact expected text.\n- Service active after `enable --now`; two consecutive restarts passed.\n- Live-message debugging: first failure was PTB 5 s download timeout\n (fixed with aiohttp download), second was ogg/opus HTTP 400 from\n whisper-server (fixed with ffmpeg WAV conversion; ogg→400, wav→ok verified).\n\n## Incident 2026-09-22 — \"Timed out\" is Telegram API, not whisper\n\nUser reported the bot replying \"не удалось расшифровать, сервис транскрибации\nнедоступен\". Diagnosis (navi rei):\n\n- Two voice jobs failed at 10:03:50 and 10:03:55 UTC:\n `job failed chat=221397150: Timed out` after only 7.2 s / 5.5 s.\n- **Not a whisper outage.** whisper-server was reachable the whole time\n (GET / → HTTP 200 from both navi host and tgbots-vm; POST /inference with\n invalid body → HTTP 400 in 0.009 s). No `whisper attempt N/3 failed`\n retry warnings in the log (whisper retries log them), and the whisper\n timeout is 600 s — a 5–7 s failure cannot be a whisper timeout.\n- Root cause: python-telegram-bot's `TimedOut` exception from a **Telegram\n Bot API call** (PTB default read timeout 5 s) — most likely\n `send_chat_action`/`send_message` around the job. The generic\n `except Exception` in `process_job` sends the misleading\n \"сервис транскрибации недоступен\" message for ANY job exception,\n including Telegram API timeouts.\n- Service self-recovered; no restart needed; later jobs succeeded.\n\n**Known failure mode (triage rule):**\n- `job failed ... Timed out` in ~5–10 s with NO `whisper attempt` warnings\n → Telegram Bot API timeout, transcription service is fine.\n- Hang up to 600 s with `whisper attempt N/3 failed` warnings\n → actual whisper-server problem (check bserv-host GPU/Ollama load).\n\n**Proposed (not yet applied) fix:** catch `telegram.error.TimedOut`\nseparately in `process_job` and send a \"попробуй ещё раз\" message instead of\nthe \"сервис недоступен\" text; optionally raise PTB timeouts.\n\n## Operations\n\n- Restart: `sudo systemctl restart gntranscribe.service`\n- Enable whitelist: edit `ALLOWED_CHAT_IDS` in `.env`, then restart.\n- Token rotation: update `BOT_TOKEN` in `.env` and the gnexus-creds secret,\n restart." + }, + "status": "applied", + "created_at": "2026-09-22T10:29:49Z", + "updated_at": "2026-09-22T10:29:50Z", + "applied_at": "2026-09-22T10:29:50Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}