diff --git a/deploy/UPDATE.md b/deploy/UPDATE.md
index 88bde41..2fc488b 100644
--- a/deploy/UPDATE.md
+++ b/deploy/UPDATE.md
@@ -77,6 +77,8 @@
git fetch origin
git pull --ff-only # on deploy: git reset --hard origin/deploy
bash deploy/install.sh # ONLY if pyproject.toml changed since the deployed commit
+# ...and it does NOT touch mcp-servers/*/.venv: an MCP server whose own
+# pyproject.toml changed needs its install too (docs/mcp.md#the-servers-own-venv)
sudo systemctl restart navi
systemctl status navi --no-pager
journalctl -u navi -n 50 --no-pager
diff --git a/docs/mcp.md b/docs/mcp.md
index e0d1806..31e9889 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -61,6 +61,11 @@
unauthenticated request as an anonymous user instead of returning 401, so
silently dropping would fail open. `McpManager.load_all` isolates a failing
server, so this shows up as one broken entry in `/mcp/status`.
+- **The explicit escape hatch is `${NAME:-fallback}`** (fallback may be empty).
+ It is for a *setting* that tunes a server — a SearXNG URL — where being unset
+ costs one backend instead of every tool the server provides. Writing the
+ fallback down is the point: it is a decision in the config, not an accident of
+ a variable nobody set. A credential never takes this form.
- **Resolution happens at transport open** (`navi/mcp/client.py`), next to the
project-relative path resolution. This is what keeps every stored and
serialised config placeholder-only: `save_mcp_servers` — reached by
@@ -70,6 +75,92 @@
A user key (`user_key`) is injected first and overwrites the header, so a
per-user credential simply replaces the placeholder — no substitution happens.
+### `navi-web` settings
+
+The search/browse server has no user key, but it still has runtime settings of its
+own, and they reach it through the same placeholder mechanism. A stdio server does
+**not** inherit the Navi process's environment: the SDK passes a safe allowlist
+(`HOME`, `PATH`, `USER`, …) plus the config's `env` block, so a variable that is
+only in `.env` is invisible to it unless the config maps it.
+
+| Config `env` key | `.env` variable | Meaning |
+|---|---|---|
+| `NAVI_WEB_MCP_TRANSPORT` | — | `stdio`, declared inline |
+| `NAVI_WEB_SEARXNG_URL` | `SEARXNG_URL` (via `${SEARXNG_URL:-}`) | Self-hosted SearXNG base URL — the primary backend |
+| `NAVI_WEB_BRAVE_API_KEY` | `BRAVE_SEARCH_API_KEY` | Optional third backend. Add the placeholder only once the key exists: an unset placeholder without a fallback fails the connection (see above) |
+| `NAVI_WEB_DDG_REGION` | — | DuckDuckGo region, default `ua-uk` |
+
+Without `NAVI_WEB_SEARXNG_URL` the server starts straight from DuckDuckGo HTML
+scraping — roughly an order of magnitude slower, and the first backend to be
+rate-limited. The `${SEARXNG_URL:-}` fallback is what keeps a host with no
+SearXNG working: an unset value costs the backend, not the server's three tools.
+`navi.config.Settings` has no search fields: `SEARXNG_URL` is read here, not by
+Navi itself.
+
+#### Batching is the latency lever
+
+`web_search` takes one query or up to three, `web_view` one URL or up to three;
+several run concurrently, so the call costs the slowest item instead of the sum
+— three pages measured 17.3 s one by one and 1.9 s together, three searches
+3.5 s and 0.6 s. The caps live in the server: three items, deduplicated, extras
+dropped with a note in the output.
+
+The model is pointed at the feature twice — by the tool descriptions, which
+travel in every request, and by `summary` in `mcp_servers.d/navi-web.json`,
+which is the line that reaches the system prompt (`instructions` carries the
+long form for `tool_manual`). The wording is a trigger, not a speed plea: batch
+when the lookups are already known to be independent, never a follow-up whose
+query depends on the previous answer.
+
+A per-item deadline is deliberate. One hung backend or page is named in the
+output rather than erasing what the others already returned, and the tool
+raises — `isError` in the journal — only when nothing at all came back.
+
+This is not Navi's `parallel_tool_calls`: that flag runs a whole batch of
+independent tool calls concurrently for *every* tool of a profile, `terminal`
+included. The server-side fan-out is scoped to these two tools, and its caps and
+budgets are enforced where the work happens.
+
+#### What `web_view` hands back
+
+The page is rendered in Chromium, then `trafilatura` picks the main content out
+by text density and the server renders that fragment as markdown: `#` headings,
+fenced code with its own indentation, table rows joined by `|`. This replaced
+`innerText` after a tag kill-list, which could not tell navigation from prose —
+462 of the asyncio page's 1018 lines were shorter than 40 characters — collapsed
+tables into one line per thousand characters (news.ycombinator.com came back as
+four lines), marked no headings and dropped every link.
+
+Links become a `Links on this page` appendix rather than `[label](url)` inline:
+at most 40, absolute, deduplicated, with self-anchors and `hide`/`vote`/`login`
+chrome filtered out. Inline links cost 3.5k characters on the docs page alone,
+and `trafilatura` drops the `#` from any heading that carries an anchor when they
+are on. Text past 20k characters is cut at a paragraph boundary, and the note
+names the sections that did not fit.
+
+A page whose content *is* its thread — a forum, a Q&A, a code review — comes
+back as a bare header when comments are excluded, so when the extractor keeps
+less than a quarter of the page's visible text it runs again with comments. If
+it still finds nothing worth sending, the tool falls back to the visible-text
+read; `metadata.extractor` says which path answered (`trafilatura` or
+`innerText`), and `metadata.links` counts the appendix.
+
+#### The server's own venv
+
+`navi-web` runs from `mcp-servers/navi-web/.venv`, which is separate from Navi's:
+`./mcp-servers/navi-web/.venv/bin/python -m app.mcp_server` with its working
+directory there. `deploy/install.sh` rebuilds only Navi's venv, so a dependency
+added to `mcp-servers/navi-web/pyproject.toml` — `trafilatura` today — needs its
+own install on the host:
+
+```bash
+VIRTUAL_ENV=mcp-servers/navi-web/.venv uv pip install -e mcp-servers/navi-web
+sudo systemctl restart navi
+```
+
+Without it the module does not import and the server fails to start, which
+`/mcp/status` shows as a disconnected server rather than as degraded output.
+
## Runtime
- `McpManager` (`navi/mcp/manager.py`) holds one `McpClient` per server,
diff --git a/docs/tools.md b/docs/tools.md
index 3500948..54d9556 100644
--- a/docs/tools.md
+++ b/docs/tools.md
@@ -14,8 +14,8 @@
| Server | Tool name | Description |
|---|---|---|
-| `navi-web` | `mcp__navi-web__web_search` | Web search (SearXNG primary, DDG fallback, Brave tertiary) |
-| `navi-web` | `mcp__navi-web__web_view` | Open a URL in a headless browser and return clean readable text |
+| `navi-web` | `mcp__navi-web__web_search` | Web search (SearXNG primary, DDG fallback, Brave tertiary). One query or up to 3, run in parallel |
+| `navi-web` | `mcp__navi-web__web_view` | Open a URL in a headless browser and return the main content as markdown plus a links appendix. One URL or up to 3, opened in parallel under a shared 40k-char text budget |
| `navi-web` | `mcp__navi-web__http_request` | Raw HTTP request (GET/POST/PUT/PATCH/DELETE) |
| `navi-3d` | `mcp__navi-3d__compile_scad` | Compile an OpenSCAD script into a binary STL file |
| `navi-3d` | `mcp__navi-3d__lint_scad` | Lightweight OpenSCAD source linting before STL compilation |
@@ -31,6 +31,9 @@
MCP tools survive `reload_tools` because they are registered as external tools in `ToolRegistry`.
MCP configs, transports and per-user credentials (BYOK) are covered in [`mcp.md`](mcp.md).
+`navi-web` takes its search backends from `SEARXNG_URL` in `.env` (mapped into the
+server's environment by `mcp_servers.d/navi-web.json`); without it the chain starts at
+DuckDuckGo scraping — see [`mcp.md`](mcp.md#navi-web-settings).
| Tool | Name | Description |
|---|---|---|
diff --git a/mcp-servers/navi-web/app/browse.py b/mcp-servers/navi-web/app/browse.py
index 9d37833..190c40a 100644
--- a/mcp-servers/navi-web/app/browse.py
+++ b/mcp-servers/navi-web/app/browse.py
@@ -2,40 +2,238 @@
from __future__ import annotations
+import asyncio
import base64
-import re
+from contextlib import suppress
from typing import Any
from playwright.async_api import async_playwright
-_TIMEOUT = 30_000
+from .extract import clean_visible_text, extract_page, truncate
+
+_TIMEOUT = 20_000
_MAX_TEXT = 20_000
+# A call may open several pages at once: pages take 1-8s each, so a batch costs
+# the slowest instead of the sum (three URLs here: 17.3s one by one, 1.9s
+# together). The shared text budget is the counterweight — three pages at the
+# single-page cap would be 60k chars of context, so a batch stops at 40k and
+# says where it stopped.
+_MAX_BATCH = 3
+_BATCH_TEXT = 40_000
+_BATCH_DEADLINE = 30.0
-def _clean(text: str) -> str:
- lines = [line.rstrip() for line in text.splitlines()]
- result: list[str] = []
- blank_run = 0
- for line in lines:
- if line == "":
- blank_run += 1
- if blank_run <= 2:
- result.append("")
- else:
- blank_run = 0
- result.append(line)
- return "\n".join(result).strip()
+# One Chromium for the life of the server. Launching a browser per call cost ~5s
+# even for ``example.com``; this process is long-lived, so the browser can be
+# too. Each call still gets a fresh context, which is what isolates cookies,
+# storage and the page from its neighbours.
+_playwright = None
+_browser = None
+_browser_lock = asyncio.Lock()
+
+# Contexts are cheap but not free; the cap keeps a burst of concurrent calls
+# (a batch, or two tool calls Navi ran in parallel) from launching a crowd.
+_view_slots = asyncio.Semaphore(_MAX_BATCH)
+
+
+def _normalize(raw: str | list[str]) -> tuple[list[str], list[str]]:
+ """Turn the argument into the URLs to open, plus notes worth telling the model."""
+ urls: list[str] = []
+ for item in [raw] if isinstance(raw, str) else raw:
+ url = str(item).strip()
+ if url and url not in urls:
+ urls.append(url)
+ notes: list[str] = []
+ if len(urls) > _MAX_BATCH:
+ notes.append(
+ f"Only the first {_MAX_BATCH} of {len(urls)} URLs were opened — "
+ "ask the rest in another call."
+ )
+ urls = urls[:_MAX_BATCH]
+ return urls, notes
+
+
+
+async def _get_browser():
+ """Return the shared browser, launching (or relaunching) it if needed."""
+ global _playwright, _browser
+ async with _browser_lock:
+ if _browser is not None and _browser.is_connected():
+ return _browser
+ # A browser that crashed leaves a Playwright driver behind; stop it
+ # before starting another, or the driver processes accumulate.
+ if _playwright is not None:
+ with suppress(Exception):
+ await _playwright.stop()
+ _playwright = None
+ _browser = None
+ _playwright = await async_playwright().start()
+ _browser = await _playwright.chromium.launch(headless=True)
+ return _browser
+
+
+async def close_browser() -> None:
+ """Stop Chromium and its Playwright driver. Called from the server lifespan."""
+ global _playwright, _browser
+ async with _browser_lock:
+ if _browser is not None:
+ with suppress(Exception):
+ await _browser.close()
+ _browser = None
+ if _playwright is not None:
+ with suppress(Exception):
+ await _playwright.stop()
+ _playwright = None
+
+
+# The page's own visible text, without the chrome that never had a reader.
+_VISIBLE_TEXT = """() => {
+ const kill = ['script','style','noscript','iframe',
+ 'nav','header','footer','aside',
+ '[role="navigation"]','[role="banner"]',
+ '[role="contentinfo"]'];
+ const clone = document.body.cloneNode(true);
+ kill.forEach(sel => {
+ clone.querySelectorAll(sel).forEach(el => el.remove());
+ });
+ return clone.innerText || clone.textContent || '';
+}"""
async def web_view(
+ url: str | list[str],
+ screenshot: bool = False,
+ wait_until: str = "networkidle",
+) -> dict[str, Any]:
+ """Open a URL — or up to three — in a headless browser and return readable text.
+
+ Returns dict with success, output, error, metadata.
+ """
+ urls, notes = _normalize(url)
+ if not urls:
+ return {
+ "success": False,
+ "output": "No URL given.",
+ "error": "empty_url",
+ "metadata": {},
+ }
+ if screenshot and len(urls) > 1:
+ return {
+ "success": False,
+ "output": (
+ "screenshot works on one URL at a time — ask for a single URL "
+ "with screenshot, or several without."
+ ),
+ "error": "screenshot_batch",
+ "metadata": {},
+ }
+
+ if len(urls) == 1 and not notes:
+ return await _view_bounded(urls[0], screenshot=screenshot, wait_until=wait_until)
+
+ outcomes = await asyncio.gather(
+ *(_view_bounded(u, wait_until=wait_until) for u in urls),
+ return_exceptions=True,
+ )
+ return _combine(urls, outcomes, notes)
+
+
+async def _view_bounded(
url: str,
screenshot: bool = False,
wait_until: str = "networkidle",
) -> dict[str, Any]:
- """Open a URL in a headless browser and return readable text.
+ """One page, bounded on its own.
- Returns dict with success, output, error, metadata.
+ The deadline is per URL: a batch waits for its slowest page, and a single
+ ceiling around the whole batch would let one hung page erase the others
+ that already loaded.
"""
+ try:
+ return await asyncio.wait_for(
+ _view_one(url, screenshot=screenshot, wait_until=wait_until),
+ timeout=_BATCH_DEADLINE,
+ )
+ except TimeoutError:
+ return {
+ "success": False,
+ "output": f"No answer within {_BATCH_DEADLINE:.0f}s: {url}",
+ "error": "timeout",
+ "metadata": {},
+ }
+
+
+def _combine(
+ urls: list[str],
+ outcomes: list[Any],
+ notes: list[str],
+) -> dict[str, Any]:
+ """One block per URL, sharing one text budget.
+
+ A URL that failed is named rather than dropped — the model can retry that
+ one page instead of repeating the batch — and a batch that opened something
+ is a success even when the rest did not.
+ """
+ blocks: list[str] = []
+ per_url: list[dict[str, Any]] = []
+ budget = _BATCH_TEXT
+ opened = 0
+
+ for url, outcome in zip(urls, outcomes, strict=True):
+ if isinstance(outcome, BaseException):
+ blocks.append(f"### {url} -> failed: {outcome}")
+ per_url.append({"url": url, "success": False, "error": str(outcome)})
+ continue
+ if not outcome["success"]:
+ blocks.append(f"### {url} -> failed: {outcome['output']}")
+ per_url.append({"url": url, "success": False, "error": outcome["error"]})
+ continue
+
+ opened += 1
+ block = f"### {url}\n{outcome['output']}"
+ if budget <= 0:
+ block = (
+ f"### {url}\n[… not shown: the batch's "
+ f"{_BATCH_TEXT // 1000}k-char text budget is used up]"
+ )
+ per_url.append({"url": url, "success": True, "truncated": True})
+ blocks.append(block)
+ continue
+ if len(block) > budget:
+ block = (
+ block[:budget]
+ + f"\n\n[… truncated: the batch's {_BATCH_TEXT // 1000}k-char "
+ "text budget is used up]"
+ )
+ per_url.append({"url": url, "success": True, "truncated": True})
+ budget = 0
+ else:
+ per_url.append({"url": url, "success": True})
+ budget -= len(block)
+ blocks.append(block)
+
+ body = "\n\n".join(notes + blocks)
+ if not opened:
+ return {
+ "success": False,
+ "output": body or "No URL could be opened.",
+ "error": "all_urls_failed",
+ "metadata": {"urls": per_url},
+ }
+ return {
+ "success": True,
+ "output": body,
+ "error": None,
+ "metadata": {"urls": per_url},
+ }
+
+
+async def _view_one(
+ url: str,
+ screenshot: bool = False,
+ wait_until: str = "networkidle",
+) -> dict[str, Any]:
+ """Open one page in its own context. Returns dict with success, output, error, metadata."""
if not url.startswith(("http://", "https://")):
return {
"success": False,
@@ -45,8 +243,8 @@
}
try:
- async with async_playwright() as pw:
- browser = await pw.chromium.launch(headless=True)
+ async with _view_slots:
+ browser = await _get_browser()
context = await browser.new_context(
viewport={"width": 1280, "height": 800},
user_agent=(
@@ -54,59 +252,11 @@
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
),
)
- page = await context.new_page()
-
try:
- await page.goto(url, wait_until=wait_until, timeout=_TIMEOUT)
- except Exception:
- pass
-
- title = await page.title()
- final_url = page.url
-
- text = await page.evaluate("""() => {
- const kill = ['script','style','noscript','iframe',
- 'nav','header','footer','aside',
- '[role="navigation"]','[role="banner"]',
- '[role="contentinfo"]'];
- const clone = document.body.cloneNode(true);
- kill.forEach(sel => {
- clone.querySelectorAll(sel).forEach(el => el.remove());
- });
- return clone.innerText || clone.textContent || '';
- }""")
-
- text = _clean(text)
- if len(text) > _MAX_TEXT:
- text = text[:_MAX_TEXT] + f"\n\n[… truncated at {_MAX_TEXT} chars]"
-
- output_parts = []
- if title:
- output_parts.append(f"Title: {title}")
- if final_url != url:
- output_parts.append(f"Final URL: {final_url}")
- output_parts.append("")
- output_parts.append(text)
- output = "\n".join(output_parts)
-
- metadata: dict[str, Any] = {}
- if screenshot:
- png = await page.screenshot(full_page=False)
- metadata = {
- "base64": base64.b64encode(png).decode(),
- "mime": "image/png",
- "is_image": True,
- }
-
- await context.close()
- await browser.close()
-
- return {
- "success": True,
- "output": output,
- "error": None,
- "metadata": metadata or {},
- }
+ return await _read_page(context, url, screenshot, wait_until)
+ finally:
+ with suppress(Exception):
+ await context.close()
except Exception as e:
return {
@@ -115,3 +265,59 @@
"error": str(e),
"metadata": {},
}
+
+
+async def _read_page(
+ context: Any,
+ url: str,
+ screenshot: bool,
+ wait_until: str,
+) -> dict[str, Any]:
+ page = await context.new_page()
+
+ try:
+ await page.goto(url, wait_until=wait_until, timeout=_TIMEOUT)
+ except Exception:
+ pass
+
+ title = await page.title()
+ final_url = page.url
+
+ # The visible text is both the fallback and the yardstick: a page whose
+ # extractor kept a fraction of it is a page the extractor did not understand.
+ visible = clean_visible_text(await page.evaluate(_VISIBLE_TEXT))
+ extracted = extract_page(await page.content(), final_url, visible_chars=len(visible))
+
+ if extracted is not None:
+ text, links = extracted
+ text = truncate(text, _MAX_TEXT)
+ if links:
+ text = f"{text}\n\nLinks on this page:\n" + "\n".join(links)
+ extractor = "trafilatura"
+ else:
+ text, links, extractor = truncate(visible, _MAX_TEXT), [], "innerText"
+
+ output_parts = []
+ if title:
+ output_parts.append(f"Title: {title}")
+ if final_url != url:
+ output_parts.append(f"Final URL: {final_url}")
+ output_parts.append("")
+ output_parts.append(text)
+ output = "\n".join(output_parts)
+
+ metadata: dict[str, Any] = {"extractor": extractor, "links": len(links)}
+ if screenshot:
+ png = await page.screenshot(full_page=False)
+ metadata |= {
+ "base64": base64.b64encode(png).decode(),
+ "mime": "image/png",
+ "is_image": True,
+ }
+
+ return {
+ "success": True,
+ "output": output,
+ "error": None,
+ "metadata": metadata,
+ }
diff --git a/mcp-servers/navi-web/app/extract.py b/mcp-servers/navi-web/app/extract.py
new file mode 100644
index 0000000..b422c42
--- /dev/null
+++ b/mcp-servers/navi-web/app/extract.py
@@ -0,0 +1,365 @@
+"""Readable text out of a rendered page.
+
+What this replaced: `innerText` of the body after a tag kill-list. That gave a
+wall of navigation crumbs, and on table-heavy pages a single line per thousand
+characters — news.ycombinator.com came back as four lines, none of them a
+sentence. Nothing was marked as a heading, and no link survived, so the model
+could neither see the page's structure nor follow anything on it.
+
+trafilatura picks the main content out by text density instead of by tag name,
+and its ``html`` output keeps the whitespace inside ``
`` that its markdown
+output throws away (``def f():`` and `` return 1`` came back at the same
+indent, i.e. as broken code). So the HTML fragment is rendered here: headings
+get their ``#``, code keeps its fences, and links are lifted into an appendix
+instead of turning every other sentence into ``[label](url)``.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+from urllib.parse import urljoin
+
+import trafilatura
+from lxml import html as lxml_html
+
+# Enough links to follow the page's own argument, not so many that the appendix
+# outweighs the text it belongs to.
+MAX_LINKS = 40
+
+# trafilatura can return a bare menu for a page whose content the browser never
+# rendered; that is not a reason to hand back a heading and nothing else.
+MIN_TEXT = 200
+
+# Links a site offers about itself rather than about its content: vote buttons,
+# "hide", login, edit-history, citation footnotes.
+_SKIP_HREF = ("hide?", "from?site", "user?id", "vote?", "login?", "action=edit",
+ "cite_note")
+
+_SPACES = re.compile(r"[ \t\r\f\v]+")
+_HEADING = re.compile(r"^(#{1,6}) \S")
+_HEADINGS = {f"h{level}": level for level in range(1, 7)}
+
+# Containers with no markdown meaning of their own: their children are blocks.
+_CONTAINERS = {"html", "body", "main", "article", "div", "section", "header",
+ "footer", "figure", "figcaption", "details", "summary", "center",
+ "form", "fieldset"}
+
+_INLINE = {"a", "span", "small", "sub", "sup", "time", "abbr", "del", "ins",
+ "mark", "u", "font", "big", "s", "strike", "cite", "q", "bdi", "bdo",
+ "ruby", "var", "samp", "kbd", "wbr", "nobr"}
+
+# trafilatura's own row dialect, and the html one it falls back to.
+_ROWS = {"row", "tr"}
+
+
+def _tag(el: Any) -> str:
+ tag = getattr(el, "tag", None)
+ return tag.lower() if isinstance(tag, str) else ""
+
+
+def _collect_links(
+ label: str, href: str | None, base: str, links: list[str]
+) -> None:
+ """Keep a link worth following: not an anchor into this page, not site chrome."""
+ label = _SPACES.sub(" ", label).strip()
+ if not label or not href:
+ return
+ target = urljoin(base, href.strip())
+ if not target.startswith(("http://", "https://")):
+ return
+ if target.rstrip("/") == base.rstrip("/"):
+ return
+ # A link whose text is just the URL adds nothing the URL does not already say.
+ if label.lower().startswith(("http://", "https://")):
+ return
+ if any(marker in target for marker in _SKIP_HREF):
+ return
+ if len(label) > 100:
+ label = label[:97].rstrip() + "…"
+ entry = f"- {label} -> {target}"
+ if entry in links or len(links) >= MAX_LINKS:
+ return
+ links.append(entry)
+
+
+def _join(parts: list[str]) -> str:
+ """Glue text pieces back together without gluing their words.
+
+ trafilatura's html output drops the spaces around loose inline elements —
+ a wiki sentence comes back as ``C++andQt`` — so two pieces
+ that both start and end on a letter get a space between them. Punctuation
+ keeps its neighbours: ``(hackerbooks.org)`` stays in parentheses.
+ """
+ out = ""
+ for part in parts:
+ if not part:
+ continue
+ if out and not out[-1].isspace() and not part[0].isspace() \
+ and out[-1].isalnum() and part[0].isalnum():
+ out += " "
+ out += part
+ return _SPACES.sub(" ", out).strip()
+
+
+def _inline(el: Any, base: str, links: list[str]) -> str:
+ """Text of an element that sits inside a paragraph, with links harvested."""
+ parts: list[str] = [el.text or ""]
+ for child in el:
+ tag = _tag(child)
+ if tag == "a":
+ label = _inline(child, base, links)
+ _collect_links(label, child.get("href"), base, links)
+ parts.append(label)
+ elif tag in ("strong", "b"):
+ inner = _inline(child, base, links).strip()
+ parts.append(f"**{inner}**" if inner else "")
+ elif tag == "img":
+ # The image itself is unavailable as text; its alt text usually is not.
+ parts.append((child.get("alt") or "").strip())
+ elif tag == "br":
+ parts.append(" ")
+ elif tag == "pre":
+ parts.append(_code(child))
+ elif tag in _INLINE or tag in ("i", "em", "code", "tt"):
+ parts.append(_inline(child, base, links))
+ else:
+ parts.append(_inline(child, base, links))
+ parts.append(child.tail or "")
+ return _join(parts)
+
+
+def _code(el: Any) -> str:
+ """A
as a fence that keeps its own indentation.
+
+ trafilatura emits its own one-liners as
too — ``
asyncio.run()
``
+ inside a sentence — so a short single line becomes inline code.
+ """
+ code = "".join(el.itertext()).replace(" ", " ").strip("\n")
+ if not code.strip():
+ return ""
+ if "\n" not in code.strip() and len(code.strip()) <= 60:
+ return f"`{code.strip()}`"
+ return f"```\n{code.rstrip()}\n```"
+
+
+def _list(el: Any, base: str, links: list[str]) -> str:
+ ordered = _tag(el) == "ol"
+ items: list[str] = []
+ number = 0
+ for child in el:
+ if _tag(child) != "li":
+ continue
+ number += 1
+ marker = f"{number}. " if ordered else "- "
+ body = _block(child, base, links).strip()
+ if not body:
+ continue
+ lines = body.splitlines()
+ pad = " " * len(marker)
+ items.append(marker + lines[0] + "".join("\n" + pad + ln for ln in lines[1:]))
+ return "\n".join(items)
+
+
+def _row(el: Any, base: str, links: list[str]) -> str:
+ """A table row — HN, wiki infoboxes — as one line of columns.
+
+ trafilatura names its own rows ``/`` and keeps real ones as
+ ``
/
``, with no column model either way and a colspan here or there,
+ so a plain ``|``-joined line keeps the shape without inventing a header.
+ """
+ cells: list[str] = []
+ nested: list[str] = []
+ for child in el:
+ tag = _tag(child)
+ text = _block(child, base, links).replace("\n", " ").strip()
+ if not text:
+ continue
+ (nested if tag in _ROWS else cells).append(text)
+ line = " | ".join(cells)
+ return "\n".join(part for part in [line, *nested] if part)
+
+
+def _block(el: Any, base: str, links: list[str]) -> str:
+ tag = _tag(el)
+ if tag in _HEADINGS:
+ title = _inline(el, base, links)
+ return f"{'#' * _HEADINGS[tag]} {title}" if title else ""
+ if tag == "pre":
+ return _code(el)
+ if tag in ("ul", "ol"):
+ return _list(el, base, links)
+ if tag == "blockquote":
+ inner = _blocks(el, base, links)
+ return "\n".join(f"> {line}".rstrip() if line else ">" for line in inner.splitlines())
+ if tag in _ROWS:
+ return _row(el, base, links)
+ if tag == "hr":
+ return "---"
+ return _inline(el, base, links)
+
+
+def _render_child(el: Any, base: str, links: list[str]) -> str:
+ """A container's child as markdown: blocks that contain blocks, or a leaf."""
+ tag = _tag(el)
+ if tag in _CONTAINERS:
+ return _blocks(el, base, links)
+ if tag in ("table", "tbody", "thead", "tfoot", "dl"):
+ return _join_pieces([
+ ("row" if _tag(grandchild) in _ROWS else _tag(grandchild),
+ _block(grandchild, base, links))
+ for grandchild in el
+ ])
+ return _block(el, base, links)
+
+
+def _join_pieces(pieces: list[tuple[str, str]]) -> str:
+ """Blank line between blocks — except between table rows, which stay adjacent.
+
+ news.ycombinator.com is a table, and a blank line after every row is what
+ turned it into a page-tall wall before.
+ """
+ out = ""
+ previous = ""
+ for kind, text in pieces:
+ if not text.strip():
+ continue
+ if not out:
+ out = text
+ elif kind == previous == "row":
+ out += "\n" + text
+ else:
+ out += "\n\n" + text
+ previous = kind
+ return out
+
+
+def _blocks(el: Any, base: str, links: list[str]) -> str:
+ """Children of a container, one block each."""
+ pieces: list[tuple[str, str]] = []
+ if (el.text or "").strip():
+ pieces.append(("text", _SPACES.sub(" ", el.text or "").strip()))
+ for child in el:
+ tag = _tag(child)
+ pieces.append((tag, _render_child(child, base, links)))
+ if (child.tail or "").strip():
+ pieces.append(("text", _SPACES.sub(" ", child.tail or "").strip()))
+ return _join_pieces(pieces)
+
+
+def _tidy(text: str) -> str:
+ """Collapse the blank runs the renderer leaves behind; never touch a fence."""
+ out: list[str] = []
+ fence = False
+ blank = 0
+ for line in text.splitlines():
+ if line.lstrip().startswith("```"):
+ fence = not fence
+ out.append(line.rstrip())
+ blank = 0
+ continue
+ if fence:
+ out.append(line.rstrip())
+ continue
+ line = line.rstrip()
+ if not line.strip():
+ blank += 1
+ if blank > 1:
+ continue
+ out.append("")
+ continue
+ blank = 0
+ out.append(line.strip())
+ # A pilcrow is how documentation sites mark a heading's own anchor.
+ return "\n".join(out).replace("¶", "").strip()
+
+
+def _extract(html: str, url: str, comments: bool) -> tuple[str, list[str]]:
+ """One trafilatura pass, rendered. ``("", [])`` when the page has no content."""
+ try:
+ fragment = trafilatura.extract(
+ html,
+ url=url,
+ output_format="html",
+ include_links=True,
+ include_tables=True,
+ include_comments=comments,
+ include_formatting=True,
+ )
+ except Exception: # noqa: BLE001 — a broken page must not break the tool
+ return "", []
+ if not fragment:
+ return "", []
+
+ try:
+ tree = lxml_html.fromstring(fragment)
+ except Exception: # noqa: BLE001 — same
+ return "", []
+
+ links: list[str] = []
+ return _tidy(_blocks(tree, url, links)), links
+
+
+def extract_page(
+ html: str, url: str, visible_chars: int = 0
+) -> tuple[str, list[str]] | None:
+ """Main content of a page as markdown, plus the links worth following.
+
+ ``None`` means the extractor found no text worth sending — the caller falls
+ back to reading the page's own visible text rather than returning a menu.
+ """
+ body, links = _extract(html, url, comments=False)
+
+ # A page whose content *is* its comment thread — a forum, a Q&A, a code
+ # review — comes back as little more than a header when comments are off.
+ # Keeping less than a quarter of what the reader can see is the signal that
+ # the extractor threw the point of the page away, so ask again for all of it.
+ if visible_chars and len(body) < visible_chars // 4:
+ wide_body, wide_links = _extract(html, url, comments=True)
+ if len(wide_body) > len(body):
+ body, links = wide_body, wide_links
+
+ if len(body) < MIN_TEXT:
+ return None
+ return body, links
+
+
+def truncate(text: str, limit: int) -> str:
+ """Cut at a paragraph boundary and name the sections that did not fit.
+
+ Cutting mid-sentence left the model with a half-page and no sign that there
+ was more; naming the headings that were dropped tells it whether the rest of
+ the page is worth another request.
+ """
+ if len(text) <= limit:
+ return text
+ cut = text.rfind("\n\n", 0, limit)
+ if cut < limit // 2:
+ cut = limit
+ head, tail = text[:cut].rstrip(), text[cut:]
+ missing = [line.strip() for line in tail.splitlines() if _HEADING.match(line)]
+
+ note = f"[… truncated at {limit} chars]"
+ if missing:
+ shown = ", ".join(missing[:8])
+ more = f" and {len(missing) - 8} more" if len(missing) > 8 else ""
+ note = f"[… truncated at {limit} chars. Sections not shown: {shown}{more}]"
+ if head.count("```") % 2:
+ head += "\n```"
+ return f"{head}\n\n{note}"
+
+
+def clean_visible_text(text: str) -> str:
+ """Fallback path: the page's own visible text, with blank runs collapsed."""
+ lines = [line.rstrip() for line in text.splitlines()]
+ result: list[str] = []
+ blank_run = 0
+ for line in lines:
+ if line == "":
+ blank_run += 1
+ if blank_run <= 2:
+ result.append("")
+ continue
+ blank_run = 0
+ result.append(line)
+ return "\n".join(result).strip()
diff --git a/mcp-servers/navi-web/app/mcp_server.py b/mcp-servers/navi-web/app/mcp_server.py
index 4f910bb..8957f83 100644
--- a/mcp-servers/navi-web/app/mcp_server.py
+++ b/mcp-servers/navi-web/app/mcp_server.py
@@ -4,12 +4,13 @@
import json
import os
+from contextlib import asynccontextmanager
from typing import Annotated, Any
from mcp.server.fastmcp import FastMCP
from pydantic import Field
-from .browse import web_view
+from .browse import close_browser, web_view
from .config import Settings
from .request import http_request
from .search import web_search
@@ -31,7 +32,16 @@
No session_id or filesystem paths are required.
""".strip()
-mcp = FastMCP("navi-web", instructions=INSTRUCTIONS)
+@asynccontextmanager
+async def _lifespan(_server: FastMCP):
+ """Keep one Chromium for the server's lifetime; stop it on shutdown."""
+ try:
+ yield {}
+ finally:
+ await close_browser()
+
+
+mcp = FastMCP("navi-web", instructions=INSTRUCTIONS, lifespan=_lifespan)
def _json(data: Any) -> str:
@@ -44,18 +54,40 @@
@mcp.tool(name="web_search")
async def search_tool(
- query: Annotated[str, Field(description="Search query.")],
+ query: Annotated[
+ str | list[str],
+ Field(
+ description=(
+ "One query, or up to 3 independent queries, which run in parallel."
+ )
+ ),
+ ],
max_results: Annotated[int, Field(description="Number of results (default 5).")] = 5,
) -> str:
- """Search the web (SearXNG primary, DDG fallback, Brave tertiary)."""
+ """Search the web (SearXNG primary, DDG fallback, Brave tertiary).
+
+ Pass several queries in one call when you already know they are independent:
+ they run concurrently, so the call costs the slowest one instead of the sum.
+ """
result = await web_search(_settings(), query, max_results)
+ if not result["success"]:
+ # A failed search is a failed tool call. Returning the JSON with
+ # "success": false kept the journal green — the MCP call itself
+ # succeeded — so a broken search left no trace except the model's own
+ # wording. Raising turns it into ``isError``, which Navi logs.
+ raise RuntimeError(result["output"])
return _json(result)
@mcp.tool(name="web_view")
async def view_tool(
- url: Annotated[str, Field(description="Full URL to open (must start with http:// or https://).")],
- screenshot: Annotated[bool, Field(description="Also capture a screenshot.")] = False,
+ url: Annotated[
+ str | list[str],
+ Field(
+ description="One URL, or up to 3 URLs, which are opened in parallel."
+ ),
+ ],
+ screenshot: Annotated[bool, Field(description="Also capture a screenshot (single URL only).")] = False,
wait_until: Annotated[
str,
Field(
@@ -63,8 +95,22 @@
),
] = "networkidle",
) -> str:
- """Open a URL in a headless browser and return clean readable text."""
+ """Open a URL in a headless browser and return its main content as markdown.
+
+ Navigation, ads and footers are stripped; headings, code blocks and table
+ rows keep their shape. Links from the page follow the text under
+ "Links on this page" with absolute URLs, so you can open one by passing it
+ to this tool.
+
+ Pass several URLs in one call when you already know which pages you need:
+ they open concurrently, so three pages cost about one page's time.
+ """
result = await web_view(url, screenshot=screenshot, wait_until=wait_until)
+ if not result["success"]:
+ # Same rule as the search: no page opened is a failed tool call, not a
+ # JSON with "success": false that leaves the journal green. A page the
+ # browser could not launch at all used to look like an answer.
+ raise RuntimeError(result["output"])
return _json(result)
diff --git a/mcp-servers/navi-web/app/search.py b/mcp-servers/navi-web/app/search.py
index 588a4ab..48faf2a 100644
--- a/mcp-servers/navi-web/app/search.py
+++ b/mcp-servers/navi-web/app/search.py
@@ -10,50 +10,205 @@
from .config import Settings
-_DDG_RETRY_DELAY = 2.0
-_TIMEOUT = 15.0
+# Every wait on this path is bounded, including the chain as a whole. The caller
+# is sitting on this call, and the backends used to add up: 15s SearXNG + two 5s
+# DDG attempts + a 2s pause + 15s Brave — half a minute in which a slow search
+# and a broken one looked the same. The budgets below make the worst case
+# ~12s, long enough for a healthy backend and short enough to still answer.
+_SEARXNG_TIMEOUT = 6.0
+_DDG_TIMEOUT = 4.0
+_BRAVE_TIMEOUT = 6.0
+_DDG_RETRY_DELAY = 1.0
+_TOTAL_TIMEOUT = 12.0
+
+# One call may carry several independent queries: they run concurrently, so the
+# call costs the slowest instead of the sum. The cap is what keeps a batch from
+# turning into a scrape of the search engine — three concurrent queries on one
+# instance still answered in under a second here.
+_MAX_BATCH = 3
+
+
+def _normalize(raw: str | list[str]) -> tuple[list[str], list[str]]:
+ """Turn the argument into the queries to run, plus notes worth telling the model."""
+ queries: list[str] = []
+ for item in [raw] if isinstance(raw, str) else raw:
+ query = str(item).strip()
+ if query and query not in queries:
+ queries.append(query)
+ notes: list[str] = []
+ if len(queries) > _MAX_BATCH:
+ notes.append(
+ f"Only the first {_MAX_BATCH} of {len(queries)} queries ran — "
+ "ask the rest in another call."
+ )
+ queries = queries[:_MAX_BATCH]
+ return queries, notes
async def web_search(
settings: Settings,
- query: str,
+ query: str | list[str],
max_results: int = 5,
) -> dict[str, Any]:
"""Search the web. Returns dict with success, output, error, metadata."""
+ queries, notes = _normalize(query)
+ if not queries:
+ return {
+ "success": False,
+ "output": "No query given.",
+ "error": "empty_query",
+ "metadata": {},
+ }
+
+ # ``return_exceptions`` rather than a try/except: ``_search_one`` already
+ # turns every backend failure into a result, so an exception here is one
+ # query's own, and the batch keeps the rest.
+ outcomes = await asyncio.gather(
+ *(_search_one(settings, q, max_results) for q in queries),
+ return_exceptions=True,
+ )
+
+ if len(queries) == 1 and not notes:
+ outcome = outcomes[0]
+ if isinstance(outcome, BaseException):
+ return {
+ "success": False,
+ "output": f"Search failed: {outcome}",
+ "error": str(outcome),
+ "metadata": {},
+ }
+ return outcome
+
+ return _combine(queries, outcomes, notes)
+
+
+def _combine(
+ queries: list[str],
+ outcomes: list[Any],
+ notes: list[str],
+) -> dict[str, Any]:
+ """One block per query. A query that failed is named, not dropped: the model
+ can retry that one instead of repeating the batch, and a batch that answered
+ something is a success even when the rest did not."""
+ blocks: list[str] = []
+ per_query: list[dict[str, Any]] = []
+ answered = 0
+
+ for query, outcome in zip(queries, outcomes, strict=True):
+ if isinstance(outcome, BaseException):
+ blocks.append(f'### "{query}" -> failed: {outcome}')
+ per_query.append({"query": query, "success": False, "error": str(outcome)})
+ elif outcome["success"]:
+ answered += 1
+ blocks.append(f'### "{query}"\n{outcome["output"]}')
+ per_query.append(
+ {
+ "query": query,
+ "success": True,
+ "source": outcome["metadata"].get("source"),
+ }
+ )
+ else:
+ blocks.append(f'### "{query}" -> failed: {outcome["error"]}')
+ per_query.append(
+ {"query": query, "success": False, "error": outcome["error"]}
+ )
+
+ body = "\n\n".join(notes + blocks)
+ if not answered:
+ reasons = "; ".join(
+ f"{item['query']}: {item['error']}" for item in per_query
+ )
+ return {
+ "success": False,
+ "output": f"No query produced results. {reasons}",
+ "error": "all_queries_failed",
+ "metadata": {"queries": per_query},
+ }
+ return {
+ "success": True,
+ "output": body,
+ "error": None,
+ "metadata": {"queries": per_query},
+ }
+
+
+async def _search_one(
+ settings: Settings,
+ query: str,
+ max_results: int,
+) -> dict[str, Any]:
+ """One query, bounded by the chain's own budget.
+
+ The deadline is per query on purpose: a batch waits for the slowest one, so
+ putting a single ceiling around the whole batch would let one hung backend
+ erase the answers the other queries already had.
+ """
+ try:
+ return await asyncio.wait_for(
+ _search_chain(settings, query, max_results), timeout=_TOTAL_TIMEOUT
+ )
+ except TimeoutError:
+ return {
+ "success": False,
+ "output": (
+ f"Search did not finish within {_TOTAL_TIMEOUT:.0f}s "
+ f"(SearXNG -> DDG -> Brave) for query: {query!r}"
+ ),
+ "error": "timeout",
+ "metadata": {},
+ }
+
+
+async def _search_chain(
+ settings: Settings,
+ query: str,
+ max_results: int,
+) -> dict[str, Any]:
last_error = ""
- # 1. SearXNG (primary)
+ # 1. SearXNG (primary). Its engine list is pinned on purpose: the instance's
+ # own defaults are mostly suspended, and without ``engines`` it returns
+ # nothing at all.
if settings.searxng_url:
try:
- results = await _searxng(settings, query, max_results)
+ results = await asyncio.wait_for(
+ _searxng(settings, query, max_results), timeout=_SEARXNG_TIMEOUT
+ )
if results:
return _format(results, source="SearXNG")
last_error = "SearXNG: empty response"
+ except TimeoutError:
+ last_error = f"SearXNG: no answer in {_SEARXNG_TIMEOUT:.0f}s"
except Exception as e:
last_error = f"SearXNG: {e}"
- # 2. DuckDuckGo html fallback, retry once
+ # 2. DuckDuckGo html fallback. One attempt is cheap; a second is worth it
+ # only when the first *failed* (a transient block or reset). An empty result
+ # is an answer — retrying it only buys another timeout.
for attempt in range(2):
try:
- results = await asyncio.to_thread(
- lambda: list(
- DDGS().text(query, backend="html", max_results=max_results, region=settings.ddg_region)
- )
- )
+ results = await asyncio.to_thread(_ddg, settings, query, max_results)
if results:
return _format(results, source="DDG")
last_error = "DDG: empty response"
+ break
except Exception as e:
last_error = f"DDG: {e}"
- if attempt == 0:
- await asyncio.sleep(_DDG_RETRY_DELAY)
+ if attempt == 0:
+ await asyncio.sleep(_DDG_RETRY_DELAY)
- # 3. Brave Search fallback
+ # 3. Brave Search fallback, only when a key is configured.
if settings.brave_api_key:
try:
- results = await _brave(settings, query, max_results)
+ results = await asyncio.wait_for(
+ _brave(settings, query, max_results), timeout=_BRAVE_TIMEOUT
+ )
if results:
return _format(results, source="Brave")
+ last_error = "Brave: empty response"
+ except TimeoutError:
+ last_error = f"Brave: no answer in {_BRAVE_TIMEOUT:.0f}s"
except Exception as e:
last_error = f"Brave: {e}"
@@ -65,8 +220,25 @@
}
+def _ddg(settings: Settings, query: str, max_results: int) -> list[dict[str, Any]]:
+ """One DuckDuckGo attempt, off the event loop.
+
+ ``timeout`` is a ``DDGS`` client argument, not a ``text()`` one — passing it
+ to ``text()`` would land in ``**kwargs`` and do nothing.
+ """
+ ddgs = DDGS(timeout=int(_DDG_TIMEOUT))
+ return list(
+ ddgs.text(
+ query,
+ backend="html",
+ max_results=max_results,
+ region=settings.ddg_region,
+ )
+ )
+
+
async def _searxng(settings: Settings, query: str, max_results: int) -> list[dict]:
- async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
+ async with httpx.AsyncClient(timeout=_SEARXNG_TIMEOUT) as client:
resp = await client.get(
f"{settings.searxng_url}/search",
params={
@@ -85,7 +257,7 @@
async def _brave(settings: Settings, query: str, max_results: int) -> list[dict]:
- async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
+ async with httpx.AsyncClient(timeout=_BRAVE_TIMEOUT) as client:
resp = await client.get(
"https://api.search.brave.com/res/v1/web/search",
params={"q": query, "count": min(max_results, 20), "country": "ALL"},
diff --git a/mcp-servers/navi-web/pyproject.toml b/mcp-servers/navi-web/pyproject.toml
index 4beab11..08f2b6a 100644
--- a/mcp-servers/navi-web/pyproject.toml
+++ b/mcp-servers/navi-web/pyproject.toml
@@ -13,6 +13,7 @@
"ddgs>=1.0",
"playwright>=1.40",
"pydantic>=2.0",
+ "trafilatura>=2.0",
]
[project.scripts]
diff --git a/mcp_servers.d/navi-web.json b/mcp_servers.d/navi-web.json
index 1d166ca..1274aaf 100644
--- a/mcp_servers.d/navi-web.json
+++ b/mcp_servers.d/navi-web.json
@@ -7,7 +7,8 @@
],
"cwd": "./mcp-servers/navi-web",
"env": {
- "NAVI_WEB_MCP_TRANSPORT": "stdio"
+ "NAVI_WEB_MCP_TRANSPORT": "stdio",
+ "NAVI_WEB_SEARXNG_URL": "${SEARXNG_URL:-}"
},
"groups": {
"search": [
@@ -20,6 +21,6 @@
"http_request"
]
},
- "instructions": "Navi Web MCP server provides web search, browsing, and raw HTTP tools.\n\nUse it when the task involves:\n- searching the web for current info, docs, or real-time data;\n- opening a URL in a browser to read human-readable content;\n- making REST API calls, webhooks, or raw HTTP requests.\n\nWorkflow:\n1. search — find relevant pages or facts.\n2. view — open promising URLs to read full content.\n3. request — call APIs or services requiring headers/auth.\n\nAll three tools are stateless and work with public URLs.\nNo session_id or filesystem paths are required.\n\nABSOLUTE RULE — NEVER bypass MCP tools:\nYou MUST NOT use filesystem, terminal, code_exec, or any direct file access to read or write web content. Use only the MCP tools listed above.",
- "summary": "Web search, page browsing and raw HTTP. Never fetch web content with filesystem, terminal or code_exec — these are the only valid interface. Details: tool_manual(\"navi-web\")."
+ "instructions": "Navi Web MCP server provides web search, browsing, and raw HTTP tools.\n\nUse it when the task involves:\n- searching the web for current info, docs, or real-time data;\n- opening a URL in a browser to read human-readable content;\n- making REST API calls, webhooks, or raw HTTP requests.\n\nWorkflow:\n1. search — find relevant pages or facts.\n2. view — open promising URLs to read full content.\n3. request — call APIs or services requiring headers/auth.\n\nBatching: web_search takes one query or up to 3, web_view one URL or up to 3 — several run in parallel, so the call costs the slowest instead of the sum. Use it when you already know the lookups are independent: after a search, open the 2-3 pages you need in one web_view call; when a request needs several unrelated facts, ask the searches together. A failed item in a batch is named in the output rather than dropping the whole call. Do not batch a follow-up that depends on what the previous answer said — that is a sequence, not a batch.\n\nAll three tools are stateless and work with public URLs.\nNo session_id or filesystem paths are required.\n\nABSOLUTE RULE — NEVER bypass MCP tools:\nYou MUST NOT use filesystem, terminal, code_exec, or any direct file access to read or write web content. Use only the MCP tools listed above.",
+ "summary": "Web search, page browsing and raw HTTP; up to 3 queries or URLs in one call run in parallel, so batch independent lookups. Never fetch web content with filesystem, terminal or code_exec — these are the only valid interface. Details: tool_manual(\"navi-web\")."
}
diff --git a/pyproject.toml b/pyproject.toml
index 289d04a..ee7fe64 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -83,6 +83,9 @@
"pytest-asyncio>=0.23",
"ruff>=0.4",
"mypy>=1.10",
+ # The navi-web MCP server renders pages through trafilatura; its tests import
+ # that server by path, so the package has to exist in this venv too.
+ "trafilatura>=2.0",
]
[tool.hatch.metadata]
diff --git a/tests/unit/mcp/conftest.py b/tests/unit/mcp/conftest.py
new file mode 100644
index 0000000..2ebb42c
--- /dev/null
+++ b/tests/unit/mcp/conftest.py
@@ -0,0 +1,77 @@
+"""Load `mcp-servers//app` for tests.
+
+The MCP servers are separate processes with their own venv, not packages of
+this project, and `pyproject.toml` collects `tests/` only — so a test reaches
+them by path. A relative import inside them (`from .config import Settings`)
+needs a real package entry in `sys.modules` with a search path, which is what
+``_load`` builds.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+import pytest
+
+_SERVERS = Path(__file__).resolve().parents[3] / "mcp-servers"
+
+
+def _load(server: str, module: str | None = None) -> ModuleType:
+ app_dir = _SERVERS / server / "app"
+ package = "mcp_" + server.replace("-", "_") + "_app"
+
+ if package not in sys.modules:
+ spec = importlib.util.spec_from_file_location(
+ package, app_dir / "__init__.py", submodule_search_locations=[str(app_dir)]
+ )
+ assert spec and spec.loader
+ package_module = importlib.util.module_from_spec(spec)
+ sys.modules[package] = package_module
+ spec.loader.exec_module(package_module)
+
+ if module is None:
+ return sys.modules[package]
+
+ name = f"{package}.{module}"
+ if name not in sys.modules:
+ spec = importlib.util.spec_from_file_location(name, app_dir / f"{module}.py")
+ assert spec and spec.loader
+ module_object = importlib.util.module_from_spec(spec)
+ sys.modules[name] = module_object
+ spec.loader.exec_module(module_object)
+ return sys.modules[name]
+
+
+@pytest.fixture(scope="session")
+def navi_web_search():
+ return _load("navi-web", "search")
+
+
+@pytest.fixture(scope="session")
+def navi_web_browse():
+ return _load("navi-web", "browse")
+
+
+@pytest.fixture(scope="session")
+def navi_web_extract():
+ """Page-to-markdown rendering — the part that needs no browser."""
+ return _load("navi-web", "extract")
+
+
+@pytest.fixture(scope="session")
+def navi_web_server():
+ """The FastMCP tool layer — where ``isError`` is decided."""
+ return _load("navi-web", "mcp_server")
+
+
+@pytest.fixture
+def navi_web_settings():
+ """Settings with every optional backend off, so each test enables its own."""
+ settings = _load("navi-web", "config").Settings()
+ settings.searxng_url = ""
+ settings.brave_api_key = ""
+ settings.ddg_region = "ru-ru"
+ return settings
diff --git a/tests/unit/mcp/test_navi_web_extract.py b/tests/unit/mcp/test_navi_web_extract.py
new file mode 100644
index 0000000..59b6fb0
--- /dev/null
+++ b/tests/unit/mcp/test_navi_web_extract.py
@@ -0,0 +1,251 @@
+"""web_view must send the page, not the page's furniture.
+
+The old path was `innerText` of the body after a tag kill-list: navigation
+crumbs survived, tables collapsed into one line per thousand characters
+(news.ycombinator.com came back as four lines), headings were unmarked and
+every link was gone. These tests pin the replacement: main content only,
+structure kept, links lifted into an appendix.
+"""
+
+PAGE = """Guide
+
+
+
Guide
+
First paragraph of the guide: real prose that the reader came here for, long
+enough to be kept by the extractor without any doubt at all about it.
+
def f():
+ if True:
+ return 1
+
+
Second section
+
Second paragraph, also prose, also worth keeping: it explains the second
+section in a sentence or two, which is what the reader wanted.
Third paragraph is here only so the page clears the extractor's minimum
+length, and it says nothing except that it exists.
+
+
+"""
+
+
+class TestExtraction:
+ def test_the_content_survives_and_the_chrome_does_not(self, navi_web_extract):
+ body, _ = navi_web_extract.extract_page(PAGE, "https://example.com/a/b")
+
+ assert "First paragraph of the guide" in body
+ assert "Second section" in body
+ assert "About us" not in body
+ assert "Boilerplate" not in body
+
+ def test_headings_and_code_keep_their_shape(self, navi_web_extract):
+ """The fence matters: markdown output from trafilatura flattens
+ `` return 1`` to column 0, which is not Python any more."""
+ body, _ = navi_web_extract.extract_page(PAGE, "https://example.com/a/b")
+
+ assert "# Guide" in body
+ assert "## Second section" in body
+ assert "```\ndef f():\n if True:\n return 1\n```" in body
+
+ def test_links_move_to_an_appendix_with_absolute_urls(self, navi_web_extract):
+ body, links = navi_web_extract.extract_page(PAGE, "https://example.com/a/b")
+
+ assert "Read the docs" in body
+ assert "https://example.com/docs/page" not in body
+ assert links == [
+ "- Read the docs -> https://example.com/docs/page",
+ "- another page -> https://other.example/x",
+ ]
+
+ def test_a_page_without_text_is_not_a_page(self, navi_web_extract):
+ html = ""
+
+ assert navi_web_extract.extract_page(html, "https://example.com/") is None
+
+ def test_a_thread_is_read_with_its_comments(self, navi_web_extract, monkeypatch):
+ """Forum pages are their comments: a header-only extract is a miss, and
+ the signal is how much of the visible page the extractor threw away."""
+ calls: list[bool] = []
+
+ def fake_extract(html, url, comments):
+ calls.append(comments)
+ if comments:
+ return "x" * 5_000, ["- reply -> https://forum.example/1"]
+ return "x" * 100, []
+
+ monkeypatch.setattr(navi_web_extract, "_extract", fake_extract)
+
+ body, links = navi_web_extract.extract_page(
+ "", "https://forum.example/1", visible_chars=100_000
+ )
+
+ assert calls == [False, True]
+ assert len(body) == 5_000
+ assert links == ["- reply -> https://forum.example/1"]
+
+ def test_a_page_that_was_understood_is_not_read_twice(
+ self, navi_web_extract, monkeypatch
+ ):
+ calls: list[bool] = []
+
+ def fake_extract(html, url, comments):
+ calls.append(comments)
+ return "x" * 1_000, []
+
+ monkeypatch.setattr(navi_web_extract, "_extract", fake_extract)
+
+ navi_web_extract.extract_page(
+ "", "https://blog.example/1", visible_chars=2_000
+ )
+
+ assert calls == [False]
+
+
+class TestLinkAppendix:
+ def test_noise_links_are_dropped(self, navi_web_extract):
+ page = """
Prose that is long enough to be kept, and it goes on for a while: it says
+a few things about the subject at hand, which is what the reader came for, and
+it keeps saying them until the extractor is satisfied.
+"""
+
+ _, links = navi_web_extract.extract_page(page, "https://example.com/post")
+
+ assert links == ["- The real target -> https://example.com/real"]
+
+ def test_the_appendix_is_capped(self, navi_web_extract):
+ anchors = "".join(f'page {i} ' for i in range(80))
+ page = f"""
Prose that is long enough to be kept, and it goes on for a while: it says
+a few things about the subject at hand, which is what the reader came for, and
+it keeps saying them until the extractor is satisfied, one page at a time.
Prose that is long enough to be kept, and it goes on for a while: it says
+a few things about the subject at hand, which is what the reader came for, and
+it keeps saying them until the extractor is satisfied.