diff --git a/deploy/UPDATE.md b/deploy/UPDATE.md index 88bde41..2fc488b 100644 --- a/deploy/UPDATE.md +++ b/deploy/UPDATE.md @@ -77,6 +77,8 @@ git fetch origin git pull --ff-only # on deploy: git reset --hard origin/deploy bash deploy/install.sh # ONLY if pyproject.toml changed since the deployed commit +# ...and it does NOT touch mcp-servers/*/.venv: an MCP server whose own +# pyproject.toml changed needs its install too (docs/mcp.md#the-servers-own-venv) sudo systemctl restart navi systemctl status navi --no-pager journalctl -u navi -n 50 --no-pager diff --git a/docs/mcp.md b/docs/mcp.md index e0d1806..31e9889 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -61,6 +61,11 @@ unauthenticated request as an anonymous user instead of returning 401, so silently dropping would fail open. `McpManager.load_all` isolates a failing server, so this shows up as one broken entry in `/mcp/status`. +- **The explicit escape hatch is `${NAME:-fallback}`** (fallback may be empty). + It is for a *setting* that tunes a server — a SearXNG URL — where being unset + costs one backend instead of every tool the server provides. Writing the + fallback down is the point: it is a decision in the config, not an accident of + a variable nobody set. A credential never takes this form. - **Resolution happens at transport open** (`navi/mcp/client.py`), next to the project-relative path resolution. This is what keeps every stored and serialised config placeholder-only: `save_mcp_servers` — reached by @@ -70,6 +75,92 @@ A user key (`user_key`) is injected first and overwrites the header, so a per-user credential simply replaces the placeholder — no substitution happens. +### `navi-web` settings + +The search/browse server has no user key, but it still has runtime settings of its +own, and they reach it through the same placeholder mechanism. A stdio server does +**not** inherit the Navi process's environment: the SDK passes a safe allowlist +(`HOME`, `PATH`, `USER`, …) plus the config's `env` block, so a variable that is +only in `.env` is invisible to it unless the config maps it. + +| Config `env` key | `.env` variable | Meaning | +|---|---|---| +| `NAVI_WEB_MCP_TRANSPORT` | — | `stdio`, declared inline | +| `NAVI_WEB_SEARXNG_URL` | `SEARXNG_URL` (via `${SEARXNG_URL:-}`) | Self-hosted SearXNG base URL — the primary backend | +| `NAVI_WEB_BRAVE_API_KEY` | `BRAVE_SEARCH_API_KEY` | Optional third backend. Add the placeholder only once the key exists: an unset placeholder without a fallback fails the connection (see above) | +| `NAVI_WEB_DDG_REGION` | — | DuckDuckGo region, default `ua-uk` | + +Without `NAVI_WEB_SEARXNG_URL` the server starts straight from DuckDuckGo HTML +scraping — roughly an order of magnitude slower, and the first backend to be +rate-limited. The `${SEARXNG_URL:-}` fallback is what keeps a host with no +SearXNG working: an unset value costs the backend, not the server's three tools. +`navi.config.Settings` has no search fields: `SEARXNG_URL` is read here, not by +Navi itself. + +#### Batching is the latency lever + +`web_search` takes one query or up to three, `web_view` one URL or up to three; +several run concurrently, so the call costs the slowest item instead of the sum +— three pages measured 17.3 s one by one and 1.9 s together, three searches +3.5 s and 0.6 s. The caps live in the server: three items, deduplicated, extras +dropped with a note in the output. + +The model is pointed at the feature twice — by the tool descriptions, which +travel in every request, and by `summary` in `mcp_servers.d/navi-web.json`, +which is the line that reaches the system prompt (`instructions` carries the +long form for `tool_manual`). The wording is a trigger, not a speed plea: batch +when the lookups are already known to be independent, never a follow-up whose +query depends on the previous answer. + +A per-item deadline is deliberate. One hung backend or page is named in the +output rather than erasing what the others already returned, and the tool +raises — `isError` in the journal — only when nothing at all came back. + +This is not Navi's `parallel_tool_calls`: that flag runs a whole batch of +independent tool calls concurrently for *every* tool of a profile, `terminal` +included. The server-side fan-out is scoped to these two tools, and its caps and +budgets are enforced where the work happens. + +#### What `web_view` hands back + +The page is rendered in Chromium, then `trafilatura` picks the main content out +by text density and the server renders that fragment as markdown: `#` headings, +fenced code with its own indentation, table rows joined by `|`. This replaced +`innerText` after a tag kill-list, which could not tell navigation from prose — +462 of the asyncio page's 1018 lines were shorter than 40 characters — collapsed +tables into one line per thousand characters (news.ycombinator.com came back as +four lines), marked no headings and dropped every link. + +Links become a `Links on this page` appendix rather than `[label](url)` inline: +at most 40, absolute, deduplicated, with self-anchors and `hide`/`vote`/`login` +chrome filtered out. Inline links cost 3.5k characters on the docs page alone, +and `trafilatura` drops the `#` from any heading that carries an anchor when they +are on. Text past 20k characters is cut at a paragraph boundary, and the note +names the sections that did not fit. + +A page whose content *is* its thread — a forum, a Q&A, a code review — comes +back as a bare header when comments are excluded, so when the extractor keeps +less than a quarter of the page's visible text it runs again with comments. If +it still finds nothing worth sending, the tool falls back to the visible-text +read; `metadata.extractor` says which path answered (`trafilatura` or +`innerText`), and `metadata.links` counts the appendix. + +#### The server's own venv + +`navi-web` runs from `mcp-servers/navi-web/.venv`, which is separate from Navi's: +`./mcp-servers/navi-web/.venv/bin/python -m app.mcp_server` with its working +directory there. `deploy/install.sh` rebuilds only Navi's venv, so a dependency +added to `mcp-servers/navi-web/pyproject.toml` — `trafilatura` today — needs its +own install on the host: + +```bash +VIRTUAL_ENV=mcp-servers/navi-web/.venv uv pip install -e mcp-servers/navi-web +sudo systemctl restart navi +``` + +Without it the module does not import and the server fails to start, which +`/mcp/status` shows as a disconnected server rather than as degraded output. + ## Runtime - `McpManager` (`navi/mcp/manager.py`) holds one `McpClient` per server, diff --git a/docs/tools.md b/docs/tools.md index 3500948..54d9556 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -14,8 +14,8 @@ | Server | Tool name | Description | |---|---|---| -| `navi-web` | `mcp__navi-web__web_search` | Web search (SearXNG primary, DDG fallback, Brave tertiary) | -| `navi-web` | `mcp__navi-web__web_view` | Open a URL in a headless browser and return clean readable text | +| `navi-web` | `mcp__navi-web__web_search` | Web search (SearXNG primary, DDG fallback, Brave tertiary). One query or up to 3, run in parallel | +| `navi-web` | `mcp__navi-web__web_view` | Open a URL in a headless browser and return the main content as markdown plus a links appendix. One URL or up to 3, opened in parallel under a shared 40k-char text budget | | `navi-web` | `mcp__navi-web__http_request` | Raw HTTP request (GET/POST/PUT/PATCH/DELETE) | | `navi-3d` | `mcp__navi-3d__compile_scad` | Compile an OpenSCAD script into a binary STL file | | `navi-3d` | `mcp__navi-3d__lint_scad` | Lightweight OpenSCAD source linting before STL compilation | @@ -31,6 +31,9 @@ MCP tools survive `reload_tools` because they are registered as external tools in `ToolRegistry`. MCP configs, transports and per-user credentials (BYOK) are covered in [`mcp.md`](mcp.md). +`navi-web` takes its search backends from `SEARXNG_URL` in `.env` (mapped into the +server's environment by `mcp_servers.d/navi-web.json`); without it the chain starts at +DuckDuckGo scraping — see [`mcp.md`](mcp.md#navi-web-settings). | Tool | Name | Description | |---|---|---| diff --git a/mcp-servers/navi-web/app/browse.py b/mcp-servers/navi-web/app/browse.py index 9d37833..190c40a 100644 --- a/mcp-servers/navi-web/app/browse.py +++ b/mcp-servers/navi-web/app/browse.py @@ -2,40 +2,238 @@ from __future__ import annotations +import asyncio import base64 -import re +from contextlib import suppress from typing import Any from playwright.async_api import async_playwright -_TIMEOUT = 30_000 +from .extract import clean_visible_text, extract_page, truncate + +_TIMEOUT = 20_000 _MAX_TEXT = 20_000 +# A call may open several pages at once: pages take 1-8s each, so a batch costs +# the slowest instead of the sum (three URLs here: 17.3s one by one, 1.9s +# together). The shared text budget is the counterweight — three pages at the +# single-page cap would be 60k chars of context, so a batch stops at 40k and +# says where it stopped. +_MAX_BATCH = 3 +_BATCH_TEXT = 40_000 +_BATCH_DEADLINE = 30.0 -def _clean(text: str) -> str: - lines = [line.rstrip() for line in text.splitlines()] - result: list[str] = [] - blank_run = 0 - for line in lines: - if line == "": - blank_run += 1 - if blank_run <= 2: - result.append("") - else: - blank_run = 0 - result.append(line) - return "\n".join(result).strip() +# One Chromium for the life of the server. Launching a browser per call cost ~5s +# even for ``example.com``; this process is long-lived, so the browser can be +# too. Each call still gets a fresh context, which is what isolates cookies, +# storage and the page from its neighbours. +_playwright = None +_browser = None +_browser_lock = asyncio.Lock() + +# Contexts are cheap but not free; the cap keeps a burst of concurrent calls +# (a batch, or two tool calls Navi ran in parallel) from launching a crowd. +_view_slots = asyncio.Semaphore(_MAX_BATCH) + + +def _normalize(raw: str | list[str]) -> tuple[list[str], list[str]]: + """Turn the argument into the URLs to open, plus notes worth telling the model.""" + urls: list[str] = [] + for item in [raw] if isinstance(raw, str) else raw: + url = str(item).strip() + if url and url not in urls: + urls.append(url) + notes: list[str] = [] + if len(urls) > _MAX_BATCH: + notes.append( + f"Only the first {_MAX_BATCH} of {len(urls)} URLs were opened — " + "ask the rest in another call." + ) + urls = urls[:_MAX_BATCH] + return urls, notes + + + +async def _get_browser(): + """Return the shared browser, launching (or relaunching) it if needed.""" + global _playwright, _browser + async with _browser_lock: + if _browser is not None and _browser.is_connected(): + return _browser + # A browser that crashed leaves a Playwright driver behind; stop it + # before starting another, or the driver processes accumulate. + if _playwright is not None: + with suppress(Exception): + await _playwright.stop() + _playwright = None + _browser = None + _playwright = await async_playwright().start() + _browser = await _playwright.chromium.launch(headless=True) + return _browser + + +async def close_browser() -> None: + """Stop Chromium and its Playwright driver. Called from the server lifespan.""" + global _playwright, _browser + async with _browser_lock: + if _browser is not None: + with suppress(Exception): + await _browser.close() + _browser = None + if _playwright is not None: + with suppress(Exception): + await _playwright.stop() + _playwright = None + + +# The page's own visible text, without the chrome that never had a reader. +_VISIBLE_TEXT = """() => { + const kill = ['script','style','noscript','iframe', + 'nav','header','footer','aside', + '[role="navigation"]','[role="banner"]', + '[role="contentinfo"]']; + const clone = document.body.cloneNode(true); + kill.forEach(sel => { + clone.querySelectorAll(sel).forEach(el => el.remove()); + }); + return clone.innerText || clone.textContent || ''; +}""" async def web_view( + url: str | list[str], + screenshot: bool = False, + wait_until: str = "networkidle", +) -> dict[str, Any]: + """Open a URL — or up to three — in a headless browser and return readable text. + + Returns dict with success, output, error, metadata. + """ + urls, notes = _normalize(url) + if not urls: + return { + "success": False, + "output": "No URL given.", + "error": "empty_url", + "metadata": {}, + } + if screenshot and len(urls) > 1: + return { + "success": False, + "output": ( + "screenshot works on one URL at a time — ask for a single URL " + "with screenshot, or several without." + ), + "error": "screenshot_batch", + "metadata": {}, + } + + if len(urls) == 1 and not notes: + return await _view_bounded(urls[0], screenshot=screenshot, wait_until=wait_until) + + outcomes = await asyncio.gather( + *(_view_bounded(u, wait_until=wait_until) for u in urls), + return_exceptions=True, + ) + return _combine(urls, outcomes, notes) + + +async def _view_bounded( url: str, screenshot: bool = False, wait_until: str = "networkidle", ) -> dict[str, Any]: - """Open a URL in a headless browser and return readable text. + """One page, bounded on its own. - Returns dict with success, output, error, metadata. + The deadline is per URL: a batch waits for its slowest page, and a single + ceiling around the whole batch would let one hung page erase the others + that already loaded. """ + try: + return await asyncio.wait_for( + _view_one(url, screenshot=screenshot, wait_until=wait_until), + timeout=_BATCH_DEADLINE, + ) + except TimeoutError: + return { + "success": False, + "output": f"No answer within {_BATCH_DEADLINE:.0f}s: {url}", + "error": "timeout", + "metadata": {}, + } + + +def _combine( + urls: list[str], + outcomes: list[Any], + notes: list[str], +) -> dict[str, Any]: + """One block per URL, sharing one text budget. + + A URL that failed is named rather than dropped — the model can retry that + one page instead of repeating the batch — and a batch that opened something + is a success even when the rest did not. + """ + blocks: list[str] = [] + per_url: list[dict[str, Any]] = [] + budget = _BATCH_TEXT + opened = 0 + + for url, outcome in zip(urls, outcomes, strict=True): + if isinstance(outcome, BaseException): + blocks.append(f"### {url} -> failed: {outcome}") + per_url.append({"url": url, "success": False, "error": str(outcome)}) + continue + if not outcome["success"]: + blocks.append(f"### {url} -> failed: {outcome['output']}") + per_url.append({"url": url, "success": False, "error": outcome["error"]}) + continue + + opened += 1 + block = f"### {url}\n{outcome['output']}" + if budget <= 0: + block = ( + f"### {url}\n[… not shown: the batch's " + f"{_BATCH_TEXT // 1000}k-char text budget is used up]" + ) + per_url.append({"url": url, "success": True, "truncated": True}) + blocks.append(block) + continue + if len(block) > budget: + block = ( + block[:budget] + + f"\n\n[… truncated: the batch's {_BATCH_TEXT // 1000}k-char " + "text budget is used up]" + ) + per_url.append({"url": url, "success": True, "truncated": True}) + budget = 0 + else: + per_url.append({"url": url, "success": True}) + budget -= len(block) + blocks.append(block) + + body = "\n\n".join(notes + blocks) + if not opened: + return { + "success": False, + "output": body or "No URL could be opened.", + "error": "all_urls_failed", + "metadata": {"urls": per_url}, + } + return { + "success": True, + "output": body, + "error": None, + "metadata": {"urls": per_url}, + } + + +async def _view_one( + url: str, + screenshot: bool = False, + wait_until: str = "networkidle", +) -> dict[str, Any]: + """Open one page in its own context. Returns dict with success, output, error, metadata.""" if not url.startswith(("http://", "https://")): return { "success": False, @@ -45,8 +243,8 @@ } try: - async with async_playwright() as pw: - browser = await pw.chromium.launch(headless=True) + async with _view_slots: + browser = await _get_browser() context = await browser.new_context( viewport={"width": 1280, "height": 800}, user_agent=( @@ -54,59 +252,11 @@ "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ), ) - page = await context.new_page() - try: - await page.goto(url, wait_until=wait_until, timeout=_TIMEOUT) - except Exception: - pass - - title = await page.title() - final_url = page.url - - text = await page.evaluate("""() => { - const kill = ['script','style','noscript','iframe', - 'nav','header','footer','aside', - '[role="navigation"]','[role="banner"]', - '[role="contentinfo"]']; - const clone = document.body.cloneNode(true); - kill.forEach(sel => { - clone.querySelectorAll(sel).forEach(el => el.remove()); - }); - return clone.innerText || clone.textContent || ''; - }""") - - text = _clean(text) - if len(text) > _MAX_TEXT: - text = text[:_MAX_TEXT] + f"\n\n[… truncated at {_MAX_TEXT} chars]" - - output_parts = [] - if title: - output_parts.append(f"Title: {title}") - if final_url != url: - output_parts.append(f"Final URL: {final_url}") - output_parts.append("") - output_parts.append(text) - output = "\n".join(output_parts) - - metadata: dict[str, Any] = {} - if screenshot: - png = await page.screenshot(full_page=False) - metadata = { - "base64": base64.b64encode(png).decode(), - "mime": "image/png", - "is_image": True, - } - - await context.close() - await browser.close() - - return { - "success": True, - "output": output, - "error": None, - "metadata": metadata or {}, - } + return await _read_page(context, url, screenshot, wait_until) + finally: + with suppress(Exception): + await context.close() except Exception as e: return { @@ -115,3 +265,59 @@ "error": str(e), "metadata": {}, } + + +async def _read_page( + context: Any, + url: str, + screenshot: bool, + wait_until: str, +) -> dict[str, Any]: + page = await context.new_page() + + try: + await page.goto(url, wait_until=wait_until, timeout=_TIMEOUT) + except Exception: + pass + + title = await page.title() + final_url = page.url + + # The visible text is both the fallback and the yardstick: a page whose + # extractor kept a fraction of it is a page the extractor did not understand. + visible = clean_visible_text(await page.evaluate(_VISIBLE_TEXT)) + extracted = extract_page(await page.content(), final_url, visible_chars=len(visible)) + + if extracted is not None: + text, links = extracted + text = truncate(text, _MAX_TEXT) + if links: + text = f"{text}\n\nLinks on this page:\n" + "\n".join(links) + extractor = "trafilatura" + else: + text, links, extractor = truncate(visible, _MAX_TEXT), [], "innerText" + + output_parts = [] + if title: + output_parts.append(f"Title: {title}") + if final_url != url: + output_parts.append(f"Final URL: {final_url}") + output_parts.append("") + output_parts.append(text) + output = "\n".join(output_parts) + + metadata: dict[str, Any] = {"extractor": extractor, "links": len(links)} + if screenshot: + png = await page.screenshot(full_page=False) + metadata |= { + "base64": base64.b64encode(png).decode(), + "mime": "image/png", + "is_image": True, + } + + return { + "success": True, + "output": output, + "error": None, + "metadata": metadata, + } diff --git a/mcp-servers/navi-web/app/extract.py b/mcp-servers/navi-web/app/extract.py new file mode 100644 index 0000000..b422c42 --- /dev/null +++ b/mcp-servers/navi-web/app/extract.py @@ -0,0 +1,365 @@ +"""Readable text out of a rendered page. + +What this replaced: `innerText` of the body after a tag kill-list. That gave a +wall of navigation crumbs, and on table-heavy pages a single line per thousand +characters — news.ycombinator.com came back as four lines, none of them a +sentence. Nothing was marked as a heading, and no link survived, so the model +could neither see the page's structure nor follow anything on it. + +trafilatura picks the main content out by text density instead of by tag name, +and its ``html`` output keeps the whitespace inside ``
`` that its markdown
+output throws away (``def f():`` and ``    return 1`` came back at the same
+indent, i.e. as broken code). So the HTML fragment is rendered here: headings
+get their ``#``, code keeps its fences, and links are lifted into an appendix
+instead of turning every other sentence into ``[label](url)``.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any
+from urllib.parse import urljoin
+
+import trafilatura
+from lxml import html as lxml_html
+
+# Enough links to follow the page's own argument, not so many that the appendix
+# outweighs the text it belongs to.
+MAX_LINKS = 40
+
+# trafilatura can return a bare menu for a page whose content the browser never
+# rendered; that is not a reason to hand back a heading and nothing else.
+MIN_TEXT = 200
+
+# Links a site offers about itself rather than about its content: vote buttons,
+# "hide", login, edit-history, citation footnotes.
+_SKIP_HREF = ("hide?", "from?site", "user?id", "vote?", "login?", "action=edit",
+              "cite_note")
+
+_SPACES = re.compile(r"[ \t\r\f\v]+")
+_HEADING = re.compile(r"^(#{1,6}) \S")
+_HEADINGS = {f"h{level}": level for level in range(1, 7)}
+
+# Containers with no markdown meaning of their own: their children are blocks.
+_CONTAINERS = {"html", "body", "main", "article", "div", "section", "header",
+               "footer", "figure", "figcaption", "details", "summary", "center",
+               "form", "fieldset"}
+
+_INLINE = {"a", "span", "small", "sub", "sup", "time", "abbr", "del", "ins",
+           "mark", "u", "font", "big", "s", "strike", "cite", "q", "bdi", "bdo",
+           "ruby", "var", "samp", "kbd", "wbr", "nobr"}
+
+# trafilatura's own row dialect, and the html one it falls back to.
+_ROWS = {"row", "tr"}
+
+
+def _tag(el: Any) -> str:
+    tag = getattr(el, "tag", None)
+    return tag.lower() if isinstance(tag, str) else ""
+
+
+def _collect_links(
+    label: str, href: str | None, base: str, links: list[str]
+) -> None:
+    """Keep a link worth following: not an anchor into this page, not site chrome."""
+    label = _SPACES.sub(" ", label).strip()
+    if not label or not href:
+        return
+    target = urljoin(base, href.strip())
+    if not target.startswith(("http://", "https://")):
+        return
+    if target.rstrip("/") == base.rstrip("/"):
+        return
+    # A link whose text is just the URL adds nothing the URL does not already say.
+    if label.lower().startswith(("http://", "https://")):
+        return
+    if any(marker in target for marker in _SKIP_HREF):
+        return
+    if len(label) > 100:
+        label = label[:97].rstrip() + "…"
+    entry = f"- {label} -> {target}"
+    if entry in links or len(links) >= MAX_LINKS:
+        return
+    links.append(entry)
+
+
+def _join(parts: list[str]) -> str:
+    """Glue text pieces back together without gluing their words.
+
+    trafilatura's html output drops the spaces around loose inline elements —
+    a wiki sentence comes back as ``C++andQt`` — so two pieces
+    that both start and end on a letter get a space between them. Punctuation
+    keeps its neighbours: ``(hackerbooks.org)`` stays in parentheses.
+    """
+    out = ""
+    for part in parts:
+        if not part:
+            continue
+        if out and not out[-1].isspace() and not part[0].isspace() \
+                and out[-1].isalnum() and part[0].isalnum():
+            out += " "
+        out += part
+    return _SPACES.sub(" ", out).strip()
+
+
+def _inline(el: Any, base: str, links: list[str]) -> str:
+    """Text of an element that sits inside a paragraph, with links harvested."""
+    parts: list[str] = [el.text or ""]
+    for child in el:
+        tag = _tag(child)
+        if tag == "a":
+            label = _inline(child, base, links)
+            _collect_links(label, child.get("href"), base, links)
+            parts.append(label)
+        elif tag in ("strong", "b"):
+            inner = _inline(child, base, links).strip()
+            parts.append(f"**{inner}**" if inner else "")
+        elif tag == "img":
+            # The image itself is unavailable as text; its alt text usually is not.
+            parts.append((child.get("alt") or "").strip())
+        elif tag == "br":
+            parts.append(" ")
+        elif tag == "pre":
+            parts.append(_code(child))
+        elif tag in _INLINE or tag in ("i", "em", "code", "tt"):
+            parts.append(_inline(child, base, links))
+        else:
+            parts.append(_inline(child, base, links))
+        parts.append(child.tail or "")
+    return _join(parts)
+
+
+def _code(el: Any) -> str:
+    """A 
 as a fence that keeps its own indentation.
+
+    trafilatura emits its own one-liners as 
 too — ``
asyncio.run()
`` + inside a sentence — so a short single line becomes inline code. + """ + code = "".join(el.itertext()).replace(" ", " ").strip("\n") + if not code.strip(): + return "" + if "\n" not in code.strip() and len(code.strip()) <= 60: + return f"`{code.strip()}`" + return f"```\n{code.rstrip()}\n```" + + +def _list(el: Any, base: str, links: list[str]) -> str: + ordered = _tag(el) == "ol" + items: list[str] = [] + number = 0 + for child in el: + if _tag(child) != "li": + continue + number += 1 + marker = f"{number}. " if ordered else "- " + body = _block(child, base, links).strip() + if not body: + continue + lines = body.splitlines() + pad = " " * len(marker) + items.append(marker + lines[0] + "".join("\n" + pad + ln for ln in lines[1:])) + return "\n".join(items) + + +def _row(el: Any, base: str, links: list[str]) -> str: + """A table row — HN, wiki infoboxes — as one line of columns. + + trafilatura names its own rows ``/`` and keeps real ones as + ``/``, with no column model either way and a colspan here or there, + so a plain ``|``-joined line keeps the shape without inventing a header. + """ + cells: list[str] = [] + nested: list[str] = [] + for child in el: + tag = _tag(child) + text = _block(child, base, links).replace("\n", " ").strip() + if not text: + continue + (nested if tag in _ROWS else cells).append(text) + line = " | ".join(cells) + return "\n".join(part for part in [line, *nested] if part) + + +def _block(el: Any, base: str, links: list[str]) -> str: + tag = _tag(el) + if tag in _HEADINGS: + title = _inline(el, base, links) + return f"{'#' * _HEADINGS[tag]} {title}" if title else "" + if tag == "pre": + return _code(el) + if tag in ("ul", "ol"): + return _list(el, base, links) + if tag == "blockquote": + inner = _blocks(el, base, links) + return "\n".join(f"> {line}".rstrip() if line else ">" for line in inner.splitlines()) + if tag in _ROWS: + return _row(el, base, links) + if tag == "hr": + return "---" + return _inline(el, base, links) + + +def _render_child(el: Any, base: str, links: list[str]) -> str: + """A container's child as markdown: blocks that contain blocks, or a leaf.""" + tag = _tag(el) + if tag in _CONTAINERS: + return _blocks(el, base, links) + if tag in ("table", "tbody", "thead", "tfoot", "dl"): + return _join_pieces([ + ("row" if _tag(grandchild) in _ROWS else _tag(grandchild), + _block(grandchild, base, links)) + for grandchild in el + ]) + return _block(el, base, links) + + +def _join_pieces(pieces: list[tuple[str, str]]) -> str: + """Blank line between blocks — except between table rows, which stay adjacent. + + news.ycombinator.com is a table, and a blank line after every row is what + turned it into a page-tall wall before. + """ + out = "" + previous = "" + for kind, text in pieces: + if not text.strip(): + continue + if not out: + out = text + elif kind == previous == "row": + out += "\n" + text + else: + out += "\n\n" + text + previous = kind + return out + + +def _blocks(el: Any, base: str, links: list[str]) -> str: + """Children of a container, one block each.""" + pieces: list[tuple[str, str]] = [] + if (el.text or "").strip(): + pieces.append(("text", _SPACES.sub(" ", el.text or "").strip())) + for child in el: + tag = _tag(child) + pieces.append((tag, _render_child(child, base, links))) + if (child.tail or "").strip(): + pieces.append(("text", _SPACES.sub(" ", child.tail or "").strip())) + return _join_pieces(pieces) + + +def _tidy(text: str) -> str: + """Collapse the blank runs the renderer leaves behind; never touch a fence.""" + out: list[str] = [] + fence = False + blank = 0 + for line in text.splitlines(): + if line.lstrip().startswith("```"): + fence = not fence + out.append(line.rstrip()) + blank = 0 + continue + if fence: + out.append(line.rstrip()) + continue + line = line.rstrip() + if not line.strip(): + blank += 1 + if blank > 1: + continue + out.append("") + continue + blank = 0 + out.append(line.strip()) + # A pilcrow is how documentation sites mark a heading's own anchor. + return "\n".join(out).replace("¶", "").strip() + + +def _extract(html: str, url: str, comments: bool) -> tuple[str, list[str]]: + """One trafilatura pass, rendered. ``("", [])`` when the page has no content.""" + try: + fragment = trafilatura.extract( + html, + url=url, + output_format="html", + include_links=True, + include_tables=True, + include_comments=comments, + include_formatting=True, + ) + except Exception: # noqa: BLE001 — a broken page must not break the tool + return "", [] + if not fragment: + return "", [] + + try: + tree = lxml_html.fromstring(fragment) + except Exception: # noqa: BLE001 — same + return "", [] + + links: list[str] = [] + return _tidy(_blocks(tree, url, links)), links + + +def extract_page( + html: str, url: str, visible_chars: int = 0 +) -> tuple[str, list[str]] | None: + """Main content of a page as markdown, plus the links worth following. + + ``None`` means the extractor found no text worth sending — the caller falls + back to reading the page's own visible text rather than returning a menu. + """ + body, links = _extract(html, url, comments=False) + + # A page whose content *is* its comment thread — a forum, a Q&A, a code + # review — comes back as little more than a header when comments are off. + # Keeping less than a quarter of what the reader can see is the signal that + # the extractor threw the point of the page away, so ask again for all of it. + if visible_chars and len(body) < visible_chars // 4: + wide_body, wide_links = _extract(html, url, comments=True) + if len(wide_body) > len(body): + body, links = wide_body, wide_links + + if len(body) < MIN_TEXT: + return None + return body, links + + +def truncate(text: str, limit: int) -> str: + """Cut at a paragraph boundary and name the sections that did not fit. + + Cutting mid-sentence left the model with a half-page and no sign that there + was more; naming the headings that were dropped tells it whether the rest of + the page is worth another request. + """ + if len(text) <= limit: + return text + cut = text.rfind("\n\n", 0, limit) + if cut < limit // 2: + cut = limit + head, tail = text[:cut].rstrip(), text[cut:] + missing = [line.strip() for line in tail.splitlines() if _HEADING.match(line)] + + note = f"[… truncated at {limit} chars]" + if missing: + shown = ", ".join(missing[:8]) + more = f" and {len(missing) - 8} more" if len(missing) > 8 else "" + note = f"[… truncated at {limit} chars. Sections not shown: {shown}{more}]" + if head.count("```") % 2: + head += "\n```" + return f"{head}\n\n{note}" + + +def clean_visible_text(text: str) -> str: + """Fallback path: the page's own visible text, with blank runs collapsed.""" + lines = [line.rstrip() for line in text.splitlines()] + result: list[str] = [] + blank_run = 0 + for line in lines: + if line == "": + blank_run += 1 + if blank_run <= 2: + result.append("") + continue + blank_run = 0 + result.append(line) + return "\n".join(result).strip() diff --git a/mcp-servers/navi-web/app/mcp_server.py b/mcp-servers/navi-web/app/mcp_server.py index 4f910bb..8957f83 100644 --- a/mcp-servers/navi-web/app/mcp_server.py +++ b/mcp-servers/navi-web/app/mcp_server.py @@ -4,12 +4,13 @@ import json import os +from contextlib import asynccontextmanager from typing import Annotated, Any from mcp.server.fastmcp import FastMCP from pydantic import Field -from .browse import web_view +from .browse import close_browser, web_view from .config import Settings from .request import http_request from .search import web_search @@ -31,7 +32,16 @@ No session_id or filesystem paths are required. """.strip() -mcp = FastMCP("navi-web", instructions=INSTRUCTIONS) +@asynccontextmanager +async def _lifespan(_server: FastMCP): + """Keep one Chromium for the server's lifetime; stop it on shutdown.""" + try: + yield {} + finally: + await close_browser() + + +mcp = FastMCP("navi-web", instructions=INSTRUCTIONS, lifespan=_lifespan) def _json(data: Any) -> str: @@ -44,18 +54,40 @@ @mcp.tool(name="web_search") async def search_tool( - query: Annotated[str, Field(description="Search query.")], + query: Annotated[ + str | list[str], + Field( + description=( + "One query, or up to 3 independent queries, which run in parallel." + ) + ), + ], max_results: Annotated[int, Field(description="Number of results (default 5).")] = 5, ) -> str: - """Search the web (SearXNG primary, DDG fallback, Brave tertiary).""" + """Search the web (SearXNG primary, DDG fallback, Brave tertiary). + + Pass several queries in one call when you already know they are independent: + they run concurrently, so the call costs the slowest one instead of the sum. + """ result = await web_search(_settings(), query, max_results) + if not result["success"]: + # A failed search is a failed tool call. Returning the JSON with + # "success": false kept the journal green — the MCP call itself + # succeeded — so a broken search left no trace except the model's own + # wording. Raising turns it into ``isError``, which Navi logs. + raise RuntimeError(result["output"]) return _json(result) @mcp.tool(name="web_view") async def view_tool( - url: Annotated[str, Field(description="Full URL to open (must start with http:// or https://).")], - screenshot: Annotated[bool, Field(description="Also capture a screenshot.")] = False, + url: Annotated[ + str | list[str], + Field( + description="One URL, or up to 3 URLs, which are opened in parallel." + ), + ], + screenshot: Annotated[bool, Field(description="Also capture a screenshot (single URL only).")] = False, wait_until: Annotated[ str, Field( @@ -63,8 +95,22 @@ ), ] = "networkidle", ) -> str: - """Open a URL in a headless browser and return clean readable text.""" + """Open a URL in a headless browser and return its main content as markdown. + + Navigation, ads and footers are stripped; headings, code blocks and table + rows keep their shape. Links from the page follow the text under + "Links on this page" with absolute URLs, so you can open one by passing it + to this tool. + + Pass several URLs in one call when you already know which pages you need: + they open concurrently, so three pages cost about one page's time. + """ result = await web_view(url, screenshot=screenshot, wait_until=wait_until) + if not result["success"]: + # Same rule as the search: no page opened is a failed tool call, not a + # JSON with "success": false that leaves the journal green. A page the + # browser could not launch at all used to look like an answer. + raise RuntimeError(result["output"]) return _json(result) diff --git a/mcp-servers/navi-web/app/search.py b/mcp-servers/navi-web/app/search.py index 588a4ab..48faf2a 100644 --- a/mcp-servers/navi-web/app/search.py +++ b/mcp-servers/navi-web/app/search.py @@ -10,50 +10,205 @@ from .config import Settings -_DDG_RETRY_DELAY = 2.0 -_TIMEOUT = 15.0 +# Every wait on this path is bounded, including the chain as a whole. The caller +# is sitting on this call, and the backends used to add up: 15s SearXNG + two 5s +# DDG attempts + a 2s pause + 15s Brave — half a minute in which a slow search +# and a broken one looked the same. The budgets below make the worst case +# ~12s, long enough for a healthy backend and short enough to still answer. +_SEARXNG_TIMEOUT = 6.0 +_DDG_TIMEOUT = 4.0 +_BRAVE_TIMEOUT = 6.0 +_DDG_RETRY_DELAY = 1.0 +_TOTAL_TIMEOUT = 12.0 + +# One call may carry several independent queries: they run concurrently, so the +# call costs the slowest instead of the sum. The cap is what keeps a batch from +# turning into a scrape of the search engine — three concurrent queries on one +# instance still answered in under a second here. +_MAX_BATCH = 3 + + +def _normalize(raw: str | list[str]) -> tuple[list[str], list[str]]: + """Turn the argument into the queries to run, plus notes worth telling the model.""" + queries: list[str] = [] + for item in [raw] if isinstance(raw, str) else raw: + query = str(item).strip() + if query and query not in queries: + queries.append(query) + notes: list[str] = [] + if len(queries) > _MAX_BATCH: + notes.append( + f"Only the first {_MAX_BATCH} of {len(queries)} queries ran — " + "ask the rest in another call." + ) + queries = queries[:_MAX_BATCH] + return queries, notes async def web_search( settings: Settings, - query: str, + query: str | list[str], max_results: int = 5, ) -> dict[str, Any]: """Search the web. Returns dict with success, output, error, metadata.""" + queries, notes = _normalize(query) + if not queries: + return { + "success": False, + "output": "No query given.", + "error": "empty_query", + "metadata": {}, + } + + # ``return_exceptions`` rather than a try/except: ``_search_one`` already + # turns every backend failure into a result, so an exception here is one + # query's own, and the batch keeps the rest. + outcomes = await asyncio.gather( + *(_search_one(settings, q, max_results) for q in queries), + return_exceptions=True, + ) + + if len(queries) == 1 and not notes: + outcome = outcomes[0] + if isinstance(outcome, BaseException): + return { + "success": False, + "output": f"Search failed: {outcome}", + "error": str(outcome), + "metadata": {}, + } + return outcome + + return _combine(queries, outcomes, notes) + + +def _combine( + queries: list[str], + outcomes: list[Any], + notes: list[str], +) -> dict[str, Any]: + """One block per query. A query that failed is named, not dropped: the model + can retry that one instead of repeating the batch, and a batch that answered + something is a success even when the rest did not.""" + blocks: list[str] = [] + per_query: list[dict[str, Any]] = [] + answered = 0 + + for query, outcome in zip(queries, outcomes, strict=True): + if isinstance(outcome, BaseException): + blocks.append(f'### "{query}" -> failed: {outcome}') + per_query.append({"query": query, "success": False, "error": str(outcome)}) + elif outcome["success"]: + answered += 1 + blocks.append(f'### "{query}"\n{outcome["output"]}') + per_query.append( + { + "query": query, + "success": True, + "source": outcome["metadata"].get("source"), + } + ) + else: + blocks.append(f'### "{query}" -> failed: {outcome["error"]}') + per_query.append( + {"query": query, "success": False, "error": outcome["error"]} + ) + + body = "\n\n".join(notes + blocks) + if not answered: + reasons = "; ".join( + f"{item['query']}: {item['error']}" for item in per_query + ) + return { + "success": False, + "output": f"No query produced results. {reasons}", + "error": "all_queries_failed", + "metadata": {"queries": per_query}, + } + return { + "success": True, + "output": body, + "error": None, + "metadata": {"queries": per_query}, + } + + +async def _search_one( + settings: Settings, + query: str, + max_results: int, +) -> dict[str, Any]: + """One query, bounded by the chain's own budget. + + The deadline is per query on purpose: a batch waits for the slowest one, so + putting a single ceiling around the whole batch would let one hung backend + erase the answers the other queries already had. + """ + try: + return await asyncio.wait_for( + _search_chain(settings, query, max_results), timeout=_TOTAL_TIMEOUT + ) + except TimeoutError: + return { + "success": False, + "output": ( + f"Search did not finish within {_TOTAL_TIMEOUT:.0f}s " + f"(SearXNG -> DDG -> Brave) for query: {query!r}" + ), + "error": "timeout", + "metadata": {}, + } + + +async def _search_chain( + settings: Settings, + query: str, + max_results: int, +) -> dict[str, Any]: last_error = "" - # 1. SearXNG (primary) + # 1. SearXNG (primary). Its engine list is pinned on purpose: the instance's + # own defaults are mostly suspended, and without ``engines`` it returns + # nothing at all. if settings.searxng_url: try: - results = await _searxng(settings, query, max_results) + results = await asyncio.wait_for( + _searxng(settings, query, max_results), timeout=_SEARXNG_TIMEOUT + ) if results: return _format(results, source="SearXNG") last_error = "SearXNG: empty response" + except TimeoutError: + last_error = f"SearXNG: no answer in {_SEARXNG_TIMEOUT:.0f}s" except Exception as e: last_error = f"SearXNG: {e}" - # 2. DuckDuckGo html fallback, retry once + # 2. DuckDuckGo html fallback. One attempt is cheap; a second is worth it + # only when the first *failed* (a transient block or reset). An empty result + # is an answer — retrying it only buys another timeout. for attempt in range(2): try: - results = await asyncio.to_thread( - lambda: list( - DDGS().text(query, backend="html", max_results=max_results, region=settings.ddg_region) - ) - ) + results = await asyncio.to_thread(_ddg, settings, query, max_results) if results: return _format(results, source="DDG") last_error = "DDG: empty response" + break except Exception as e: last_error = f"DDG: {e}" - if attempt == 0: - await asyncio.sleep(_DDG_RETRY_DELAY) + if attempt == 0: + await asyncio.sleep(_DDG_RETRY_DELAY) - # 3. Brave Search fallback + # 3. Brave Search fallback, only when a key is configured. if settings.brave_api_key: try: - results = await _brave(settings, query, max_results) + results = await asyncio.wait_for( + _brave(settings, query, max_results), timeout=_BRAVE_TIMEOUT + ) if results: return _format(results, source="Brave") + last_error = "Brave: empty response" + except TimeoutError: + last_error = f"Brave: no answer in {_BRAVE_TIMEOUT:.0f}s" except Exception as e: last_error = f"Brave: {e}" @@ -65,8 +220,25 @@ } +def _ddg(settings: Settings, query: str, max_results: int) -> list[dict[str, Any]]: + """One DuckDuckGo attempt, off the event loop. + + ``timeout`` is a ``DDGS`` client argument, not a ``text()`` one — passing it + to ``text()`` would land in ``**kwargs`` and do nothing. + """ + ddgs = DDGS(timeout=int(_DDG_TIMEOUT)) + return list( + ddgs.text( + query, + backend="html", + max_results=max_results, + region=settings.ddg_region, + ) + ) + + async def _searxng(settings: Settings, query: str, max_results: int) -> list[dict]: - async with httpx.AsyncClient(timeout=_TIMEOUT) as client: + async with httpx.AsyncClient(timeout=_SEARXNG_TIMEOUT) as client: resp = await client.get( f"{settings.searxng_url}/search", params={ @@ -85,7 +257,7 @@ async def _brave(settings: Settings, query: str, max_results: int) -> list[dict]: - async with httpx.AsyncClient(timeout=_TIMEOUT) as client: + async with httpx.AsyncClient(timeout=_BRAVE_TIMEOUT) as client: resp = await client.get( "https://api.search.brave.com/res/v1/web/search", params={"q": query, "count": min(max_results, 20), "country": "ALL"}, diff --git a/mcp-servers/navi-web/pyproject.toml b/mcp-servers/navi-web/pyproject.toml index 4beab11..08f2b6a 100644 --- a/mcp-servers/navi-web/pyproject.toml +++ b/mcp-servers/navi-web/pyproject.toml @@ -13,6 +13,7 @@ "ddgs>=1.0", "playwright>=1.40", "pydantic>=2.0", + "trafilatura>=2.0", ] [project.scripts] diff --git a/mcp_servers.d/navi-web.json b/mcp_servers.d/navi-web.json index 1d166ca..1274aaf 100644 --- a/mcp_servers.d/navi-web.json +++ b/mcp_servers.d/navi-web.json @@ -7,7 +7,8 @@ ], "cwd": "./mcp-servers/navi-web", "env": { - "NAVI_WEB_MCP_TRANSPORT": "stdio" + "NAVI_WEB_MCP_TRANSPORT": "stdio", + "NAVI_WEB_SEARXNG_URL": "${SEARXNG_URL:-}" }, "groups": { "search": [ @@ -20,6 +21,6 @@ "http_request" ] }, - "instructions": "Navi Web MCP server provides web search, browsing, and raw HTTP tools.\n\nUse it when the task involves:\n- searching the web for current info, docs, or real-time data;\n- opening a URL in a browser to read human-readable content;\n- making REST API calls, webhooks, or raw HTTP requests.\n\nWorkflow:\n1. search — find relevant pages or facts.\n2. view — open promising URLs to read full content.\n3. request — call APIs or services requiring headers/auth.\n\nAll three tools are stateless and work with public URLs.\nNo session_id or filesystem paths are required.\n\nABSOLUTE RULE — NEVER bypass MCP tools:\nYou MUST NOT use filesystem, terminal, code_exec, or any direct file access to read or write web content. Use only the MCP tools listed above.", - "summary": "Web search, page browsing and raw HTTP. Never fetch web content with filesystem, terminal or code_exec — these are the only valid interface. Details: tool_manual(\"navi-web\")." + "instructions": "Navi Web MCP server provides web search, browsing, and raw HTTP tools.\n\nUse it when the task involves:\n- searching the web for current info, docs, or real-time data;\n- opening a URL in a browser to read human-readable content;\n- making REST API calls, webhooks, or raw HTTP requests.\n\nWorkflow:\n1. search — find relevant pages or facts.\n2. view — open promising URLs to read full content.\n3. request — call APIs or services requiring headers/auth.\n\nBatching: web_search takes one query or up to 3, web_view one URL or up to 3 — several run in parallel, so the call costs the slowest instead of the sum. Use it when you already know the lookups are independent: after a search, open the 2-3 pages you need in one web_view call; when a request needs several unrelated facts, ask the searches together. A failed item in a batch is named in the output rather than dropping the whole call. Do not batch a follow-up that depends on what the previous answer said — that is a sequence, not a batch.\n\nAll three tools are stateless and work with public URLs.\nNo session_id or filesystem paths are required.\n\nABSOLUTE RULE — NEVER bypass MCP tools:\nYou MUST NOT use filesystem, terminal, code_exec, or any direct file access to read or write web content. Use only the MCP tools listed above.", + "summary": "Web search, page browsing and raw HTTP; up to 3 queries or URLs in one call run in parallel, so batch independent lookups. Never fetch web content with filesystem, terminal or code_exec — these are the only valid interface. Details: tool_manual(\"navi-web\")." } diff --git a/pyproject.toml b/pyproject.toml index 289d04a..ee7fe64 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -83,6 +83,9 @@ "pytest-asyncio>=0.23", "ruff>=0.4", "mypy>=1.10", + # The navi-web MCP server renders pages through trafilatura; its tests import + # that server by path, so the package has to exist in this venv too. + "trafilatura>=2.0", ] [tool.hatch.metadata] diff --git a/tests/unit/mcp/conftest.py b/tests/unit/mcp/conftest.py new file mode 100644 index 0000000..2ebb42c --- /dev/null +++ b/tests/unit/mcp/conftest.py @@ -0,0 +1,77 @@ +"""Load `mcp-servers//app` for tests. + +The MCP servers are separate processes with their own venv, not packages of +this project, and `pyproject.toml` collects `tests/` only — so a test reaches +them by path. A relative import inside them (`from .config import Settings`) +needs a real package entry in `sys.modules` with a search path, which is what +``_load`` builds. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path +from types import ModuleType + +import pytest + +_SERVERS = Path(__file__).resolve().parents[3] / "mcp-servers" + + +def _load(server: str, module: str | None = None) -> ModuleType: + app_dir = _SERVERS / server / "app" + package = "mcp_" + server.replace("-", "_") + "_app" + + if package not in sys.modules: + spec = importlib.util.spec_from_file_location( + package, app_dir / "__init__.py", submodule_search_locations=[str(app_dir)] + ) + assert spec and spec.loader + package_module = importlib.util.module_from_spec(spec) + sys.modules[package] = package_module + spec.loader.exec_module(package_module) + + if module is None: + return sys.modules[package] + + name = f"{package}.{module}" + if name not in sys.modules: + spec = importlib.util.spec_from_file_location(name, app_dir / f"{module}.py") + assert spec and spec.loader + module_object = importlib.util.module_from_spec(spec) + sys.modules[name] = module_object + spec.loader.exec_module(module_object) + return sys.modules[name] + + +@pytest.fixture(scope="session") +def navi_web_search(): + return _load("navi-web", "search") + + +@pytest.fixture(scope="session") +def navi_web_browse(): + return _load("navi-web", "browse") + + +@pytest.fixture(scope="session") +def navi_web_extract(): + """Page-to-markdown rendering — the part that needs no browser.""" + return _load("navi-web", "extract") + + +@pytest.fixture(scope="session") +def navi_web_server(): + """The FastMCP tool layer — where ``isError`` is decided.""" + return _load("navi-web", "mcp_server") + + +@pytest.fixture +def navi_web_settings(): + """Settings with every optional backend off, so each test enables its own.""" + settings = _load("navi-web", "config").Settings() + settings.searxng_url = "" + settings.brave_api_key = "" + settings.ddg_region = "ru-ru" + return settings diff --git a/tests/unit/mcp/test_navi_web_extract.py b/tests/unit/mcp/test_navi_web_extract.py new file mode 100644 index 0000000..59b6fb0 --- /dev/null +++ b/tests/unit/mcp/test_navi_web_extract.py @@ -0,0 +1,251 @@ +"""web_view must send the page, not the page's furniture. + +The old path was `innerText` of the body after a tag kill-list: navigation +crumbs survived, tables collapsed into one line per thousand characters +(news.ycombinator.com came back as four lines), headings were unmarked and +every link was gone. These tests pin the replacement: main content only, +structure kept, links lifted into an appendix. +""" + +PAGE = """Guide + +
+

Guide

+

First paragraph of the guide: real prose that the reader came here for, long +enough to be kept by the extractor without any doubt at all about it.

+
def f():
+    if True:
+        return 1
+
+

Second section

+

Second paragraph, also prose, also worth keeping: it explains the second +section in a sentence or two, which is what the reader wanted.

+

Read the docs and +another page.

+

Third paragraph is here only so the page clears the extractor's minimum +length, and it says nothing except that it exists.

+
+
© 2026 Boilerplate Inc.
+""" + + +class TestExtraction: + def test_the_content_survives_and_the_chrome_does_not(self, navi_web_extract): + body, _ = navi_web_extract.extract_page(PAGE, "https://example.com/a/b") + + assert "First paragraph of the guide" in body + assert "Second section" in body + assert "About us" not in body + assert "Boilerplate" not in body + + def test_headings_and_code_keep_their_shape(self, navi_web_extract): + """The fence matters: markdown output from trafilatura flattens + `` return 1`` to column 0, which is not Python any more.""" + body, _ = navi_web_extract.extract_page(PAGE, "https://example.com/a/b") + + assert "# Guide" in body + assert "## Second section" in body + assert "```\ndef f():\n if True:\n return 1\n```" in body + + def test_links_move_to_an_appendix_with_absolute_urls(self, navi_web_extract): + body, links = navi_web_extract.extract_page(PAGE, "https://example.com/a/b") + + assert "Read the docs" in body + assert "https://example.com/docs/page" not in body + assert links == [ + "- Read the docs -> https://example.com/docs/page", + "- another page -> https://other.example/x", + ] + + def test_a_page_without_text_is_not_a_page(self, navi_web_extract): + html = "" + + assert navi_web_extract.extract_page(html, "https://example.com/") is None + + def test_a_thread_is_read_with_its_comments(self, navi_web_extract, monkeypatch): + """Forum pages are their comments: a header-only extract is a miss, and + the signal is how much of the visible page the extractor threw away.""" + calls: list[bool] = [] + + def fake_extract(html, url, comments): + calls.append(comments) + if comments: + return "x" * 5_000, ["- reply -> https://forum.example/1"] + return "x" * 100, [] + + monkeypatch.setattr(navi_web_extract, "_extract", fake_extract) + + body, links = navi_web_extract.extract_page( + "", "https://forum.example/1", visible_chars=100_000 + ) + + assert calls == [False, True] + assert len(body) == 5_000 + assert links == ["- reply -> https://forum.example/1"] + + def test_a_page_that_was_understood_is_not_read_twice( + self, navi_web_extract, monkeypatch + ): + calls: list[bool] = [] + + def fake_extract(html, url, comments): + calls.append(comments) + return "x" * 1_000, [] + + monkeypatch.setattr(navi_web_extract, "_extract", fake_extract) + + navi_web_extract.extract_page( + "", "https://blog.example/1", visible_chars=2_000 + ) + + assert calls == [False] + + +class TestLinkAppendix: + def test_noise_links_are_dropped(self, navi_web_extract): + page = """

Prose that is long enough to be kept, and it goes on for a while: it says +a few things about the subject at hand, which is what the reader came for, and +it keeps saying them until the extractor is satisfied.

+

Then it links onwards: upvote, +alice, https://bare.example/x, +and The real target, which is the only one worth following.

+
""" + + _, links = navi_web_extract.extract_page(page, "https://example.com/post") + + assert links == ["- The real target -> https://example.com/real"] + + def test_the_appendix_is_capped(self, navi_web_extract): + anchors = "".join(f'page {i} ' for i in range(80)) + page = f"""

Prose that is long enough to be kept, and it goes on for a while: it says +a few things about the subject at hand, which is what the reader came for, and +it keeps saying them until the extractor is satisfied, one page at a time.

+

{anchors}

""" + + _, links = navi_web_extract.extract_page(page, "https://example.com/") + + assert len(links) == navi_web_extract.MAX_LINKS + + def test_the_same_link_is_listed_once(self, navi_web_extract): + page = """

Prose that is long enough to be kept, and it goes on for a while: it says +a few things about the subject at hand, which is what the reader came for, and +it keeps saying them until the extractor is satisfied.

+

Both link to The real target and again to +The real target, which should still be one line.

+
""" + + _, links = navi_web_extract.extract_page(page, "https://example.com/") + + assert links == ["- The real target -> https://example.com/real"] + + +class TestTruncate: + def test_short_text_is_untouched(self, navi_web_extract): + assert navi_web_extract.truncate("# T\n\nbody", 100) == "# T\n\nbody" + + def test_a_cut_names_the_sections_that_did_not_fit(self, navi_web_extract): + text = "# A\n\n" + "a" * 400 + "\n\n## Kept\n\n" + "b" * 400 + "\n\n## Lost\n\n" + "c" * 400 + + cut = navi_web_extract.truncate(text, 500) + + assert cut.startswith("# A") + assert "## Kept" in cut + assert "## Lost" not in cut.split("[")[0] + assert "Sections not shown: ## Lost" in cut + + def test_a_cut_inside_a_fence_closes_it(self, navi_web_extract): + text = "# T\n\n```\n" + "code line\n" * 100 + + cut = navi_web_extract.truncate(text, 300) + + assert cut.count("```") % 2 == 0 + + def test_the_cut_lands_on_a_paragraph_boundary(self, navi_web_extract): + text = ("paragraph words " * 10 + "\n\n") * 10 + + cut = navi_web_extract.truncate(text, 300) + + assert cut.split("[")[0].rstrip().endswith("words") + + +class _FakePage: + def __init__(self, html: str, visible: str, title: str = "Guide"): + self._html = html + self._visible = visible + self._title = title + self.url = "https://example.com/final" + self.closed = False + + async def goto(self, *args, **kwargs): + return None + + async def title(self): + return self._title + + async def evaluate(self, _script): + return self._visible + + async def content(self): + return self._html + + async def screenshot(self, **kwargs): + return b"\x89PNG" + + async def close(self): + self.closed = True + + +class _FakeContext: + def __init__(self, page: _FakePage): + self._page = page + + async def new_page(self): + return self._page + + +class TestReadPage: + async def test_the_output_carries_the_body_and_the_links(self, navi_web_browse): + page = _FakePage(PAGE, "visible text") + + result = await navi_web_browse._read_page( + _FakeContext(page), "https://example.com/a", False, "networkidle" + ) + + assert result["output"].startswith("Title: Guide\nFinal URL: https://example.com/final") + assert "## Second section" in result["output"] + assert "Links on this page:\n- Read the docs -> https://example.com/docs/page" in result["output"] + assert result["metadata"] == {"extractor": "trafilatura", "links": 2} + + async def test_a_page_the_extractor_cannot_read_falls_back_to_its_text( + self, navi_web_browse, monkeypatch + ): + monkeypatch.setattr(navi_web_browse, "extract_page", lambda *a, **k: None) + page = _FakePage("", "plain visible text\n\n\n\nafter blanks") + + result = await navi_web_browse._read_page( + _FakeContext(page), "https://example.com/a", False, "networkidle" + ) + + assert "plain visible text" in result["output"] + assert result["metadata"]["extractor"] == "innerText" + assert result["metadata"]["links"] == 0 + + async def test_a_long_page_is_cut_with_a_note(self, navi_web_browse, monkeypatch): + page = _FakePage(PAGE, "visible text") + monkeypatch.setattr(navi_web_browse, "_MAX_TEXT", 120) + + result = await navi_web_browse._read_page( + _FakeContext(page), "https://example.com/a", False, "networkidle" + ) + + assert "truncated at 120 chars" in result["output"] + + async def test_a_screenshot_keeps_the_text_metadata(self, navi_web_browse): + page = _FakePage(PAGE, "visible text") + + result = await navi_web_browse._read_page( + _FakeContext(page), "https://example.com/a", True, "networkidle" + ) + + assert result["metadata"]["is_image"] is True + assert result["metadata"]["extractor"] == "trafilatura" diff --git a/tests/unit/mcp/test_navi_web_search.py b/tests/unit/mcp/test_navi_web_search.py new file mode 100644 index 0000000..20d692c --- /dev/null +++ b/tests/unit/mcp/test_navi_web_search.py @@ -0,0 +1,360 @@ +"""web_search must prefer SearXNG, fall back in order, and never run unbounded. + +The server is a separate stdio process: its environment is fixed at connect time, +so the SearXNG URL reaches it only through the ``${SEARXNG_URL:-}`` placeholder in +``mcp_servers.d/navi-web.json``. What these tests pin down is the chain itself — +order, per-backend budgets, the retry rule, the batch rules — because until +2026-11 those waits added up to half a minute and a broken backend was +indistinguishable from a slow one. + +These tests live in the root suite rather than in ``mcp-servers/navi-web/tests/`` +because ``pyproject.toml`` collects ``testpaths = ["tests"]`` only; the loader is +in ``conftest.py`` next to this file. +""" + +import asyncio +import time + + +def _hit(title: str = "t") -> dict: + return {"title": title, "url": "https://example.com", "content": ""} + + +def _ok(source: str = "SearXNG") -> dict: + return {"success": True, "output": f"[{source}]\n[1] t", "error": None, + "metadata": {"source": source, "results": [_hit()]}} + + +class TestOrder: + async def test_searxng_answers_first(self, navi_web_search, monkeypatch, navi_web_settings): + calls: list[str] = [] + + async def searxng(s, q, n): + calls.append("searxng") + return [_hit()] + + def ddg(s, q, n): + calls.append("ddg") + return [_hit()] + + monkeypatch.setattr(navi_web_search, "_searxng", searxng) + monkeypatch.setattr(navi_web_search, "_ddg", ddg) + navi_web_settings.searxng_url = "http://searx" + + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["success"] is True + assert result["metadata"]["source"] == "SearXNG" + assert calls == ["searxng"] + + async def test_empty_searxng_falls_back_to_ddg( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def searxng(s, q, n): + return [] + + monkeypatch.setattr(navi_web_search, "_searxng", searxng) + monkeypatch.setattr(navi_web_search, "_ddg", lambda s, q, n: [_hit()]) + navi_web_settings.searxng_url = "http://searx" + + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["metadata"]["source"] == "DDG" + + async def test_brave_is_last_and_needs_a_key( + self, navi_web_search, monkeypatch, navi_web_settings + ): + calls: list[str] = [] + + async def searxng(s, q, n): + calls.append("searxng") + return [] + + def ddg(s, q, n): + calls.append("ddg") + return [] + + async def brave(s, q, n): + calls.append("brave") + return [_hit()] + + monkeypatch.setattr(navi_web_search, "_searxng", searxng) + monkeypatch.setattr(navi_web_search, "_ddg", ddg) + monkeypatch.setattr(navi_web_search, "_brave", brave) + navi_web_settings.searxng_url = "http://searx" + + without_key = await navi_web_search.web_search(navi_web_settings, "q") + assert without_key["success"] is False + assert "brave" not in calls + + navi_web_settings.brave_api_key = "key" + with_key = await navi_web_search.web_search(navi_web_settings, "q") + assert with_key["metadata"]["source"] == "Brave" + assert calls[-1] == "brave" + + async def test_all_backends_fail_names_the_last_error( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def searxng(s, q, n): + return [] + + monkeypatch.setattr(navi_web_search, "_searxng", searxng) + monkeypatch.setattr(navi_web_search, "_ddg", lambda s, q, n: []) + navi_web_settings.searxng_url = "http://searx" + + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["success"] is False + assert result["error"] == "DDG: empty response" + assert "All search backends failed" in result["output"] + + +class TestBudgets: + async def test_slow_searxng_falls_through_quickly( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def slow(s, q, n): + await asyncio.sleep(10) + + monkeypatch.setattr(navi_web_search, "_searxng", slow) + monkeypatch.setattr(navi_web_search, "_ddg", lambda s, q, n: [_hit()]) + monkeypatch.setattr(navi_web_search, "_SEARXNG_TIMEOUT", 0.05) + navi_web_settings.searxng_url = "http://searx" + + started = time.monotonic() + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["metadata"]["source"] == "DDG" + assert time.monotonic() - started < 1.0 + + async def test_total_budget_is_the_hard_cap( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def slow(s, q, n): + await asyncio.sleep(10) + + monkeypatch.setattr(navi_web_search, "_searxng", slow) + monkeypatch.setattr(navi_web_search, "_TOTAL_TIMEOUT", 0.05) + navi_web_settings.searxng_url = "http://searx" + + started = time.monotonic() + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["success"] is False + assert result["error"] == "timeout" + assert time.monotonic() - started < 1.0 + + +class TestDdgRetry: + async def test_retries_once_after_an_error( + self, navi_web_search, monkeypatch, navi_web_settings + ): + attempts: list[int] = [] + + def flaky(s, q, n): + attempts.append(len(attempts) + 1) + if len(attempts) == 1: + raise RuntimeError("blocked") + return [_hit()] + + monkeypatch.setattr(navi_web_search, "_ddg", flaky) + monkeypatch.setattr(navi_web_search, "_DDG_RETRY_DELAY", 0.0) + + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["metadata"]["source"] == "DDG" + assert len(attempts) == 2 + + async def test_empty_result_is_not_retried( + self, navi_web_search, monkeypatch, navi_web_settings + ): + attempts: list[int] = [] + + def empty(s, q, n): + attempts.append(1) + return [] + + monkeypatch.setattr(navi_web_search, "_ddg", empty) + + result = await navi_web_search.web_search(navi_web_settings, "q") + + assert result["success"] is False + assert len(attempts) == 1 + + +class TestDdgClient: + def test_budget_and_arguments_reach_ddgs( + self, navi_web_search, monkeypatch, navi_web_settings + ): + seen: dict = {} + + class FakeDDGS: + def __init__(self, timeout=None): + seen["timeout"] = timeout + + def text(self, query, **kwargs): + seen["query"] = query + seen.update(kwargs) + return [_hit()] + + monkeypatch.setattr(navi_web_search, "DDGS", FakeDDGS) + + assert navi_web_search._ddg(navi_web_settings, "q", 3)[0]["title"] == "t" + assert seen["timeout"] == int(navi_web_search._DDG_TIMEOUT) + assert seen["query"] == "q" + assert seen["backend"] == "html" + assert seen["max_results"] == 3 + assert seen["region"] == "ru-ru" + + +class TestBatch: + """One call may carry up to three independent queries. + + The point of the batch is wall time — three queries cost the slowest, not the + sum — and the point of the rules around it is that a batch never hides which + query answered and which did not. + """ + + async def test_a_string_keeps_the_single_query_shape( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def chain(s, q, n): + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + result = await navi_web_search.web_search(navi_web_settings, "one") + + assert result["metadata"]["source"] == "SearXNG" + assert "###" not in result["output"] + + async def test_queries_run_concurrently( + self, navi_web_search, monkeypatch, navi_web_settings + ): + started: list[str] = [] + + async def chain(s, q, n): + started.append(q) + await asyncio.sleep(0.1) + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + began = time.monotonic() + result = await navi_web_search.web_search(navi_web_settings, ["a", "b", "c"]) + elapsed = time.monotonic() - began + + assert result["success"] is True + assert started == ["a", "b", "c"] + assert elapsed < 0.25, f"three 0.1s queries took {elapsed:.2f}s — not concurrent" + assert result["output"].count("### ") == 3 + + async def test_blocks_follow_the_request_order( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def chain(s, q, n): + await asyncio.sleep(0.1 if q == "a" else 0) + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + result = await navi_web_search.web_search(navi_web_settings, ["a", "b"]) + + assert result["output"].index('"a"') < result["output"].index('"b"') + + async def test_more_than_three_are_capped_with_a_note( + self, navi_web_search, monkeypatch, navi_web_settings + ): + started: list[str] = [] + + async def chain(s, q, n): + started.append(q) + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + result = await navi_web_search.web_search( + navi_web_settings, ["a", "b", "c", "d", "e"] + ) + + assert started == ["a", "b", "c"] + assert "Only the first 3 of 5 queries ran" in result["output"] + + async def test_blanks_and_duplicates_collapse( + self, navi_web_search, monkeypatch, navi_web_settings + ): + started: list[str] = [] + + async def chain(s, q, n): + started.append(q) + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + await navi_web_search.web_search(navi_web_settings, ["a", " ", "a", " a "]) + + assert started == ["a"] + + async def test_one_failed_query_does_not_erase_the_others( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def chain(s, q, n): + if q == "b": + return {"success": False, "output": "x", "error": "DDG: blocked", + "metadata": {}} + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + result = await navi_web_search.web_search(navi_web_settings, ["a", "b", "c"]) + + assert result["success"] is True + assert '### "b" -> failed: DDG: blocked' in result["output"] + assert [item["success"] for item in result["metadata"]["queries"]] == [ + True, False, True, + ] + + async def test_a_query_timing_out_does_not_erase_the_others( + self, navi_web_search, monkeypatch, navi_web_settings + ): + async def chain(s, q, n): + if q == "b": + await asyncio.sleep(10) + return _ok() + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + monkeypatch.setattr(navi_web_search, "_TOTAL_TIMEOUT", 0.05) + + result = await navi_web_search.web_search(navi_web_settings, ["a", "b"]) + + assert result["success"] is True + assert '### "b" -> failed: timeout' in result["output"] + + + + +class TestEmptyBatch: + async def test_no_query_is_an_error(self, navi_web_search, navi_web_settings): + result = await navi_web_search.web_search(navi_web_settings, [" ", ""]) + + assert result["success"] is False + assert result["error"] == "empty_query" + + async def test_a_batch_with_no_answers_fails( + self, navi_web_search, monkeypatch, navi_web_settings + ): + """The tool raises on this, which is what makes a dead search visible in + the journal — so a batch that answered nothing must stay a failure.""" + + async def chain(s, q, n): + return {"success": False, "output": "x", "error": "DDG: blocked", + "metadata": {}} + + monkeypatch.setattr(navi_web_search, "_search_chain", chain) + + result = await navi_web_search.web_search(navi_web_settings, ["a", "b"]) + + assert result["success"] is False + assert result["error"] == "all_queries_failed" + assert "a: DDG: blocked" in result["output"] + assert "b: DDG: blocked" in result["output"] diff --git a/tests/unit/mcp/test_navi_web_view.py b/tests/unit/mcp/test_navi_web_view.py new file mode 100644 index 0000000..78ab624 --- /dev/null +++ b/tests/unit/mcp/test_navi_web_view.py @@ -0,0 +1,250 @@ +"""web_view must open up to three URLs at once and never lose the pages it got. + +Pages take a second to several, so a batch is worth it only if it actually runs +concurrently — and only if the shared text budget and the per-URL deadlines keep +one slow or broken page from costing the others. The loader lives in +``conftest.py`` next to this file. +""" + +import asyncio +import time + + +def _page(text: str = "hello") -> dict: + return {"success": True, "output": f"Title: t\n\n{text}", "error": None, + "metadata": {}} + + +def _failed(error: str = "invalid_url") -> dict: + return {"success": False, "output": "URL must start with http:// or https://", + "error": error, "metadata": {}} + + +class TestSingleUrl: + async def test_one_url_keeps_the_plain_output( + self, navi_web_browse, monkeypatch + ): + seen: dict = {} + + async def view_one(url, screenshot=False, wait_until="networkidle"): + seen["url"] = url + seen["screenshot"] = screenshot + return _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view("https://example.com", screenshot=True) + + assert result == _page() + assert seen == {"url": "https://example.com", "screenshot": True} + + async def test_an_invalid_url_is_reported_without_launching( + self, navi_web_browse + ): + result = await navi_web_browse.web_view("ftp://example.com") + + assert result["success"] is False + assert result["error"] == "invalid_url" + + +class TestBatch: + async def test_urls_open_concurrently(self, navi_web_browse, monkeypatch): + opened: list[str] = [] + + async def view_one(url, screenshot=False, wait_until="networkidle"): + opened.append(url) + await asyncio.sleep(0.1) + return _page(url) + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + began = time.monotonic() + result = await navi_web_browse.web_view(["https://a", "https://b", "https://c"]) + elapsed = time.monotonic() - began + + assert result["success"] is True + assert opened == ["https://a", "https://b", "https://c"] + assert elapsed < 0.25, f"three 0.1s pages took {elapsed:.2f}s — not concurrent" + assert result["output"].count("### ") == 3 + + async def test_blocks_follow_the_request_order(self, navi_web_browse, monkeypatch): + async def view_one(url, screenshot=False, wait_until="networkidle"): + await asyncio.sleep(0.1 if url == "https://a" else 0) + return _page(url) + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view(["https://a", "https://b"]) + + assert result["output"].index("### https://a") < result["output"].index( + "### https://b" + ) + + async def test_more_than_three_are_capped_with_a_note( + self, navi_web_browse, monkeypatch + ): + opened: list[str] = [] + + async def view_one(url, screenshot=False, wait_until="networkidle"): + opened.append(url) + return _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view( + [f"https://{i}" for i in range(5)] + ) + + assert len(opened) == 3 + assert "Only the first 3 of 5 URLs were opened" in result["output"] + + async def test_duplicates_collapse(self, navi_web_browse, monkeypatch): + opened: list[str] = [] + + async def view_one(url, screenshot=False, wait_until="networkidle"): + opened.append(url) + return _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + await navi_web_browse.web_view(["https://a", " https://a ", "https://b"]) + + assert opened == ["https://a", "https://b"] + + +class TestFailures: + async def test_one_failed_page_does_not_erase_the_others( + self, navi_web_browse, monkeypatch + ): + async def view_one(url, screenshot=False, wait_until="networkidle"): + return _failed() if url == "https://b" else _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view(["https://a", "https://b", "https://c"]) + + assert result["success"] is True + assert "### https://b -> failed:" in result["output"] + assert [item["success"] for item in result["metadata"]["urls"]] == [ + True, False, True, + ] + + async def test_a_page_timing_out_does_not_erase_the_others( + self, navi_web_browse, monkeypatch + ): + async def view_one(url, screenshot=False, wait_until="networkidle"): + if url == "https://b": + await asyncio.sleep(10) + return _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + monkeypatch.setattr(navi_web_browse, "_BATCH_DEADLINE", 0.05) + + result = await navi_web_browse.web_view(["https://a", "https://b"]) + + assert result["success"] is True + assert "### https://b -> failed:" in result["output"] + + async def test_a_batch_with_no_pages_fails(self, navi_web_browse, monkeypatch): + async def view_one(url, screenshot=False, wait_until="networkidle"): + return _failed() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view(["https://a", "https://b"]) + + assert result["success"] is False + assert result["error"] == "all_urls_failed" + + async def test_no_url_is_an_error(self, navi_web_browse): + result = await navi_web_browse.web_view([" ", ""]) + + assert result["success"] is False + assert result["error"] == "empty_url" + + async def test_a_screenshot_batch_is_refused_up_front( + self, navi_web_browse, monkeypatch + ): + """`screenshot` returns image metadata for one page; with three there is + nowhere to put them, so say so instead of silently dropping two.""" + opened: list[str] = [] + + async def view_one(url, screenshot=False, wait_until="networkidle"): + opened.append(url) + return _page() + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view( + ["https://a", "https://b"], screenshot=True + ) + + assert result["success"] is False + assert result["error"] == "screenshot_batch" + assert opened == [] + + +class TestTextBudget: + async def test_the_batch_stops_at_its_shared_text_budget( + self, navi_web_browse, monkeypatch + ): + """Three pages at the single-page cap would be 60k chars of context; the + batch stops at 40k and says where it stopped.""" + big = "x" * (navi_web_browse._MAX_TEXT) + + async def view_one(url, screenshot=False, wait_until="networkidle"): + return _page(big) + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view(["https://a", "https://b", "https://c"]) + + assert len(result["output"]) < navi_web_browse._BATCH_TEXT + 2_000 + assert "40k-char" in result["output"] + assert result["metadata"]["urls"][-1]["truncated"] is True + + async def test_a_single_page_keeps_its_own_cap(self, navi_web_browse, monkeypatch): + async def view_one(url, screenshot=False, wait_until="networkidle"): + return _page("x" * navi_web_browse._MAX_TEXT) + + monkeypatch.setattr(navi_web_browse, "_view_one", view_one) + + result = await navi_web_browse.web_view("https://a") + + assert "40k-char" not in result["output"] + + +class TestToolLayer: + """A call that opened nothing must reach the journal as an error. + + ``view_tool`` used to return the JSON with ``"success": false`` and no + ``isError`` — which is how a server whose browser could not launch at all + looked like a working one, with a 1400-char payload I mistook for page text. + """ + + async def test_no_page_opened_raises(self, navi_web_server, monkeypatch): + async def failed(url, screenshot=False, wait_until="networkidle"): + return {"success": False, "output": "Browser error: no browser", + "error": "all_urls_failed", "metadata": {}} + + monkeypatch.setattr(navi_web_server, "web_view", failed) + + try: + await navi_web_server.view_tool(["https://a", "https://b"]) + except RuntimeError as exc: + assert "Browser error" in str(exc) + else: + raise AssertionError("a failed web_view did not raise") + + async def test_a_partial_batch_still_returns( + self, navi_web_server, monkeypatch + ): + async def partly(url, screenshot=False, wait_until="networkidle"): + return {"success": True, "output": "### https://a\nok", "error": None, + "metadata": {"urls": [{"url": "https://a", "success": True}]}} + + monkeypatch.setattr(navi_web_server, "web_view", partly) + + body = await navi_web_server.view_tool(["https://a", "https://b"]) + + assert '"success": true' in body diff --git a/uv.lock b/uv.lock index 9f028ee..444494e 100644 --- a/uv.lock +++ b/uv.lock @@ -386,6 +386,15 @@ ] [[package]] +name = "babel" +version = "2.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" }, +] + +[[package]] name = "certifi" version = "2026.7.22" source = { registry = "https://pypi.org/simple" } @@ -658,6 +667,20 @@ ] [[package]] +name = "courlan" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "babel" }, + { name = "tld" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bb/16/2a771612ee0b3acaa95ac21cc7e8a3319e815d6360f8ffc5987d1ce28499/courlan-1.4.0.tar.gz", hash = "sha256:fbbac7b7fcde2195ea08e707609503c81cf39c891e8d26cdb1fed4585782d63d", size = 208997, upload-time = "2026-06-01T17:30:17.306Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/38/ce65091ff20a16e06d17418c4353af5f56d3190821b1a06983c79ae79274/courlan-1.4.0-py3-none-any.whl", hash = "sha256:ad1dbdefd912ca7238d4607dc855df5df097f56bac175dd662c84eed3802f49e", size = 34193, upload-time = "2026-06-01T17:30:14.984Z" }, +] + +[[package]] name = "cryptography" version = "50.0.2" source = { registry = "https://pypi.org/simple" } @@ -727,6 +750,21 @@ ] [[package]] +name = "dateparser" +version = "1.4.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dateutil" }, + { name = "pytz" }, + { name = "regex" }, + { name = "tzlocal" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c7/5d/bd21ba1519b6b1e222b29878301d2e1fb928e890dc7d085fa4222ac5671b/dateparser-1.4.3.tar.gz", hash = "sha256:bab8c43a746266e68142f4926e69438ce551441aa88e54e78bb6410bf3ee7000", size = 362152, upload-time = "2026-09-03T10:07:54.545Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/30/5f/63f927b5ffd1cc2c030fc2bd362bf920890e8806cf42f701198de5b3f9c7/dateparser-1.4.3-py3-none-any.whl", hash = "sha256:cbce86c64e0cea5c54c84c015d34c903d46c51bfad50632bf2f05a6141d10c05", size = 322384, upload-time = "2026-09-03T10:07:52.913Z" }, +] + +[[package]] name = "ddgs" version = "9.16.0" source = { registry = "https://pypi.org/simple" } @@ -983,6 +1021,22 @@ ] [[package]] +name = "htmldate" +version = "1.11.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "charset-normalizer" }, + { name = "dateparser" }, + { name = "lxml" }, + { name = "python-dateutil" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/44/4d/3303caa82c75cd81e43d81906fa0f57c02c1e2e92ef9a3709cfe57fa678a/htmldate-1.11.0.tar.gz", hash = "sha256:e099a58143d5ff32471f2aa8770247e12f0f5bdff62f2df74f833277fd0071b6", size = 52776, upload-time = "2026-10-02T14:24:47.381Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/42/f8/6139fc95b889f44e9a4b373778f10315ade81b395a6298adf3c08415de64/htmldate-1.11.0-py3-none-any.whl", hash = "sha256:2a5a0d80cce38d3f986a64bb9ab64ba03e5953159b38fd78a841492131ead348", size = 33365, upload-time = "2026-10-02T14:24:45.42Z" }, +] + +[[package]] name = "http-ece" version = "1.2.1" source = { registry = "https://pypi.org/simple" } @@ -1269,6 +1323,18 @@ ] [[package]] +name = "justext" +version = "3.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml", extra = ["html-clean"] }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/f3/45890c1b314f0d04e19c1c83d534e611513150939a7cf039664d9ab1e649/justext-3.0.2.tar.gz", hash = "sha256:13496a450c44c4cd5b5a75a5efcd9996066d2a189794ea99a49949685a0beb05", size = 828521, upload-time = "2025-02-25T20:21:49.934Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f2/ac/52f4e86d1924a7fc05af3aeb34488570eccc39b4af90530dd6acecdf16b5/justext-3.0.2-py2.py3-none-any.whl", hash = "sha256:62b1c562b15c3c6265e121cc070874243a443bfd53060e869393f09d6b6cc9a7", size = 837940, upload-time = "2025-02-25T20:21:44.179Z" }, +] + +[[package]] name = "librt" version = "0.16.0" source = { registry = "https://pypi.org/simple" } @@ -1525,6 +1591,23 @@ { url = "https://files.pythonhosted.org/packages/03/5c/91fe48856f9f8089be3096fa4dbe4b3fb5526f3bf3e852ea9497f399cb9f/lxml-6.1.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bc8dd3d9c93e70c3df974a201ac2958b6d77b465d813c51d1f15fa8e645763ae", size = 3511258, upload-time = "2026-09-02T14:46:49.046Z" }, ] +[package.optional-dependencies] +html-clean = [ + { name = "lxml-html-clean" }, +] + +[[package]] +name = "lxml-html-clean" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/7e/2b7118b28d5bb69dfc4f349e1ca92334cf986601d3c809caa62cfa3bfab6/lxml_html_clean-0.4.6.tar.gz", hash = "sha256:ea9db067c3053854fd87f8d5a1cf01221dc893b54dbf6019c689052e05c64f22", size = 26385, upload-time = "2026-10-26T11:14:45.978Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f5/36/4898b5e3951635846ed0e445e042fadf02b7c5e811a9e43da5b3557bc8d2/lxml_html_clean-0.4.6-py3-none-any.whl", hash = "sha256:cc037270ba500a65e7235409dcddcae8ef37300f26dea826b61d917bf0236880", size = 14952, upload-time = "2026-10-26T11:14:44.73Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -1857,6 +1940,7 @@ { name = "pytest" }, { name = "pytest-asyncio" }, { name = "ruff" }, + { name = "trafilatura" }, ] [package.metadata] @@ -1892,6 +1976,7 @@ { name = "structlog", specifier = ">=24.1" }, { name = "tenacity", specifier = ">=8.3" }, { name = "textual", specifier = ">=0.70" }, + { name = "trafilatura", marker = "extra == 'dev'", specifier = ">=2.0" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.29" }, { name = "websockets", specifier = ">=12.0" }, ] @@ -2492,6 +2577,15 @@ ] [[package]] +name = "pytz" +version = "2026.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/b0/168030b0bfcc334cf09fa3313afdb888d892c8a0027324683d1de0ec277e/pytz-2026.6.tar.gz", hash = "sha256:3b0fc2727707663d6e517936f444fd8a8b240f1fc32f4bf7ee03274245945284", size = 318520, upload-time = "2026-10-29T09:05:47.011Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b6/25/564f3fc5b70c1398d706c51f2b1470d0ea0b1b2f96fee67877b6f5bdf3b6/pytz-2026.6-py2.py3-none-any.whl", hash = "sha256:7fed3fee8fd6bd4a2b7b75a8d56021d0624127bf69bd5691f115db929ac570b3", size = 506321, upload-time = "2026-10-29T09:05:45.14Z" }, +] + +[[package]] name = "pywebpush" version = "2.5.0" source = { registry = "https://pypi.org/simple" } @@ -2599,6 +2693,126 @@ ] [[package]] +name = "regex" +version = "2026.9.29" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/f2/af1da9d3ceed77bfcdce40427d49ba0be94e4fe84245e3bfef68c10e75b6/regex-2026.9.29.tar.gz", hash = "sha256:8b5fcc4771732191b2b7d1dd68d8f0353f47f8d90b6150f6dce58bf1112442cb", size = 419199, upload-time = "2026-09-29T00:49:58.298Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/6b/6dea87689c3a06a6e79d254bf824e6f3e3d724b5ba027c6112559aa6cd2c/regex-2026.9.29-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6abb75ab16bc3281714a5b99548a2225db70dba1f995f6d7f7419b76eb5a8fbe", size = 495413, upload-time = "2026-09-29T00:46:14.51Z" }, + { url = "https://files.pythonhosted.org/packages/3a/a5/0c791a0e83ad1013d262c13247c4c77e0f4a8d05bdc167df96aba6681c0d/regex-2026.9.29-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b7b893976e7fe42053da64f2aa27239c24252fd2ec6df471e1be197c0addc3b1", size = 295145, upload-time = "2026-09-29T00:46:16.292Z" }, + { url = "https://files.pythonhosted.org/packages/b1/07/9bf3607d8d13a12e436ab9d63f9791e10706827d535695b23964ad79fd79/regex-2026.9.29-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:066d0e3dbfdd739bce2bf8c2a41dd16f73e3d8adc2eb06dd803a36a307f56075", size = 292123, upload-time = "2026-09-29T00:46:17.646Z" }, + { url = "https://files.pythonhosted.org/packages/64/6b/32c2e6fc617e1d3f247e250fea31a9a35b1265bd32f585968aa13b9999b9/regex-2026.9.29-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7020ed44df30b3aa492c00ee3b52d0548c1f30c2c6c5bb13ae897680900d3413", size = 799537, upload-time = "2026-09-29T00:46:18.976Z" }, + { url = "https://files.pythonhosted.org/packages/bf/72/f041177f3c7a4606f7c81a95fe7eea03e2a0c4e8bff9e439a01432cbc9f2/regex-2026.9.29-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ae4613d7d9dda60fcba95f846cc6f808017f1843f392cf9daad14a6534493d71", size = 873255, upload-time = "2026-09-29T00:46:20.684Z" }, + { url = "https://files.pythonhosted.org/packages/d0/4e/a78948e11dd715e0e46716c2e0f3404b3fe6a44e2a2e9abdc7d965cab2b3/regex-2026.9.29-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bec37990e3d6121f29ecfb594bd8f1bf009e9f7926daba2e50e3b27d3892a783", size = 914570, upload-time = "2026-09-29T00:46:22.599Z" }, + { url = "https://files.pythonhosted.org/packages/8a/70/aa08d1d2b294894b365e5f8ba5380fe3f8546acdb81f10639dfd74209c37/regex-2026.9.29-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:612b709381c0355b70d89cdb51b7f670591ed5cbbc0e3b5337488019dc667b65", size = 804978, upload-time = "2026-09-29T00:46:23.981Z" }, + { url = "https://files.pythonhosted.org/packages/21/32/1b03534c4715aca3b564416d28d518083ed4dab3bc913267600d2256140d/regex-2026.9.29-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a760da040b47767b4b873adfb7c3b691e9ba2fc60f113f9d0b88f1a62f323e85", size = 779297, upload-time = "2026-09-29T00:46:25.318Z" }, + { url = "https://files.pythonhosted.org/packages/76/a7/378f6f558d9e4444af315a307c5953565a511d1e3666f1bb7bdc82012b6b/regex-2026.9.29-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:49ee178ca31c94621294bf9b8b676a92a2e6bba8af0529591753719e57edb621", size = 786961, upload-time = "2026-09-29T00:46:26.963Z" }, + { url = "https://files.pythonhosted.org/packages/59/13/79f0b1846f5f342f92ddbd4b27b18bcb86da96d902c1a0be26520bde98d7/regex-2026.9.29-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:5eeb8edc6110d9194a4d0d54610f64c37a31c605b5dbb7e407fc6ec7fa34a4a1", size = 863234, upload-time = "2026-09-29T00:46:28.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/19/05af70dec9f2eed6ba34e08d2dcc6a48e7ae5e307659d5fe4201a5d7bbee/regex-2026.9.29-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:ccb64d887a9db1cd76dbc0f92051a1a478a2a67e7f56c62d915cb881d7734704", size = 766487, upload-time = "2026-09-29T00:46:29.941Z" }, + { url = "https://files.pythonhosted.org/packages/01/e1/9c7486d4afe8fdd1fe0ad60139f8aa91427381f409af6a29b609d8fdcb3a/regex-2026.9.29-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:9e4482589065c8ecd761cff522dcd85f2d39e62f551e37e025d1c7d54772def3", size = 854792, upload-time = "2026-09-29T00:46:31.358Z" }, + { url = "https://files.pythonhosted.org/packages/26/c7/49d008ff5f741d9a9799d7315556f3a12b983ff0fcd2cdfb62904bedafbf/regex-2026.9.29-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d60030baaa7bfbb02d650c126cdcddcb6e33dbff14d819434c8fa2fdcaeeeba5", size = 793050, upload-time = "2026-09-29T00:46:32.775Z" }, + { url = "https://files.pythonhosted.org/packages/cb/a1/46ba549e65562ca04608b24179b8a7bb6f146ae0e7c6d7f5e70f3339c8ba/regex-2026.9.29-cp311-cp311-win32.whl", hash = "sha256:18ae8eed4526e35bdb754d61562b90bf5c00a67fdcf3cc1380dd59597486631b", size = 268940, upload-time = "2026-09-29T00:46:34.179Z" }, + { url = "https://files.pythonhosted.org/packages/4d/4a/aab232183c70fdcf77bcf0c51819da02ec522e393e6a0bf00bcf2142e21f/regex-2026.9.29-cp311-cp311-win_amd64.whl", hash = "sha256:1043aedf5917caa861bcb25a9c11460049656bdf0017a90a309fa8f255467725", size = 280641, upload-time = "2026-09-29T00:46:35.484Z" }, + { url = "https://files.pythonhosted.org/packages/33/b1/7c05954af0f51de376df2ba97f7f78a8b79334c7e5b3d2d9f2aead1f4d3d/regex-2026.9.29-cp311-cp311-win_arm64.whl", hash = "sha256:352cf115a810b357caa35193ab656ecf5ef41056855e82f292c99e8514f8d954", size = 279411, upload-time = "2026-09-29T00:46:37.193Z" }, + { url = "https://files.pythonhosted.org/packages/84/48/3fdcde9a0baa84d7d25571223265d6e434e114763b438601d54a8028bf3e/regex-2026.9.29-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:dc79d36d0618752265f0d575915bdc5c5130ecb9c9f6b3bcefeae32e4bdfafcf", size = 497903, upload-time = "2026-09-29T00:46:38.938Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1c/4ee3e97c76f53940488dfe7a7e18705e78daac8cd7fb161d246b9e328449/regex-2026.9.29-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3a21a9509d0ee88e7a70e1ad228cd2f0e0fd1e187458db132e8a8d18c97daf9d", size = 296416, upload-time = "2026-09-29T00:46:40.406Z" }, + { url = "https://files.pythonhosted.org/packages/37/14/f3f0ba083d2094392d5eabf56db5ea6ba469fd6e927afd187042054ea68a/regex-2026.9.29-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f57dc6b8fef170f105d2cf5cdce254f47b137d7755086cf7050f47e16582abba", size = 293633, upload-time = "2026-09-29T00:46:41.959Z" }, + { url = "https://files.pythonhosted.org/packages/c9/72/67e7a8ce17f1aea49df215564048efb49cc8c2b31a0e0fc30f36838f8516/regex-2026.9.29-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f93bc1c3486ef3747e07c9d7c1d0a147b8fbaab975f80e348aed6f71309dfaca", size = 805885, upload-time = "2026-09-29T00:46:43.373Z" }, + { url = "https://files.pythonhosted.org/packages/f6/78/25436bcfd4d2260b4b4090094d55d7ab53ec8a1ab4865a0b8bcb33c7d5c0/regex-2026.9.29-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e1d3a4cb7993b708f0ada8d0c84590efd853f169e7147d2202c9da503180242", size = 878344, upload-time = "2026-09-29T00:46:45.328Z" }, + { url = "https://files.pythonhosted.org/packages/97/e6/a09ec3a23ae41d6179880e67f0aace9284b2d95f2d7b326eff203f8eec5e/regex-2026.9.29-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:dabee8f4935e731fb46b2a3091bdda0d3d94b3bbfb907d2b4f12eefce4009619", size = 919181, upload-time = "2026-09-29T00:46:47.041Z" }, + { url = "https://files.pythonhosted.org/packages/26/83/d2fbd2e4e3afb1167daa825187d196f313cbaa1a4768f311fb041bb0e3d2/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:39ab5894d971f9ac68baa6eca5c50387db579cfcacf36ae8df3feceb1815e6d0", size = 807783, upload-time = "2026-09-29T00:46:48.894Z" }, + { url = "https://files.pythonhosted.org/packages/46/0b/eb429a7016610d44fc89a597163f8c9127505f0d7dc724dc9effbb6a3ac0/regex-2026.9.29-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c1a9a6651197fbed6f0212591418b9def774fc3f8324f78d1bf0e6a63e5f8aa1", size = 783465, upload-time = "2026-09-29T00:46:50.64Z" }, + { url = "https://files.pythonhosted.org/packages/1b/07/58a3c0153c7476898430f6a7cf3d9062a1d17fbea4f43399ecaf411c7b4c/regex-2026.9.29-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:87fb80cbe3557e27e7b28b995c2b2eedf689b8886f941ab93e0e288f0976518a", size = 793519, upload-time = "2026-09-29T00:46:52.396Z" }, + { url = "https://files.pythonhosted.org/packages/2a/e8/161b94d39164520e21a7befe0245569bf7fda4c7cf1fc4e2df2b5def49da/regex-2026.9.29-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:3c5c2ef13797466aa64170cbb66ad98a32351dd4127694cea7199f80f213750d", size = 869293, upload-time = "2026-09-29T00:46:54.128Z" }, + { url = "https://files.pythonhosted.org/packages/8f/07/3b02ed829aa2decdc1955d222bd1e2f99d1c8bb4873bbb9a66b2f0a36bff/regex-2026.9.29-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:59b49507f47479e299a9e1bc41b5cb83a7afda0540625f1dbae886615978acbf", size = 770239, upload-time = "2026-09-29T00:46:56.106Z" }, + { url = "https://files.pythonhosted.org/packages/42/5b/ba61f6fe062eb8562e742367d177bb75370434138ef6c9d2a27114f8d613/regex-2026.9.29-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:0dd8af32e9f7b56b7f95cc1fd79b23054c3bdc172392ae560acc24d57b7ffe71", size = 861973, upload-time = "2026-09-29T00:46:57.665Z" }, + { url = "https://files.pythonhosted.org/packages/cc/27/767259b20e8a842948990f5e99138d6c077248fd42f8b5468b1d9ca4b814/regex-2026.9.29-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:db5e82ba15c142425b8406690032df89e39cca4a2e8afbbb9a3d84edc2373ac3", size = 796470, upload-time = "2026-09-29T00:46:59.236Z" }, + { url = "https://files.pythonhosted.org/packages/a0/05/2566c4ba849b68a8ab81a6bf428fa79d20aae7ddee83979103c0381df254/regex-2026.9.29-cp312-cp312-win32.whl", hash = "sha256:d0c3082bf79bcd6a614d55916590ad4b8f93200e10b97f463ea5d9d07c9b5f23", size = 269327, upload-time = "2026-09-29T00:47:01.135Z" }, + { url = "https://files.pythonhosted.org/packages/93/19/489bc8db91196381c935752df01ba3f607140daece33b78d88573f028e64/regex-2026.9.29-cp312-cp312-win_amd64.whl", hash = "sha256:fdd88ed5e20b1bcdd234421e454962c971aa44b653bdb7f1ea9ef683e90fb649", size = 280334, upload-time = "2026-09-29T00:47:04.436Z" }, + { url = "https://files.pythonhosted.org/packages/0b/47/fb88ba779d0e5e7d4b0ec1aceeb13845948a2cb876bd572a2d1dfdba090b/regex-2026.9.29-cp312-cp312-win_arm64.whl", hash = "sha256:4fe97894d1b306c919b4e50def1e6f6c522f4d03a7283811f4d108f1ce5d3ac2", size = 279614, upload-time = "2026-09-29T00:47:06.541Z" }, + { url = "https://files.pythonhosted.org/packages/79/d5/6080f7d1a6e7e36aa720f806ac93c035ba39c209ae6cc510e8ef4c0279c6/regex-2026.9.29-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f1a0d5117230dd46b399a30a38afa44f79c99f3168988fdc4f425c3f928b39df", size = 497622, upload-time = "2026-09-29T00:47:08.251Z" }, + { url = "https://files.pythonhosted.org/packages/00/71/c87fc7a2e21a42f9d57489db32951c37eef56d153840459a80d464f0321d/regex-2026.9.29-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f0fe9834e5aeccaf19a0d8feb296d66a24be1a7c9922002f842a682cd5abb787", size = 296281, upload-time = "2026-09-29T00:47:09.764Z" }, + { url = "https://files.pythonhosted.org/packages/11/9e/aa0f4cde3bc4688c1d58b0cd8415edd708339bc0bc401a195b0b1e8c8f0c/regex-2026.9.29-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c90fcf7804ea0a54b896ce0f2b9565350220b8d4890fd0db461a476a4c687963", size = 293472, upload-time = "2026-09-29T00:47:11.723Z" }, + { url = "https://files.pythonhosted.org/packages/90/d4/e835c487850ed922a8d6074f953b888c8ea99775c76b9ed5f8a4d72eab92/regex-2026.9.29-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e11edba5bc344a32b029a7af9d4b3173982dd79eeafa0b9dbd787364414b0509", size = 805966, upload-time = "2026-09-29T00:47:13.235Z" }, + { url = "https://files.pythonhosted.org/packages/2c/57/ba8809847fbae8d2cbc71367c6ded510a7ec88bf52493c65efc1acf4effb/regex-2026.9.29-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bb90e7177944b6684738c1fc36aabd2dd00d1de3be7dbe09f91e196f1bc0dc81", size = 878346, upload-time = "2026-09-29T00:47:14.877Z" }, + { url = "https://files.pythonhosted.org/packages/1a/52/e3da19fc3cc15ef67ab67e121e87887c3bccfdb683a7a9ec557c460ca5b7/regex-2026.9.29-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d06fcdecc10fc7954d7c8f27a03c96055fe525274dc84a7b0dbdc3d6b9e03dab", size = 919250, upload-time = "2026-09-29T00:47:16.622Z" }, + { url = "https://files.pythonhosted.org/packages/9a/8e/c1ed81f55f992f6aa0b699a592a50c1ce9e6d44ff1aee2c14c0537dcef9c/regex-2026.9.29-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d49c18f1ea294cf4adde2e5ac256e98c82ea9d708462ce4bf799dffa7cfe8a2c", size = 807902, upload-time = "2026-09-29T00:47:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/ad/bc/5a6886eb470e41040e21e05b75024a18b6ebfe7ea400b72094a60f949101/regex-2026.9.29-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3e778bfccd63075167709136afbc251c1f683758d5bf49c803c60ac3f894ce6b", size = 783514, upload-time = "2026-09-29T00:47:19.921Z" }, + { url = "https://files.pythonhosted.org/packages/cb/52/6d951d453b023c6edb880f1ba474291b53b8ce1cc438b96a9db6d791d991/regex-2026.9.29-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:686ac5350fceae63830bb98805fcb8039325bf4c06d9f6f048ff65229d5bffa5", size = 793466, upload-time = "2026-09-29T00:47:21.552Z" }, + { url = "https://files.pythonhosted.org/packages/99/b9/d5a41adc08360f5eee0dc4846c578f002366947211fc8af5a69a64ee7b9f/regex-2026.9.29-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:26ec4ccce55aa533fbd603d08911b01101a8fcfec987845ac3ae2c7087b2bde3", size = 869464, upload-time = "2026-09-29T00:47:23.276Z" }, + { url = "https://files.pythonhosted.org/packages/4b/32/d76c9d91f5d798e2e9e67f6f85ec4ae35445ac425f7454797311cecb80ca/regex-2026.9.29-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:a655d34b2a6943af32401f3d94f72e9d731f6ad16285815550bf2b4ee69d420a", size = 770278, upload-time = "2026-09-29T00:47:25.193Z" }, + { url = "https://files.pythonhosted.org/packages/24/00/aeebdb540c620a0f7317f6d6fad80a47729ecf0599a24b5c34ec155351f5/regex-2026.9.29-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:0c992c19cd45058a4b92f68f139c93db168b48fb1f322c9a7cd620806afb6b51", size = 861949, upload-time = "2026-09-29T00:47:27.005Z" }, + { url = "https://files.pythonhosted.org/packages/12/62/d0314bcedfd3586197e4596931fa220260eb2385bf53184e5b9ae67db24b/regex-2026.9.29-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ebb8912f565b8cdbbf27debfe00df04202c20e2f651b9e32767930c5eace3621", size = 796602, upload-time = "2026-09-29T00:47:29.233Z" }, + { url = "https://files.pythonhosted.org/packages/ae/c7/d5a8c13a613facb03e0fb55c1ebaaf7bb35d8e2c1abe8bef8dca809fc1d9/regex-2026.9.29-cp313-cp313-win32.whl", hash = "sha256:4d7d93613b01b0199961330e49cfc52d479b3d5776c56c691db31130c0a07d91", size = 269306, upload-time = "2026-09-29T00:47:31.14Z" }, + { url = "https://files.pythonhosted.org/packages/80/a7/bf93a3a6afa5f7bc16b7afb94ae581b01cae620b8ad56bd8f9572a985959/regex-2026.9.29-cp313-cp313-win_amd64.whl", hash = "sha256:61956f074ecd123f55adca68ee3eab46e6a07ad3f8e64e6db95dfacb444f55c4", size = 280307, upload-time = "2026-09-29T00:47:32.709Z" }, + { url = "https://files.pythonhosted.org/packages/b2/7d/388274e53605a86297f433a08102a7bbdcf9379d47683d307ccaefd88e2c/regex-2026.9.29-cp313-cp313-win_arm64.whl", hash = "sha256:bfc71e6d970419c1309b3640305298643e2a734cad3f7cfb6d2ddee4175ab53d", size = 279599, upload-time = "2026-09-29T00:47:34.674Z" }, + { url = "https://files.pythonhosted.org/packages/93/1f/d9dc6f02f569625faf67a4daec926cd5023472dcd69bb44286dccd5a5ab3/regex-2026.9.29-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:957bb708e8057ab1649ba566456429d691ec9b90d1c9ad1af1ba7ffbbeaf05f2", size = 497598, upload-time = "2026-09-29T00:47:36.541Z" }, + { url = "https://files.pythonhosted.org/packages/9c/83/9b693a3fd1451381e812031a8961ec5b3b8f0c8cc6871f14c5223642804d/regex-2026.9.29-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c9b602fae1e00b7c035d661ce85575365719192a7b46784bd71cf64c68053aa0", size = 296250, upload-time = "2026-09-29T00:47:38.233Z" }, + { url = "https://files.pythonhosted.org/packages/dd/5f/52bc2abc3fef040cd9de76ab29c918d6a717a454ae2b9dd7938b0c95656d/regex-2026.9.29-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0166844493626c5015c6088ee15c9ca2fd060ca15b7641d1657da6a58432ae33", size = 293518, upload-time = "2026-09-29T00:47:39.957Z" }, + { url = "https://files.pythonhosted.org/packages/dc/fc/cf50671215ee0057046980b4571ef8646a005819bb67f0957e779ed107a5/regex-2026.9.29-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b97a38fb4c732b6832db6bf108963adbcd82ef1268ba2025dce390f45af75efa", size = 806037, upload-time = "2026-09-29T00:47:41.676Z" }, + { url = "https://files.pythonhosted.org/packages/14/4b/dddef8fc15c63e4347cc9efb138d0cd306f30e6c98acbcc81a8f780083b9/regex-2026.9.29-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a540abfab208e1b7ef2df231c40ef3b6cbb30a0aad6204e9b6a81c10a6794628", size = 878881, upload-time = "2026-09-29T00:47:43.755Z" }, + { url = "https://files.pythonhosted.org/packages/9f/cb/38daabed32d28f7e58a06e9344ce00dc67952e9996bc578ed6a29fe1240e/regex-2026.9.29-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ddfa987262763c3c22a8367d2a49c244b018a74c3a8e3ab1a864119ad45c5633", size = 918684, upload-time = "2026-09-29T00:47:45.594Z" }, + { url = "https://files.pythonhosted.org/packages/a9/4d/041d9458a645fee4fce4d642a89d27271a3cfcd91095104f6dde44da70bf/regex-2026.9.29-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2f7f7aa47b229f2b39a2ae2596d2ad5625d77b5eb9856fac2dab3eb506cdd0a0", size = 807176, upload-time = "2026-09-29T00:47:47.372Z" }, + { url = "https://files.pythonhosted.org/packages/bf/c4/4383eed7aa5aef67616cb1b3f3ad06b7c624c4e6cced48630cd5ce133d85/regex-2026.9.29-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d9b77b25b4f395f92de6099ab08e8ae2bc7e51dfe157f22900902243a5cc90c7", size = 784315, upload-time = "2026-09-29T00:47:49.518Z" }, + { url = "https://files.pythonhosted.org/packages/5c/a6/0086ad31cebb183c637d3198547075aa493afde308e1ff61fccccb29ba6e/regex-2026.9.29-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:34b6925af9853bf461950e6508910f179fd6e9b1a7ec8548e069606b7e51a26b", size = 793748, upload-time = "2026-09-29T00:47:51.279Z" }, + { url = "https://files.pythonhosted.org/packages/d5/a0/f9005cba3f629a859573fc5d1224ea4e1f97919ec8581d018e03a351a604/regex-2026.9.29-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:addd736a0547d553283adaf4e05d7104e7f2c7b0b092e9b4d28756825f14531f", size = 870302, upload-time = "2026-09-29T00:47:53.368Z" }, + { url = "https://files.pythonhosted.org/packages/01/4f/e1a3e46bb5315a4e18b01a990e7a28e2a16595609d50c442baf2815a3c65/regex-2026.9.29-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:fe3fa1dd453ed5c7f5ea23a26218329790ed7197a99b90e94330e313959a7f52", size = 770299, upload-time = "2026-09-29T00:47:55.606Z" }, + { url = "https://files.pythonhosted.org/packages/2c/fe/f303b4acfda44e1ff1379368748c1ef2dad04a6a8e9c0ecbc970b19d97ca/regex-2026.9.29-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:0cc63b5e47c12a48d90c7e9d7de6a035dd14f62868aaedbb4e0ff8ba2b8bfe7b", size = 861570, upload-time = "2026-09-29T00:47:57.617Z" }, + { url = "https://files.pythonhosted.org/packages/60/b6/b4f7e99249f596017c60ccad5faf9310fc8e3e59bb2244940a90a1b0bdff/regex-2026.9.29-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:724184b4aafed865e4f13ca313fdcb43024300c028ec67319cfa16847d84685e", size = 795967, upload-time = "2026-09-29T00:47:59.922Z" }, + { url = "https://files.pythonhosted.org/packages/fb/d3/fc865a4638d9f6762192b6bab5b7aa1f33a90e9e99578c2e111e2a63c8c3/regex-2026.9.29-cp314-cp314-win32.whl", hash = "sha256:c6c8fabf1dafc1f1ddcbb67896d3f93efb092e8c4b6322d7389b944e76a484e5", size = 274758, upload-time = "2026-09-29T00:48:01.8Z" }, + { url = "https://files.pythonhosted.org/packages/31/e2/c2b466924ccbeb874862968ca638051b15a8fd29d994a0e99004a5cbf78e/regex-2026.9.29-cp314-cp314-win_amd64.whl", hash = "sha256:1c2a0026062abcc321a53db4a185ceba0b59a66b5d37b0808917a88b55a5257f", size = 283817, upload-time = "2026-09-29T00:48:03.614Z" }, + { url = "https://files.pythonhosted.org/packages/c6/42/ea0f8dbaa924fa75c6338935eaee2f44dab369b27f02db1e03d74344b049/regex-2026.9.29-cp314-cp314-win_arm64.whl", hash = "sha256:121a76a0985db80ceae9e171c337f8c927868e37d01b54e3ce87bc87f9c6a208", size = 283663, upload-time = "2026-09-29T00:48:05.624Z" }, + { url = "https://files.pythonhosted.org/packages/44/48/d58e5081119f5c223bbb37d2340acde3d069e1df8e8cd166c37502eee4da/regex-2026.9.29-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:e31f72490b7c12f7790e1e25c3afffd20503ee1bfb43461d7838b871ff244b19", size = 501393, upload-time = "2026-09-29T00:48:07.833Z" }, + { url = "https://files.pythonhosted.org/packages/72/3c/c49945287d4f9efee7d41f98072f8ad880efb8f430595a612fbdea996a4e/regex-2026.9.29-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:80ea96f5c1a30bf09007d48466521d9c294bebe197c708c3359096e3e3691632", size = 298237, upload-time = "2026-09-29T00:48:09.684Z" }, + { url = "https://files.pythonhosted.org/packages/f9/1f/688cb61c3d4cf7bcc1ed444b5cc49399eba3e51c469ae285cf87fea3022e/regex-2026.9.29-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:554bffadcbcb6d5f4e5fb10a61cc52084b9a63d1dab5f10bcd2c4343972e8e2c", size = 295936, upload-time = "2026-09-29T00:48:11.454Z" }, + { url = "https://files.pythonhosted.org/packages/26/a3/de43ac6b877b7d09c19a3a426b1bd5acdd209eaaf68f406466f80439ccf6/regex-2026.9.29-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:864e9b87ac33c3fb9fb4ad48166d4fdb579c351d5c77deb0d34bccb36a775cd9", size = 816905, upload-time = "2026-09-29T00:48:13.321Z" }, + { url = "https://files.pythonhosted.org/packages/62/14/9940763201c51d537786304984c67d0fc3d2ed18837ffb6f09a869f6b6c9/regex-2026.9.29-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:044265d77d94f5e3cb2fd72c76723807c429cb8c533e9d4672d0334a6f14f588", size = 881527, upload-time = "2026-09-29T00:48:15.313Z" }, + { url = "https://files.pythonhosted.org/packages/d3/e1/c842d8df0b23245ebf202f8ab9c39fd48e2db39959454ec39a41c8c72082/regex-2026.9.29-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2089fe39c406784d90101c726755ffa1497bb74638fd434300d2b88006186de8", size = 923115, upload-time = "2026-09-29T00:48:17.328Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c1/98622479e3c354a446a75232e522d747d2b3df23092dcd8a5309380a2020/regex-2026.9.29-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0def9fb6abac55492d6d51cddb7225d07d6f279e774e0adc08569a54a5fc8d46", size = 820674, upload-time = "2026-09-29T00:48:19.32Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d0/5808c95f9c79ed27b5eedaafc3df6239ec56a49f2e23ea8f831b18427c82/regex-2026.9.29-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:888d60953908dcf761aa320c3e390ab8556efbdb551ace63921de90f6ae0848d", size = 793306, upload-time = "2026-09-29T00:48:21.615Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d3/021ca2638671ad20603bcd9b4d5bfa35d2610cd216a043ea7f0b44ea39f6/regex-2026.9.29-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ed511a0708e2297e1d6431e7fb217e3402791e491e02da800658ace4973df1bb", size = 803103, upload-time = "2026-09-29T00:48:23.871Z" }, + { url = "https://files.pythonhosted.org/packages/6b/2d/755c6d13ef9c657378013676c391c7a402166b3f419a464a3e058dcbe533/regex-2026.9.29-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:e1172147d28d8fbcf8cb8d26c41506169f5ad8fe9ec969cb116835a19d4d8eca", size = 872175, upload-time = "2026-09-29T00:48:26.255Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fc/e1cab183b9dafe8597f58c1c766da9bf96204d3b2f232bcf3eeb75ff7b6c/regex-2026.9.29-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:92f05c9c42bde5785dc48770bc2194d9f7442544156f951e19cd31b096cec562", size = 777362, upload-time = "2026-09-29T00:48:28.389Z" }, + { url = "https://files.pythonhosted.org/packages/06/7c/e10ea17fba31fb4a1f9d13ed53a2d2a9066a2aea58d7557e263f6d99e7b0/regex-2026.9.29-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:f37964e4a5e993d2fd45147741e9dff7f34a2d8c00ab94c4ea0514a4677f959e", size = 865606, upload-time = "2026-09-29T00:48:30.4Z" }, + { url = "https://files.pythonhosted.org/packages/8e/6e/69824d9aee1fd41c54ea7264654a47c8d9d84d8a228e11c2bcf4c201ed81/regex-2026.9.29-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:951733b1bbdb71e377cec567b409f1a7881b47cfcad84121aa74cb575fa425ea", size = 807945, upload-time = "2026-09-29T00:48:32.375Z" }, + { url = "https://files.pythonhosted.org/packages/89/22/857050a86e21ce60193e02a8ef662521f2e263a645c8b1b905fc136b61a7/regex-2026.9.29-cp314-cp314t-win32.whl", hash = "sha256:65b408d8fcb273e3499e7ef2ce796810da1becd208c7fb4373692a242d79d461", size = 276758, upload-time = "2026-09-29T00:48:34.72Z" }, + { url = "https://files.pythonhosted.org/packages/4d/96/56808fe029553d7d4c703414f2a527faad2ea2bfa9ca094a2e7f8762b530/regex-2026.9.29-cp314-cp314t-win_amd64.whl", hash = "sha256:bf48516e35cf848390ea68850aba53e7c333720d2945b4d2c25b69fc5171723f", size = 286527, upload-time = "2026-09-29T00:48:36.864Z" }, + { url = "https://files.pythonhosted.org/packages/01/aa/074e2cfb3d8101a6a764aba5f7c5d1e21de087483e35bdc0c4ce2eb60364/regex-2026.9.29-cp314-cp314t-win_arm64.whl", hash = "sha256:9173db3be74a35cb6731701094b98120f7ee4876a287882a59cdea1fa7da342f", size = 285954, upload-time = "2026-09-29T00:48:38.901Z" }, + { url = "https://files.pythonhosted.org/packages/a7/dc/d84990386c9dfdf8c377f00f371b241fdc9a2c8aea0e3d66941b2e51be0b/regex-2026.9.29-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:c3589f40749acce747510bf5d589d54e376cb0930ea58b35effac97e5312b0c1", size = 497836, upload-time = "2026-09-29T00:48:40.858Z" }, + { url = "https://files.pythonhosted.org/packages/c2/ab/a569ebde875fa12ff8c6c9a30e07503620f195e4be4d54c3d3ee8eecc283/regex-2026.9.29-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:32ab11df9677ca80bcbb5fe4eb1da9109a5019239a054836efc6fa1c64e683cf", size = 296244, upload-time = "2026-09-29T00:48:42.952Z" }, + { url = "https://files.pythonhosted.org/packages/f3/3e/7d548e82a108e7c8b2d5246650e397a2f8db599f9b2e975466939c5b4e70/regex-2026.9.29-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:7c03031610e3e6ed1768a2b7a8fc84637c1257b50c5eacaf094c6e17a84fc563", size = 293748, upload-time = "2026-09-29T00:48:44.985Z" }, + { url = "https://files.pythonhosted.org/packages/40/34/a8e19a52f452bbb07b32a2bef70dcdf90c2737049749f74cc12d7486fb4f/regex-2026.9.29-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:42e82e578c904445d4c8a35b8f28052cf567593215fa5db06266fbc6f77aaa2e", size = 807840, upload-time = "2026-09-29T00:48:46.948Z" }, + { url = "https://files.pythonhosted.org/packages/88/7b/11fbd4640b3bb82b72822a63c20ade4013d562d291703a9debeedc24e682/regex-2026.9.29-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0b65c72739f981377c9c22e0c5c3cd7f42da7bd8a3c9209330fac772c7d893ed", size = 879330, upload-time = "2026-09-29T00:48:49.168Z" }, + { url = "https://files.pythonhosted.org/packages/f3/55/de58c74f1f4e31586d83eb39c56872d686c4e0d0966d151884c833b94ced/regex-2026.9.29-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4408b2b27a95ca8cc48b7411945753773353b5c93b307754781086c99d3a576f", size = 919251, upload-time = "2026-09-29T00:48:51.322Z" }, + { url = "https://files.pythonhosted.org/packages/81/42/a8c480f6dd5ac59fa28ddae79afd9d7ac7e596fdb61813adc65bb6e674b8/regex-2026.9.29-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a714befaacbd10092ffe4cea0d3c5f008fb9efe9bc322c715bcdfdee414b9a3d", size = 808808, upload-time = "2026-09-29T00:48:53.529Z" }, + { url = "https://files.pythonhosted.org/packages/68/60/0bc0d1ec8b37ad64be6fa30e035251f11de9667a0fac9e82ee74517d81be/regex-2026.9.29-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:33026515aebc0e70d1c89978e53e8d695d35d9e472f8d5b34465ba3c74028650", size = 789907, upload-time = "2026-09-29T00:48:56.036Z" }, + { url = "https://files.pythonhosted.org/packages/da/84/116a3ef19b3acfe81077f0bf2cbc7714a5e94bc8935b7243ab61cb0f1c3c/regex-2026.9.29-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:31b003f9a070335e2a8233ee9b14a3ca8e6d792012ae011f741bf0aaf11744c5", size = 795770, upload-time = "2026-09-29T00:48:58.284Z" }, + { url = "https://files.pythonhosted.org/packages/96/ba/e38c3f203e7e7e18c957d48e6cb6dbf96c11e95a44efa4a480522afc5d6d/regex-2026.9.29-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:c03c6eb6ece86dfdcbb34799efaa339b093132e1aceed491ba5e08fe06cdf699", size = 870671, upload-time = "2026-09-29T00:49:00.506Z" }, + { url = "https://files.pythonhosted.org/packages/2f/0f/9ee0b0cb76c55f63684bd7fff554978e8773b4fc86e2bcb2d50772dc1086/regex-2026.9.29-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:a5300757f8a68f5b6cc33f57338d72a0e3589c5cc9ad5f8504ea06f028be582a", size = 778631, upload-time = "2026-09-29T00:49:02.984Z" }, + { url = "https://files.pythonhosted.org/packages/b6/19/e6e3eeb226af5872c4958002f6edef4e4f40ea4cc5f5665023f2019eb045/regex-2026.9.29-cp315-cp315-musllinux_1_2_s390x.whl", hash = "sha256:80c7cadd3fd2bfde5df8aa0787e315812cad0c313a753095d02f4c2b6c01677b", size = 862187, upload-time = "2026-09-29T00:49:05.264Z" }, + { url = "https://files.pythonhosted.org/packages/5b/62/823c102e106bb2711d6b7dfe5981552fe4467b2969c46a20c5c383cf498c/regex-2026.9.29-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:3f1e6cb402a89457582cd696f982559217d13484a193202c394015297968c86d", size = 798423, upload-time = "2026-09-29T00:49:07.644Z" }, + { url = "https://files.pythonhosted.org/packages/37/e0/e927776258fa70b2f6feffc3be584ffc85ba4c1e20a320f0aee9a632fc7d/regex-2026.9.29-cp315-cp315-win32.whl", hash = "sha256:a64b85a4760337cfefdb27d42da6ed8b58e8cde3f2d57b6ef43e76ef6ea9ef47", size = 274757, upload-time = "2026-09-29T00:49:10.513Z" }, + { url = "https://files.pythonhosted.org/packages/77/04/358de85d1860238e1b4fa98fc2c80c990124a25d2e14739e28cc02c25562/regex-2026.9.29-cp315-cp315-win_amd64.whl", hash = "sha256:b3e445b66c80b4eb4234e855ce94d9adc183eedbd632816228d89930b91b2c5b", size = 283824, upload-time = "2026-09-29T00:49:12.849Z" }, + { url = "https://files.pythonhosted.org/packages/92/d3/d5c5b264784a5ab2b0f8cf620c1eeb4dbf3440d306761905e7d99345bef5/regex-2026.9.29-cp315-cp315-win_arm64.whl", hash = "sha256:8f39588af4731c8923c26810eb3b33f76f17633985e40f59c3cd45a33805a895", size = 283664, upload-time = "2026-09-29T00:49:15.331Z" }, + { url = "https://files.pythonhosted.org/packages/02/dc/f63ec2c201445ce1150fe780f5c56f16a10124d9a9da3a93161dbb0d8892/regex-2026.9.29-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:fb99cc9d45f48895d9d67f6a0b8a57f08d39c174d9f25ad97a313e0470267b1c", size = 501549, upload-time = "2026-09-29T00:49:17.705Z" }, + { url = "https://files.pythonhosted.org/packages/9a/5c/d2a698dc6bfc11fbce03f1cb0249c13284e93b79ed11f893edf6fac431c9/regex-2026.9.29-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:720537c7ea6f80dc61913184edb0ce2497a306b39ef19f28505b322553d52bdb", size = 298187, upload-time = "2026-09-29T00:49:20.171Z" }, + { url = "https://files.pythonhosted.org/packages/85/b7/88dcdb38cd3935d4ee9e9ce9b8e56cb3b3518d1f020acfa7dd62ad289bf8/regex-2026.9.29-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:0fd2c901cc307a745ad4bc87f20060d7a0825a3371d1e93488af22e7a387f78f", size = 296157, upload-time = "2026-09-29T00:49:22.342Z" }, + { url = "https://files.pythonhosted.org/packages/d3/8e/ba6c01dde33a69fc294b38b43f6677baaa5735a6248f39708031a738158a/regex-2026.9.29-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b11b589e00095ec69cf79841a76360f9b079e95b0368a25b5ebb951ab0c157ff", size = 818468, upload-time = "2026-09-29T00:49:24.612Z" }, + { url = "https://files.pythonhosted.org/packages/2a/f1/2586693e3a2d6b1247852593d37a6c17b42a92ee44f7cdcb9a0c1494e64a/regex-2026.9.29-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d7cab119d0df0b9413f106b4d7fc34f2872d3574ed3806fb48959c830b1537da", size = 882825, upload-time = "2026-09-29T00:49:26.996Z" }, + { url = "https://files.pythonhosted.org/packages/30/51/084f3e7bdcd0e9c33665c938cf5d134dc3548cbb4a75f0197ec7bfd754b1/regex-2026.9.29-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b89efc38431793d28b7cd91227e2f952ad7c48df19132b17f43a5fec3c14143b", size = 923314, upload-time = "2026-09-29T00:49:29.822Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f1/066c6fc23b7dc229789c21c880b5ba5ad689fb95fed12e078266f55a1f9b/regex-2026.9.29-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80a5ea3b4fd9d6a5b9a44f7976a9acaaab35aa3c1f6b29e5bd857dfabaded223", size = 822222, upload-time = "2026-09-29T00:49:32.404Z" }, + { url = "https://files.pythonhosted.org/packages/0a/56/592cd46fdb8f2f8682a1d7fd1310e4d0bcb93fbd0e6bbe4141ac28240227/regex-2026.9.29-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:19959129885356df0e97556856f77eb2888380dac18bed075a7c05c5128c618d", size = 794882, upload-time = "2026-09-29T00:49:35.076Z" }, + { url = "https://files.pythonhosted.org/packages/ee/4d/d65384bb071c864b01aa8314e3a6a687845ebd57588390976edc960c218b/regex-2026.9.29-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:6a1a824fbed817e0a891103886b68f063b1e83cc51bc97192a90a60195a9291f", size = 805887, upload-time = "2026-09-29T00:49:37.395Z" }, + { url = "https://files.pythonhosted.org/packages/65/b6/358de0d8f40d5178e4f7e7e121cfd5b961c812b77a055d11f5079e3f8fd7/regex-2026.9.29-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:1ba8c6a416569ce0d37e83e28a254a61dc99a419084dfb6476cea02d997f74fa", size = 872901, upload-time = "2026-09-29T00:49:39.927Z" }, + { url = "https://files.pythonhosted.org/packages/00/06/6bfded72d043240c6b52bbb5e16f639d81affbf7484b4fe2ec45f3d4afc9/regex-2026.9.29-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:446654b29bfaa30500d80947eda42cef1449dc8a87f4e3cf061cc8485d3a1f0b", size = 782970, upload-time = "2026-09-29T00:49:42.581Z" }, + { url = "https://files.pythonhosted.org/packages/5a/20/9f418a50baa78b3ed8308fcb0cc49e472dd000b7ef935a7295af202ea744/regex-2026.9.29-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:bf3c49863c23a1ad6da9c30351aed6cff8d5ddbeb63c5c8420ae54e98c7d0138", size = 865441, upload-time = "2026-09-29T00:49:45.238Z" }, + { url = "https://files.pythonhosted.org/packages/2c/29/817c7eacdeaf8463123e949bd394c39ad024eea1ec38ddf5ad141da2f3bd/regex-2026.9.29-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:01000ddf0e3ffef97f2413ceb514f6313040106b6d18a03ee00a4fe35c1eb1db", size = 809431, upload-time = "2026-09-29T00:49:47.878Z" }, + { url = "https://files.pythonhosted.org/packages/63/0b/83aab3b5b739947f744135a7a3a446e25433ebc92b05e01aae197ccbfdda/regex-2026.9.29-cp315-cp315t-win32.whl", hash = "sha256:c4e38dd8f39c43a91d2410ad2b85610701b0979342c3df1d69eaf8e838c757d8", size = 276964, upload-time = "2026-09-29T00:49:50.524Z" }, + { url = "https://files.pythonhosted.org/packages/72/f2/6314b5fc68789b5dcc38885bc6e3d6986b34fb3372b7231088ee5cecaa05/regex-2026.9.29-cp315-cp315t-win_amd64.whl", hash = "sha256:e2c89e9b762c57f59d5e99ee8b20202adb892e35f8d3485741340999ca55058e", size = 286487, upload-time = "2026-09-29T00:49:53.224Z" }, + { url = "https://files.pythonhosted.org/packages/56/bc/97b2245c8c7b2dd01f2db74f2bea003cd33c15009b4996a2447f46b5325c/regex-2026.9.29-cp315-cp315t-win_arm64.whl", hash = "sha256:e8c65ef3862a8ad6e86492b6ed9327805dd66904c012bd3649dc67d822ed6c34", size = 285955, upload-time = "2026-09-29T00:49:55.655Z" }, +] + +[[package]] name = "requests" version = "2.34.2" source = { registry = "https://pypi.org/simple" } @@ -2866,6 +3080,33 @@ ] [[package]] +name = "tld" +version = "0.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5c/5d/76b4383ac4e5b5e254e50c09807b3e13820bed6d6c11cd540264988d6802/tld-0.13.2.tar.gz", hash = "sha256:d983fa92b9d717400742fca844e29d5e18271079c7bcfabf66d01b39b4a14345", size = 467175, upload-time = "2026-03-06T23:50:34.498Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/90/39a85a4b63c84213e78b3c17d22e1bf45328acf8ebb33ef93be30d0a3911/tld-0.13.2-py2.py3-none-any.whl", hash = "sha256:9b8fdbdb880e7ba65b216a4937f2c94c49a7226723783d5838fc958ac76f4e0c", size = 296743, upload-time = "2026-03-06T23:50:32.465Z" }, +] + +[[package]] +name = "trafilatura" +version = "2.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "courlan" }, + { name = "htmldate" }, + { name = "justext" }, + { name = "lxml" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c3/e7/d359fcf887675c4305fb1f4548372484a2d8f0bc793dc6f2da3a748c1253/trafilatura-2.3.1.tar.gz", hash = "sha256:2e5557aeab32d33b9da7ff0debbb677a676c5b648a927e50d91b8277011e1105", size = 350794, upload-time = "2026-10-06T19:30:26.202Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f5/20/e95d135c417aceaa3d34d1b4dff9aceb85e98800480eca1b033933717169/trafilatura-2.3.1-py3-none-any.whl", hash = "sha256:f86bad2ee36f82884e14dee84c83aae0f0ac0127da5c65764e82f3846716b822", size = 158660, upload-time = "2026-10-06T19:30:24.385Z" }, +] + +[[package]] name = "truststore" version = "0.10.4" source = { registry = "https://pypi.org/simple" } @@ -2896,6 +3137,27 @@ ] [[package]] +name = "tzdata" +version = "2026.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/29/6a1391d31cd00ff9e927215a224a880e1e84e826b64ba80a499e51c074d8/tzdata-2026.6.tar.gz", hash = "sha256:85d2cd9c09f81f0bb29db27759fc83af4f1e620efb814f8c5bf8b4cc4feece6d", size = 201017, upload-time = "2026-10-28T09:57:09.137Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0a/66/9a06c2227fedfd4919f0eaaa8b94377d9730882a7e4f95823c9a331ab5e1/tzdata-2026.6-py2.py3-none-any.whl", hash = "sha256:0a10eec6e13ef16b627fabed9a5f2ba61a62b1c67f327305f4e7e412a70b4aec", size = 348210, upload-time = "2026-10-28T09:57:07.276Z" }, +] + +[[package]] +name = "tzlocal" +version = "5.4.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d8/89/2bd035df331499589fc92e59375c17c6cea6a500453b2c5f6a6ec9572825/tzlocal-5.4.5.tar.gz", hash = "sha256:ba976898bad27945178c9ca25c222c19e322b30c10342b9f40b46714f70d2d5f", size = 31483, upload-time = "2026-10-22T09:09:26.38Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/c1/9be4e1a8a30db8729a86628073910e56f85aaf8a659e053f376a2e6ea5a8/tzlocal-5.4.5-py3-none-any.whl", hash = "sha256:ecab534f6d65d9deb4c4ea52c121ab6dd5958e9b878e71dcb102578269752454", size = 18215, upload-time = "2026-10-22T09:09:24.447Z" }, +] + +[[package]] name = "urllib3" version = "2.8.0" source = { registry = "https://pypi.org/simple" }