navi-web: search backends, three-at-a-time batches, and pages read as markdown
...
web_search runs SearXNG first, DuckDuckGo with one retry after a real failure,
and Brave last; measured 0.2-1.4s through SearXNG against 1.7-11.8s on
scraping. Both tools take one item or up to three and run them concurrently —
three pages went from 17.3s to 1.9s, three searches from 3.5s to 0.6s — each
under its own deadline, so one hung backend or page is named in the output
instead of erasing what the others already returned. Text is capped at 20k for
one page and 40k for a batch. A call that opened nothing now raises: a green
journal line carrying "success": false is how a server whose browser could not
launch at all looked like a working one, and how 604 search calls in two weeks
reported no errors.
web_view returns the main content as markdown, not innerText after a tag
kill-list. The old path could not tell navigation from prose (462 of the asyncio
page's 1018 lines were shorter than 40 characters), collapsed tables into one
line per thousand characters — news.ycombinator.com came back as four lines —
marked no headings and dropped every link. trafilatura now picks the content by
density, and its html output keeps the whitespace inside <pre> that its markdown
output throws away, so the fragment is rendered here: # headings, fenced code
with its indentation, table rows joined by |, and up to 40 links lifted into a
"Links on this page" appendix rather than sprinkled through the sentences.
Threads are read again with comments when the extractor keeps less than a
quarter of the visible text; if it still finds nothing, the old visible-text
read answers and metadata.extractor says which one did.
trafilatura goes into the server's own venv, which deploy/install.sh does not
build — docs/mcp.md and deploy/UPDATE.md carry the install line. The tests load
the server by path, so it also joins Navi's dev extra.
Eugene Sukhodolskiy
committed
1 day ago
