Enhance native toolset and add persistent KV store ...

- Add PostgreSQL-backed KvStore (navi/store/) for session-scoped data.
- Migrate todo and scratchpad from in-memory dicts to KvStore.
- Filesystem: add copy, grep, diff actions; compress description.
- CodeExec: remove language param, expose working_dir in schema.
- ImageView: resize to 1024px JPEG + Content-Type guard for URLs.
- Memory list: return distinct categories instead of all facts.
- SSH: add scp action with upload/download support.
- Update CLAUDE.md (Postgres-only), docs/tools.md, add docs/store.md.
- Fix agent/planning/context_builder async signatures for todo helpers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor: move tool helper modules into _internal subpackage ...

Moves non-tool infrastructure out of navi/tools/ root so that only
actual tool classes live there:

  base.py           → _internal/base.py
  loader.py         → _internal/loader.py
  middleware.py     → _internal/middleware.py
  logging_middleware.py → _internal/logging_middleware.py
  _time_parser.py   → _internal/time_parser.py

All imports updated across core/, api/, mcp/, tools/, and tests/.
No proxy files remain in navi/tools/ root.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: filesystem empty path validation and terminal timeout defaults ...

filesystem:
- Reject empty string paths in _check_path to prevent writing to cwd
- Clear error message when 'write' action uses 'destination' instead of 'path'

terminal:
- Reduce default timeout from 300s to 20s
- Clamp user-provided timeout between 1 and 300s
- Update parameter description to mention max and common use-cases

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Clarify knowledge persistence prompts

Add deterministic line-based file editing (edit_lines), rating UI fix, and session refresh ...

- filesystem.py: add edit_lines action (deterministic line ops via operations array)
           + numbered param for read (1-based line numbers in output)
           + clarify four editing modes in tool description
- chat.js: fix rating IDs for streaming messages (assign h_ ID on stream_end)
- SessionList.vue: mobile pull-to-refresh with PTR_THRESHOLD=80
- AppSidebar.vue: desktop refresh button next to Conversations header
- planning.py: knowledge source assessment in Phase 1
- debug panel: MCP servers tab + resolved tools per profile
- NAVI.md: reposition as neutral quick-reference

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add multi-user sandbox: filesystem, terminal, code_exec, security policy ...

- filesystem, share_file: sandbox non-admin users to user_data/<user_id>/
- terminal: working_dir sandbox + allowlist + dangerous pattern block for users
- code_exec: sandbox CWD and temp files to user_data/<user_id>/ for users
- context_builder: inject dynamic security policy into LLM context (user/admin)
- config: terminal_user_allowed_commands setting
- agent: wire user_id/user_role through ContextBuilder.build()
- base: add current_user_role ContextVar; run_ephemeral inherits role

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add per-user filesystem sandbox via current_user_id ContextVar ...

- tools/base.py: add current_user_id ContextVar (set by Agent before
  every tool call, cleared after)
- core/agent.py: set current_user_id in run_stream from session.user_id
  and in run_ephemeral from parent_session.user_id; restore in finally
- tools/filesystem.py: _check_path resolves all paths inside
  user_data/<user_id>/ when current_user_id is present; legacy mode
  (no user_id) falls back to FS_ALLOWED_PATHS
- tools/share_file.py: validate source path is inside user sandbox

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Improve content publishing UX

filesystem mkdir: parents=False to prevent accidental path chain creation ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add mkdir action to filesystem tool (parents=True, exist_ok=True) ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Make server I/O non-blocking; update docs ...

- Wrap all heavy filesystem ops in asyncio.to_thread: filesystem tool
  (read/write/append/list/find/info/move/delete/query/smart_edit),
  image_view (read_bytes), share_file (shutil.copy2), write_tool
  (write_text, _register_user_tool), session_files (shutil.rmtree,
  iterdir), sessions upload endpoint (sync open/write → to_thread)
- Make delete_session_dir async; update its caller in sessions.py
- docs/config.md: fix wrong defaults (threshold 0.70, keep_recent 8),
  remove phantom SESSION_FILES_TTL_HOURS, add LLM timeouts, DATABASE_URL,
  PUBLIC_URL, Gmail, CONTEXT_SUMMARY_MAX_TOKENS sections
- docs/profiles.md: add missing tool_developer profile to table
- android-client: add WebView remote debugging; remove unused toolbar menu
- Remove stale helper scripts and test files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Agent improvements: mandatory planning, tool cleanup, smart_edit fixes ...

- Planning now mandatory on first message of every session (force_plan)
- RESOURCES, COMMITMENTS, ATOMICITY fields added to planning phase 1
- Todo auto-injected at iteration 0 so model tracks steps immediately
- Execution trigger injected after plan to prevent model treating plan as response
- Split developer profile: tool_developer (Navi tools) vs developer (general code)
- Simplified persona.txt: trimmed redundant content now handled by mechanics
- AIHelper.ask(): 120s timeout via asyncio.wait_for to prevent smart_edit hangs
- filesystem._smart_edit(): atomic write via temp file + os.replace()
- Removed 5 junk user tools (game project artifacts, trivial utilities)
- Removed instagram tools (to be rewritten); cleaned enabled.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Planning debug panel, todo auto-populate, scratchpad/persona improvements ...

- Planning debug panel: new Planning tab in debug/index.html shows raw
  phase 1/2 outputs and token counts per planning run, stored in
  session.planning_logs (new column in both SQLite and PostgreSQL)
- New GET /sessions/{id}/planning API endpoint
- PlanningDebugData internal event wires _run_planning() output into
  session storage; never forwarded to WebSocket clients
- Phase 3 (plan critic) disabled — to be reworked with reflect integration
- Todo tool: auto-populated from plan steps after phase 2; model only
  needs to call update/view, not set
- Scratchpad: clarified description and persona instructions; removed
  context_transfer from user-facing docs (internal mechanism only)
- web_search: switched to ddgs package, SearXNG as primary backend,
  DDG html-only fallback; added find_up action to filesystem tool
- Persona: added SCRATCHPAD and TODO sections with clear usage rules;
  added NAVI.md project context instructions
- chat.js: fixed subagent planning event fallthrough into parent UI;
  statusLabel cleared on first stream delta

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve filesystem tool description: prioritize AI actions with examples ...

- Description now opens with explicit ALWAYS PREFER rule for query/smart_edit
- Each AI action has concrete examples (function lookup, renaming, config search)
- Standard actions demoted to 'use only when AI not applicable'
- question/instruction parameters include examples so the model understands
  the full range of applicable cases

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add AIHelper + filesystem query/smart_edit AI actions ...

AIHelper (navi/core/ai_helper.py):
- Reusable LLM utility for AI-enhanced tools: ask() and ask_json()
- Reads current_model ContextVar (set per-turn) so tools always use
  the session's active model without extra wiring
- Robust JSON extraction: strips markdown fences, bracket-matching fallback

current_model ContextVar (navi/tools/base.py):
- New ContextVar set by run_stream() and run_ephemeral() after profile
  is resolved; AIHelper reads it to pick the right model automatically

filesystem query action:
- Natural language question about any file, chunked at ~20k tokens of
  content (~80k chars) with 30-line overlap between chunks
- Single-chunk: one LLM call; multi-chunk: partial answers accumulated
  then synthesized in a final call

filesystem smart_edit action:
- Natural language edit instruction on files up to ~200k chars
- LLM outputs JSON patch ops: replace / delete / insert (1-based lines)
- Ops validated then applied bottom-up to preserve line numbers
- Returns unified diff of changes; preserves trailing newline

registry: AIHelper created once, OllamaBackend reused (no double init),
FilesystemTool receives ai_helper at construction

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve filesystem, web search, context guard, and subagent narration ...

filesystem: add find (glob), info (stat), move, append actions; read now
supports offset/limit with hard 1MB guard; list shows sizes, dates,
optional recursion.

web_search: retry DDG across auto/html/lite backends; add optional Brave
Search API and SearXNG fallbacks configured via .env.

agent: fix ContextTooLargeError to surface as Navi response instead of
raw system error; fix _check_context_size to calculate from remaining
budget (window - output_reserve) rather than a fixed 92% threshold.

persona: add ReAct narration instruction to subagent briefing template.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Wire self-extension tools into all profiles; improve tool descriptions ...

All profiles now include write_tool, list_tools, tool_manual, reload_tools.
User tools from enabled.json merged in at runtime via Agent._tool_list().
Built-in tool descriptions rewritten to be more LLM-actionable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Filesystem tool: unrestricted mode by default ...

FS_ALLOWED_PATHS=* (default) allows any path, consistent with terminal.
Path check is now lazy (per-request) instead of module-level,
so config is always fresh after restart.
~ expansion added.

Initial implementation of the agent system core ...

- FastAPI server with REST API and WebSocket streaming
- Modular LLM backend abstraction (Ollama implemented, OpenAI stub)
- Tool system: web_search (ddgs), filesystem, http_request, code_exec, terminal
- Agent profiles: smart_home, server_admin, secretary
- Tool-calling loop with concurrent tool execution
- In-memory session store with SessionStore ABC for future persistence
- Registry pattern for tools, profiles, and backends
- Orchestrator stub as foundation for multi-agent scenarios