feat: background queue worker for async pipeline processing
- Endpoint /ingest now only validates payload, creates raw_data with
  status=pending, commits and returns 202 (no longer blocks on AI).
- QueueWorker polls DB for pending jobs every 1s, grabs one with
  FOR UPDATE SKIP LOCKED, marks it processing, runs PropertyPipeline.
- PipelineFactory extracted from deps.py for reuse by both HTTP deps
  and the background worker.
- Lifespan starts QueueWorker as asyncio.Task; on shutdown signals
  stop_event, awaits worker termination (60s timeout) before closing
  Ollama client and active_jobs.
- Worker checks pipeline result status and logs completed/invalid/failed
  appropriately. Unhandled exceptions mark raw_data failed explicitly.

Co-Authored-By: Claude <noreply@anthropic.com>
1 parent f587f56 commit 4012d038a5a15372fad7eaa2ca0ea566a3f33064
@Eugene Sukhodolskiy Eugene Sukhodolskiy authored 1 day ago
Showing 5 changed files
View
src/vmk_data_collector/api/deps.py
View
src/vmk_data_collector/api/v1/router_properties.py
View
src/vmk_data_collector/main.py
View
src/vmk_data_collector/services/pipeline_factory.py 0 → 100644
View
src/vmk_data_collector/services/queue_worker.py 0 → 100644