|
feat: background queue worker for async pipeline processing
- Endpoint /ingest now only validates payload, creates raw_data with status=pending, commits and returns 202 (no longer blocks on AI). - QueueWorker polls DB for pending jobs every 1s, grabs one with FOR UPDATE SKIP LOCKED, marks it processing, runs PropertyPipeline. - PipelineFactory extracted from deps.py for reuse by both HTTP deps and the background worker. - Lifespan starts QueueWorker as asyncio.Task; on shutdown signals stop_event, awaits worker termination (60s timeout) before closing Ollama client and active_jobs. - Worker checks pipeline result status and logs completed/invalid/failed appropriately. Unhandled exceptions mark raw_data failed explicitly. Co-Authored-By: Claude <noreply@anthropic.com> |
|---|
|
|
| src/vmk_data_collector/api/deps.py |
|---|
| src/vmk_data_collector/api/v1/router_properties.py |
|---|
| src/vmk_data_collector/main.py |
|---|
| src/vmk_data_collector/services/pipeline_factory.py 0 → 100644 |
|---|
| src/vmk_data_collector/services/queue_worker.py 0 → 100644 |
|---|