Commit Graph

7 Commits

Author SHA1 Message Date
Woody f2115ae563 feat: structured LLM output for decompose + citation fuzzy matching (Phase 5)
Phase 5.1 — Structured LLM output for query decomposition:
- Add SubQuestions Pydantic model with sub_question, keywords, rationale
- Add LLMClient.complete_structured() using langchain with_structured_output
- Update QueryDecomposer with structured output path + legacy json.loads fallback
- Update SQLite seed templates: add subq+citation labeling requirement
- Add tests: structured output, subquestions model validation, logging

Phase 5.2 — Citation format alignment and fallback links:
- Add document_id to SourceMetadata (backend + frontend types)
- Rewrite citationParser.ts with fuzzy matching and fallback document links
- Add RAGDatabasePage auto-expand from ?document= URL param
- Tighten generate_per_subq seed prompt: 'Copy exact bracket labels shown'
- Add citation parser tests for fuzzy match and fallback link scenarios
- Defer: DOCX/TXT PDF generation → Phase 5.3 (fallback links sufficient)
2026-04-28 15:39:17 +08:00
Woody 475306f2b1 feat(history): Phase 3.5 — Query History backend (service, API, timing, XML capture) 2026-04-25 22:59:53 +08:00
Woody f9dda7bd18 feat(backend): rename keywords to extracted_questions in query pipeline (sub-phase 2.3)
Change QueryDecomposer prompt to generate 2-5 sub-questions instead of keywords. Rename API field from keywords to extracted_questions across models, service, and router.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-24 16:23:53 +08:00
Woody 33b960f786 fix(backend): extract JSON from markdown code blocks in LLM responses
The LLM (Qwen3.5 via OpenRouter) returns JSON wrapped in markdown code blocks:

```json

["project manager", "limits", ...]

```

But the code was trying to parse this directly with json.loads(), causing:

- QueryDecomposer to return empty keywords

- RelevanceFilter to fail with "Expecting value: line 1 column 1"

Changes:

- Added _extract_json_from_markdown() helper function to both modules

- Strips markdown code block markers (```json and ```) before JSON parsing

- Added unit tests for markdown code block handling

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus \u003cclio-agent@sisyphuslabs.ai\u003e
2026-04-23 16:28:07 +08:00
Woody f5cfe44183 feat(backend): add LLM monitoring with step names, timing, and prompt logging
- LLMClient.complete() now accepts step_name parameter to identify processing step

- Logs prompt preview (first 100 + last 100 chars) at INFO level

- Logs processing time in milliseconds with token usage stats

- Updated QueryDecomposer, RelevanceFilter, and RAGService to pass step names

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-23 14:51:57 +08:00
Woody f4d78b0b77 refactor(backend): update query decomposer, relevance filter, and RAG service
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-23 13:26:56 +08:00
Woody 181f4eca5b feat: Phase 1.3 query pipeline with decomposition, relevance filter, and response
- Add QueryDecomposer: extracts keywords from question via LLM JSON response
- Add RelevanceFilter: batch scores chunks 0-10, filters by threshold
- Add POST /api/v1/query endpoint with full 3-step pipeline:
  1. QueryDecomposer.decompose() → keywords
  2. RAGService.retrieve() → chunks from ChromaDB
  3. RelevanceFilter.filter() → score and filter chunks
  4. RAGService.generate_response() → bullet-point answer
- Fix SourceMetadata.upload_date type from datetime to str for flexibility
- Test-first: 13 new tests pass (5 decomposer, 5 relevance filter, 3 query endpoint)
- All Phase 1 tests: 41 passed, 2 skipped
2026-04-22 17:19:21 +08:00