woody
/
legco_ai_assistant
1
1
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
legco_ai_assistant/.plans/package8_enhancement_plan.md

36 KiB
Raw Blame History

Package 8 Enhancement Plan — Q&A-Pair Chunking Strategy

Source: User request (2026-05-15) Scope: New "chunked by question" RAG algorithm that splits LegCo meeting documents into Q&A (問/答) pairs using LLM-assisted boundary detection, with vision-model-based table-to-markdown extraction. Coexists alongside existing "chunked by token" strategy — user selects strategy on the RAG Database page. Status: Draft

Objective

Feature 1: Q&A-Pair Chunking Algorithm

Split LegCo meeting transcript PDFs by question-answer pair boundaries rather than fixed token windows. Each Q&A pair becomes one ChromaDB chunk. The page number reference points to the question (問) location. This preserves the natural semantic unit of legislative documents where questions and answers form self-contained knowledge units.

Feature 2: Table-to-Markdown Extraction

Answers in LegCo documents frequently contain complex tables (budget figures, capacity tables, allowance rates). These tables must be converted to markdown format so the LLM can correctly read table contents during retrieval. Use vision LLM (GPT-4o or compatible) to identify tables in answer sections and render them as markdown tables.

Feature 3: Strategy Selection UI

Allow users to choose between "Chunked by token" (existing) and "Chunked by question" (new) on the RAG Database page. Strategy metadata is stored per-document and displayed in the document/chunk list views.

Non-goals: Does NOT modify the existing TokenChunkingStrategy. Does NOT change the ChromaDB storage schema beyond adding metadata fields. Does NOT affect the retrieval/query pipeline. Does NOT replace the existing per-chunk PDF generation (continues for both strategies).

Current State

Existing Chunking Pipeline

PDF → parse_pdf_by_page() [pypdf extract_text per page]
    → chunk_pages() [TokenChunkingStrategy: 1 chunk/page, 200-token overlap]
    → extract_metadata() [filename, date, summary, page_number, chunk_file_path]
    → rag.ingest_document() → collection.add() → ChromaDB

Chunking Strategy Abstraction

File: backend/app/utils/chunking.py

  • ChunkingStrategy ABC (line 13): defines chunk(text: str) -> List[str]
  • TokenChunkingStrategy(ChunkingStrategy) (line 28): token-based sliding window
  • chunk_pages(pages, overlap_tokens) (line 75): page-aware variant (PDF only)
  • NO strategy selection mechanism — TokenChunkingStrategy is hardcoded in the ingest router

Ingest Router

File: backend/app/routers/ingest.py

  • Line 44: from app.utils.chunking import TokenChunkingStrategy
  • Line 71: chunker = TokenChunkingStrategy(chunk_size=settings.chunk_size, overlap=settings.chunk_overlap)
  • Line 39: async def ingest_document(file: UploadFile = File(...)) — no strategy param
  • Three format branches (PDF/DOCX/TXT), each instantiates the same chunker

Config

File: backend/app/core/config.py

  • chunk_size: int = 1000 (line 41)
  • chunk_overlap: int = 200 (line 42)
  • No strategy config exists

Frontend

File: frontend/src/pages/RAGDatabasePage.tsx

  • handleUpload() calls ingestDocumentMutation.mutateAsync(file) — no strategy param
  • No strategy selector UI component exists

File: frontend/src/components/DocumentList.tsx

  • Displays filename, chunk_count, upload_date — no strategy info

File: frontend/src/components/ChunkList.tsx

  • Shows Page: {chunk.page_number} — no concept of question index

Document Analysis — LTT Test Materials

Four LegCo supplementary papers in test materials/LTT/:

File A: 詳細數字一覽表 (18 pages) — No Q&A format

  • Pure data tables appendix (17/18 pages are tables)
  • 8 tables covering development statistics, costs, compensation stats
  • Will fall back to token-based chunking (Q&A strategy produces 0 pairs)

File B: 古洞北粉嶺北新發展區概覽 (8 pages) — 3 Q&A pairs

  • 發言要點 (speaking points) section at start
  • 備用問答 section at end with 3 pairs: 問 B1/答 B1, 問 B2/答 B2, 問 B3/答 B3
  • Contains milestone tables and planning review result tables

File E: 基礎設施 (27 pages) — 12 Chinese Q&A + 3 "Others"

  • 發言要點 (pages 1-2): narrative briefing points
  • 備用問答 (pages 2-23): Q&A organized by topic sections:
    • (A) 排水系統 (Drainage): 問 A1-A3 / 答 A1-A3
    • (B) 污水收集 (Sewage): 問 B1 / 答 B1
    • (C) 供水系統 (Water supply): 問 C1 / 答 C1
    • (D) 公用設施共同溝 (Common utility enclosure): 問 D1-D7 / 答 D1-D7
  • Others section (pages 23-27): O1-O4 (combined Q&A without explicit 問/答 markers)
  • Contains tables within answers (treatment plant capacity table, common utility comparison)
  • Contains [內部參考...] / [內部補充資料...] internal reference brackets
  • Contains [如被追問...] followup answer patterns

File L: 補償及安置安排 (36 pages) — 24 English Q&A

  • Table of contents + 8 narrative policy sections (pages 1-13)
  • 備用問題 (pages 14-36): Q&A organized by topic:
    • 住戶／鄉村遷置: Q1-Q6
    • 業務經營者: Q7-Q16
    • 務農人士: Q17-Q21
    • 墳墓／金塔: Q22-Q24
  • No explicit "A" answer marker — answer ends at next Q or section header
  • Contains tables within answers (allowance rate tables, compensation amount tables)
  • Contains [如被追問...] / [如追問...] followup patterns
  • Contains [內部參考...] brackets

Key Structural Findings

Two distinct Q&A formats:

Style Files Q Pattern A Pattern Boundary Rule
Chinese 問/答 B, E 問\s*[A-Z]\d+\s*[︰：] 答\s*[A-Z]\d+\s*[︰：] Q ends at 答; A ends at next 問 or section header
English Q-only L ^Q\d+\s None (implicit) Q ends at end of Q text; A ends at next ^Q\d+ or topic line

Page layout conventions:

  • Page headers: (letter) on first page
  • Page footers: {letter}-{page_number}\n{YYYY-MM-DD} — these appear interspersed in extracted text at page boundaries
  • Section headers: (A) Section Name within Q&A blocks — serve as grouping markers
  • 備用問答 / 備用問題: section headers marking Q&A start

Answer content complexity:

  • Multi-paragraph answers (spanning 2-7 pages for complex topics)
  • Tables embedded mid-answer (capacity tables, financial tables)
  • Internal reference brackets [內部參考...] with technical data
  • Followup Q&A [如被追問...] — nested Q&A within answers
  • Bullet lists with various markers (⚫, ➢, -, i. ii. iii.)

Page boundary interference: Q&A pairs frequently span page boundaries. The pypdf extracted text retains page number markers like E-2, E-3 mid-text. These must be stripped or treated as boundaries.

Research Findings — External Approaches

Q&A Chunking Approaches (from librarian research)

1. FAQ-RAG (Pro-GenAI/FAQ-RAG): Treats each Q&A as an atomic retrieval unit. Stores dual embeddings per pair (question + answer). Gold standard for QA-native documents. Adaptation: Instead of generating FAQs, detect existing Q&A pairs in LegCo documents.

2. LLM Boundary Detection (LlamaChunk / semchunk): Uses LLM logprobs or prompt-based boundary detection to identify semantic breakpoints. semchunk AI mode achieves 37.7% correctness vs 33.3% for fixed-size chunking on legal RAG benchmarks. Relevance: For edge cases where regex fails (nested Q&A, tables spanning boundaries), use an LLM verification pass.

3. Regex + LLM Hybrid: Two-pass approach recommended by multiple sources:

  • Pass 1: Regex-based Q&A boundary detection (free, fast, 95%+ accurate for marked documents)
  • Pass 2: LLM verification for edge cases (multi-paragraph answers, embedded tables, nested Q&A)
  • Validated by 2026 EACL RAG error taxonomy: 29.7% of RAG errors from chunking mistakes

4. Index-RAG: Multi-embedding per chunk (question text + answer summary + topic + keywords). Optional enhancement — store multiple embeddings per Q&A pair for richer retrieval.

Vision Table Extraction (from librarian research)

1. gptpdf (CosmosShadow/gptpdf — 3,550 stars): 293 lines of Python. Uses PyMuPDF to detect non-text regions → marks with colored rectangles → GPT-4o converts page images to markdown. Cost: ~$0.013/page. Best fit: Lightweight, handles Chinese well, supports multiple vision LLMs.

2. llm-based-ocr (yigitkonur/llm-based-ocr — 890 stars): FastAPI service. PDF → Markdown with table preservation. Parallel processing (50-page PDF in seconds). Batch API cost: ~$4/1000 pages. Best fit for production: Cost-effective at scale.

3. Instructor + GPT-4o: Structured table extraction as Pydantic models with MarkdownDataFrame. Provides typed, structured table data directly usable in RAG. Best fit: When tables need to be queryable as structured data.

4. markdrop: Comprehensive PDF → Markdown with AI descriptions for images and tables. Supports 6 LLM providers (OpenAI, Gemini, Claude, Groq, OpenRouter, LiteLLM). Best fit: Full-pipeline solution but heavier than gptpdf.

Recommended Approach (synthesized)

PDF Transcript
    │
    ▼
[gptpdf / llm-ocr]  ← Vision LLM converts page images to markdown
    │                  Tables rendered as markdown tables inline
    ▼
Clean Markdown with 問/答 markers preserved
    │
    ▼
[Regex Splitter]  ← Primary: split on 問...答 boundaries
    │               Fallback: split on Q1, Q2... boundaries
    ▼
[LLM Boundary Verifier]  ← Edge cases: nested Q&A, tables at boundaries
    │
    ▼
Q&A Pair Chunks → extract_metadata(qa_style, question_index, ...)
    │
    ▼
ChromaDB (with strategy_type="question" in metadata)

Target State

Architecture

POST /api/v1/ingest?strategy=token|question
    │
    ├─ strategy="token" → TokenChunkingStrategy (existing, unchanged)
    │
    └─ strategy="question" → QuestionChunkingStrategy (NEW)
         │
          ├─ [PDF only] Render pages as images → LLM vision (base64) → markdown
          │   └─ Uses existing Qwen3.5-35B-A3B (native vision model, no separate API)
          │   └─ Tables converted to markdown table syntax inline
         │
         ├─ [Detect Q&A style] Chinese 問/答 vs English Q-only
         │
         ├─ [Primary split] Regex-based Q&A boundary detection
         │   ├─ Chinese: split on `問\s*[A-Z]\d+\s*[︰：]` → `答\s*([A-Z]\d+)`
         │   └─ English: split on `^Q\d+\s` → next Q/section = answer end
         │
         ├─ [LLM verification] Edge case handling:
         │   ├─ Multi-paragraph answers exceeding max chunk size
         │   ├─ Tables spanning Q&A boundaries (keep with answer)
         │   ├─ Nested Q&A (如被追問 within answers)
         │   └─ Ambiguous boundaries (O1-O4 style without explicit 問/答)
         │
         ├─ [Size check] Oversized pairs → recursive split on \n\n → \n
         │   └─ Split answer portions only; keep question + answer prefix together
         │
         ├─ [Fallback] Sections with no Q&A (發言要點, narrative policy)
         │   └─ Chunk by section headers (章節) or fall back to token-based
         │
         └─ [Metadata] Per-chunk:
              ├─ strategy_type: "question"
              ├─ qa_style: "chinese" | "english"
              ├─ question_index: int (0-based within document)
              ├─ question_id: str ("A1", "Q3", etc.)
              ├─ question_text: str (full question text)
              ├─ answer_contains_table: bool
              ├─ topic_section: str (e.g. "(A) 排水系統")
              └─ source_page_range: [int, int] (page range of Q&A pair)

Algorithm Detail: Q&A Boundary Detection

Step 1 — Preprocess extracted text:

  1. Concatenate all pages from parse_pdf_by_page() with \n separator
  2. Strip page footer markers: regex ^[A-Z]-\d+\s*\n\d{4}-\d{2}-\d{2}$ and ^[A-Z]-\d+$
  3. Strip page header letters ^(\([A-Z]\))\s*$ (keep on page 1 as doc identifier)
  4. Normalize colons: ： → :, ︰ → :

Step 2 — Detect Q&A style:

def detect_qa_style(text: str) -> str:
    chinese_pattern = r'問\s*[A-Z]\d+\s*[︰：]'
    english_pattern = r'^Q\d+\s'
    if re.search(chinese_pattern, text, re.MULTILINE):
        return "chinese"
    if re.search(english_pattern, text, re.MULTILINE):
        return "english"
    return "none"  # fall back to token-based

Step 3 — Split by Q&A (Chinese style):

QA_PATTERN_CHINESE = re.compile(
    r'(?:^|\n)\s*問\s*([A-Z]\d+)\s*[︰：]\s*(.*?)\s*'
    r'(?:\n\s*答\s*\1\s*[︰：]\s*)'  # matched answer marker
    r'(.*?)'
    r'(?=\n\s*(?:問\s*[A-Z]\d+\s*[︰：]|\([A-Z]\)|$))',
    re.DOTALL
)
# Group 1: question_id (e.g. "A1")
# Group 2: question_text
# Group 3: answer_text (everything until next 問 or section header or EOF)

Step 3b — Split by Q&A (English style):

QA_PATTERN_ENGLISH = re.compile(
    r'(?:^|\n)\s*(Q\d+)\s+(.*?)\s*'
    r'(?=\n\s*(?:Q\d+|[A-Z]\d+\s|'
    r'[\u4e00-\u9fff]{2,}\s*$|'  # Chinese-only topic line
    r'$))',
    re.DOTALL
)
# Group 1: question_id (e.g. "Q1")
# Group 2: full text (question + answer combined, since no explicit A marker)

Step 4 — LLM verification pass (for edge cases only): For each Q&A pair, if any of these conditions are true, send pair to LLM for verification:

  • Answer length > 2000 tokens (may contain multiple Q&A or tables)
  • Contains [如被追問 or [如追問 (nested Q&A)
  • Contains [內部參考 or [內部補充 (supplementary content to handle)
  • Answer text contains regex pattern for table rows

LLM prompt:

You are processing a Hong Kong Legislative Council document. 
Analyze this Q&A pair and determine:

1. Does the answer contain tables? If yes, output them as markdown tables.
2. Are there nested followup Q&A (追問) within the answer? If yes, separate them.
3. Does the answer contain [內部參考] supplementary content? Flag it.
4. Is the Q&A boundary correct? If not, suggest the correct boundary.

Return JSON:
{
  "correct_boundary": true/false,
  "tables_markdown": ["table1...", "table2..."],
  "nested_qa": [{"q": "...", "a": "..."}],
  "internal_refs": ["ref1...", "ref2..."],
  "clean_answer": "answer text with tables replaced by markdown"
}

Algorithm Detail: Table-to-Markdown

Option A — Vision LLM per page (Primary for PDF):

  1. Render each page of the Q&A section as an image (300 DPI, pypdf + PIL)
  2. Send to the existing LLM model (LLM_MODEL_NAME, currently qwen/qwen3.5-35b-a3b) which is a native vision-language model that accepts base64 images via the standard OpenAI Chat Completions API: 
    {
  "role": "user",
  "content": [
    { "type": "text", "text": "Convert this page to Markdown. For any tables, use |---|---| syntax..." },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } }
  ]
}
  3. No separate vision model or API key needed — uses the existing LLM_BASE_URL/LLM_API_KEY config
  4. Prompt: 
    Convert this page to Markdown. For any tables:
- Use proper markdown table syntax with |---|---| alignment
- Preserve all column headers and row labels
- Do not modify or translate the content
- If a table spans multiple pages, note it

Option B — Table detection + LLM extraction (Fallback for non-PDF):

  1. Detect table regions in extracted text using heuristics (rows of aligned numbers, pipe-delimited data)
  2. Send detected table text blocks to LLM with prompt: 
    Convert this raw table text extracted from a PDF into a markdown table.
Preserve all data exactly. Detect column boundaries and alignment.
  3. No vision API cost — uses text-only LLM

Option C — Hybrid (Recommended):

  • Use vision LLM (same model) for pages with complex multi-column/multi-row tables
  • Use text-based LLM for simple tables detected via regex
  • Cache vision results per page to avoid re-processing on re-ingest

Config Additions

# backend/app/core/config.py
class Settings(BaseSettings):
    # Existing...
    chunk_size: int = 1000
    chunk_overlap: int = 200

    # NEW: Chunking strategy selection
    default_chunking_strategy: str = "token"  # "token" | "question"

    # NEW: Q&A chunking config
    qa_vision_enabled: bool = True   # Toggle vision-based table extraction (uses existing LLM_MODEL_NAME)
    qa_max_chunk_tokens: int = 3000  # Max tokens per Q&A chunk (before forced split)
    qa_verification_model: str = ""  # LLM for boundary verification (empty = use LLM_MODEL_NAME)
    qa_include_internal_refs: bool = True  # Include [內部參考] in chunks
    qa_cache_vision_results: bool = True   # Cache vision results per page

    # Env var mappings
    # DEFAULT_CHUNKING_STRATEGY=token
    # QA_VISION_ENABLED=true
    # QA_MAX_CHUNK_TOKENS=3000
    # QA_VERIFICATION_MODEL=
    # QA_INCLUDE_INTERNAL_REFS=true
    # QA_CACHE_VISION_RESULTS=true

    # NOTE: Vision table extraction uses the existing LLM_MODEL_NAME
    # (qwen/qwen3.5-35b-a3b on OpenRouter — a native vision-language model).
    # No separate vision model config needed. Base64 images are sent
    # via the standard OpenAI Chat Completions API image_url format.

Implementation Plan

Sub-Phase 8.0: Config & Enums (Foundation)

Objective: Add all config settings and type definitions needed by subsequent sub-phases.

Test Files:

  • backend/app/test/test_phase1_config.py — add: test_qa_chunking_config_defaults
  • frontend/src/test/ (TBD) — type-level validation

Implementation Tasks:

  1. Add Q&A chunking config fields to backend/app/core/config.py
  2. Define ChunkingStrategyType = Literal["token", "question"] in backend/app/models/ingest.py
  3. Add VALID_CHUNKING_STRATEGIES = {"token", "question"} constant
  4. Update backend/.env.example with new env vars
  5. Add strategy fields to frontend types (ChunkingStrategy type, extend IngestResponse, DocumentInfo, ChunkInfo)

Sub-Phase 8.1: Q&A Chunking Strategy Class

Objective: Implement QuestionChunkingStrategy as a new ChunkingStrategy subclass.

Test Files:

  • backend/app/test/test_phase8_qa_chunking.py — new file
    • test_chinese_qa_detection — detect and split Chinese 問/答 pairs
    • test_english_qa_detection — detect and split English Q-only pairs
    • test_no_qa_fallback — section-based chunking when no Q&A found
    • test_multi_page_qa — Q&A spanning page boundaries
    • test_nested_qa_handling — [如被追問] patterns
    • test_table_preservation — tables embedded in answers
    • test_abc_contract — satisfies ChunkingStrategy ABC
    • test_page_number_reference_question — page ref points to 問 location
    • test_size_limit — oversized pairs get recursive split

Implementation Tasks:

  1. Create backend/app/utils/qa_chunking.py:

    • detect_qa_style(text: str) -> str — returns "chinese", "english", or "none"
    • preprocess_text(pages: List[Tuple[int, str]]) -> str — strip footers/headers, normalize colons, reconstruct full text
    • split_chinese_qa(text: str) -> List[QAPair] — regex-based splitting for 問/答 format
    • split_english_qa(text: str) -> List[QAPair] — regex-based splitting for Q-only format
    • QAPair dataclass: {qa_id, question, answer, start_page, end_page, topic_section, has_table, internal_refs}

  2. Create QuestionChunkingStrategy(ChunkingStrategy) in backend/app/utils/chunking.py:

    • chunk(text: str) -> List[str] — entry point for DOCX/TXT
    • chunk_pages(pages: List[Tuple[int, str]]) -> List[Tuple[str, int]] — entry point for PDF
    • Returns chunks with page number = question start page
    • Handles fallback: if no Q&A detected, falls back to token-based or section-based

  3. Create backend/app/utils/table_extraction.py:

    • extract_tables_vision(page_images: List[Image]) -> List[str] — vision LLM → markdown tables
    • extract_tables_text(text: str) -> List[str] — heuristic table detection → LLM markdown conversion
    • inject_tables_into_answer(answer: str, tables_md: List[str]) -> str — replace raw tables with markdown
    • cache_vision_result(page_hash: str) -> Optional[str] — disk cache for vision results

  4. Add get_chunking_strategy(name: str, settings: Settings) -> ChunkingStrategy factory function in backend/app/utils/chunking.py

Sub-Phase 8.2: Metadata Enrichment

Objective: Extend extract_metadata() to support Q&A-specific fields.

Test Files:

  • backend/app/test/test_phase8_metadata.py — new file
    • test_qa_metadata_fields — strategy_type, question_index, question_id, question_text
    • test_qa_metadata_topic_section
    • test_token_metadata_unchanged — existing metadata unchanged for token strategy
    • test_page_number_from_question — page ref points to question location

Implementation Tasks:

  1. Extend extract_metadata() in backend/app/utils/metadata.py:

    • New param: strategy_type: str = "token"
    • New param: qa_pairs: Optional[List[QAPair]] = None
    • When strategy_type == "question": populate qa_id, question_index, question_text, topic_section, qa_style, source_page_range, has_table
    • Maintain backward compatibility: all existing fields still populated

  2. Update backend/app/models/documents.py:

    • ChunkInfo: add strategy_type: str, question_index: Optional[int] = None, question_id: Optional[str] = None, question_text: Optional[str] = None, topic_section: Optional[str] = None

  3. Update backend/app/models/ingest.py:

    • IngestResponse: add strategy: str
    • Add IngestRequest model with strategy: str = "token"

Sub-Phase 8.3: Ingest API Integration

Objective: Wire the strategy selection into the ingest endpoint.

Test Files:

  • backend/app/test/test_phase8_ingest.py — new file
    • test_ingest_with_strategy_token — existing behavior unchanged
    • test_ingest_with_strategy_question — Q&A chunking applied
    • test_ingest_invalid_strategy_rejected — 400 on unknown strategy
    • test_ingest_question_strategy_pdf — PDF with Chinese Q&A
    • test_ingest_question_strategy_docx — DOCX with Q&A format
    • test_ingest_question_strategy_no_qa_fallback — document without Q&A falls back
    • test_ingest_response_includes_strategy — IngestResponse has strategy field

Implementation Tasks:

  1. Update backend/app/routers/ingest.py:

    • Line 39: accept strategy: str = Query("token") param
    • Validate against VALID_CHUNKING_STRATEGIES
    • Lines 70-73: replace hardcoded TokenChunkingStrategy with get_chunking_strategy(strategy, settings)
    • Lines 108-115, 148-151, 183-186: pass strategy_type=strategy and QA metadata to extract_metadata()
    • Line 192: include strategy in IngestResponse

  2. Update backend/app/services/rag.py:

    • list_documents(): extract strategy_type from first chunk's metadata per document
    • list_chunks(): return strategy_type, question_index, question_id, question_text, topic_section from metadata

  3. Update backend/app/routers/documents.py:

    • Map new fields from RAG service to API response models

Sub-Phase 8.4: Frontend Strategy Selector

Objective: Add strategy selection UI on the RAG Database page, display strategy info in document/chunk lists.

Test Files:

  • frontend/src/test/components/ChunkList.test.tsx — add Q&A chunk rendering tests
  • frontend/src/test/components/DocumentList.test.tsx — add strategy badge tests
  • frontend/src/test/pages/RAGDatabasePage.test.tsx (new) — strategy selector integration

Implementation Tasks:

  1. Update frontend/src/types/index.ts:

    • Add ChunkingStrategy type
    • Extend IngestResponse, DocumentInfo, ChunkInfo with new fields

  2. Update frontend/src/lib/api.ts:

    • ingestDocument(file, strategy = "token"): append strategy as query param

  3. Update frontend/src/lib/queries.tsx:

    • useIngestDocument: accept { file: File; strategy: ChunkingStrategy }

  4. Update frontend/src/pages/RAGDatabasePage.tsx:

    • Add chunkingStrategy state (default: "token")
    • Add strategy selector UI (two radio buttons or toggle):
      • "Chunk by Token" (icon: text icon, description: "Split by token windows with overlap")
      • "Chunk by Question" (icon: Q&A icon, description: "Detect Q&A pairs, extract tables")
    • Pass strategy to handleUpload
    • Show strategy info in success toast

  5. Update frontend/src/components/DocumentList.tsx:

    • Add strategy badge: chunked by token or chunked by question
    • Color-code: token = gray, question = blue

  6. Update frontend/src/components/ChunkList.tsx:

    • For Q&A chunks: show Q: {question_id} — {question_text} instead of page number
    • Show Topic: {topic_section} if available
    • Show Pages {start}-{end} for the page range
    • Show ⚠ Contains table badge if has_table is true

  7. Update frontend/src/components/DocumentUpload.tsx:

    • Disable upload button when no file selected (if not already)
    • Show strategy selector inline or accept selectedStrategy prop

Sub-Phase 8.5: Integration & Acceptance Testing

Objective: End-to-end verification with real LegCo PDFs from test materials/LTT/.

Test Files:

  • backend/app/test/acceptance/test_acceptance_phase8_qa_chunking.py — new file
    • test_real_qa_chunking_fileE — File E produces 12 Chinese Q&A pairs + 3 Others
    • test_real_qa_chunking_fileL — File L produces 24 English Q&A pairs
    • test_real_qa_chunking_fileB — File B produces 3 Chinese Q&A pairs + narrative sections
    • test_real_qa_chunking_fileA — File A falls back (no Q&A, should not error)
    • test_table_extraction_fileE — Tables in File E answers converted to markdown
    • test_table_extraction_fileL — Tables in File L answers converted to markdown
    • test_qa_page_references — Each Q&A chunk's page number points to question location
    • test_full_pipeline_question_strategy — Full ingest → retrieve → query with Q&A chunks

Implementation Tasks:

  1. Run Q&A chunking on all 4 LTT PDFs, verify pair counts and content
  2. Verify table extraction quality (spot-check markdown tables)
  3. Verify page number references point to question start
  4. Run pytest backend/app/test/test_phase8*.py -v — all pass
  5. Run pytest backend/app/test/acceptance/test_acceptance_phase8*.py -v -m acceptance — all pass
  6. Test frontend: upload File E with "chunk by question" → verify chunks displayed with Q&A metadata
  7. Verify strategy coexistence: upload File E with both strategies → both appear in document list with correct badges

Sub-Phase 8.6: Polish & Edge Cases

Objective: Handle remaining edge cases and polish UI.

Tasks:

  1. Handle Others section in File E (O1-O04 without explicit 問/答)
  2. Handle [如被追問] nested Q&A: include as part of answer or split as sub-chunks
  3. Handle [內部參考] inclusion/exclusion toggle via config
  4. Add chunking strategy filter to query page (optional — skip if out of scope)
  5. Add loading state for vision table extraction (can take seconds for large docs)
  6. Track chunking strategy in query history (bonus: show which strategy produced results)

File Change Summary

New Files

File Purpose
backend/app/utils/qa_chunking.py Q&A boundary detection, splitting, QAPair dataclass
backend/app/utils/table_extraction.py Vision LLM table extraction, heuristic fallback, caching
backend/app/test/test_phase8_qa_chunking.py Tests for Q&A chunking logic
backend/app/test/test_phase8_metadata.py Tests for Q&A metadata enrichment
backend/app/test/test_phase8_ingest.py Tests for strategy selection in ingest API
backend/app/test/acceptance/test_acceptance_phase8_qa_chunking.py Acceptance tests with real LTT PDFs

Modified Files

File Changes
backend/app/utils/chunking.py Add QuestionChunkingStrategy class + get_chunking_strategy() factory
backend/app/core/config.py Add Q&A chunking config fields (5 new — vision uses existing LLM model)
backend/app/utils/metadata.py Add strategy_type and Q&A fields to metadata output
backend/app/models/ingest.py Add strategy to IngestResponse, add IngestRequest
backend/app/models/documents.py Extend ChunkInfo with Q&A fields, DocumentInfo with chunking_strategy
backend/app/routers/ingest.py Accept strategy query param, use factory, pass strategy to metadata
backend/app/services/rag.py list_documents()/list_chunks() return strategy info
backend/app/routers/documents.py Map new fields to API responses
backend/.env.example Add new env vars
frontend/src/types/index.ts Add ChunkingStrategy type, extend interfaces
frontend/src/lib/api.ts ingestDocument() accepts strategy param
frontend/src/lib/queries.tsx useIngestDocument() mutation signature updated
frontend/src/pages/RAGDatabasePage.tsx Add strategy selector state + UI
frontend/src/components/DocumentUpload.tsx Accept strategy prop / integrate selector
frontend/src/components/DocumentList.tsx Display strategy badge
frontend/src/components/ChunkList.tsx Conditional Q&A display (question_id instead of page_number)

Test Files (Modified)

File Changes
backend/app/test/test_phase1_chunking.py Add test for strategy factory
backend/app/test/test_phase1_ingest.py Add strategy param test
backend/app/test/test_phase1_documents_router.py Add strategy field verification
backend/app/test/test_phase1_metadata.py Add Q&A metadata field tests
backend/app/test/test_phase1_config.py Add Q&A config tests
frontend/src/test/components/ChunkList.test.tsx Add Q&A chunk display tests
frontend/src/test/components/DocumentList.test.tsx Add strategy badge test
frontend/src/test/components/IngestPanel.test.tsx Update for strategy selection

Acceptance Criteria

  • AC1: User can select "Chunk by token" or "Chunk by question" on the RAG Database page before uploading
  • AC2: "Chunk by token" uploads produce chunks identical to current behavior (no regression)
  • AC3: "Chunk by question" upload of File E produces 12+ Chinese Q&A chunks with correct question_id, question_text, topic_section metadata
  • AC4: "Chunk by question" upload of File L produces 24 English Q&A chunks with correct boundaries
  • AC5: Each Q&A chunk's page number references the question (問) page, not the answer
  • AC6: Tables within answers are converted to markdown table syntax (vision LLM or heuristic)
  • AC7: Document list shows chunking strategy badge per document
  • AC8: Chunk list shows question_id/question_text for Q&A chunks instead of page_number
  • AC9: Documents without Q&A format (File A, narrative sections) fall back to token/section-based chunking without error
  • AC10: Both strategies coexist — documents stored with different strategies appear side-by-side in document list
  • AC11: Strategy param is validated — invalid values return 400
  • AC12: All backend tests pass (pytest backend/app/test/test_phase8*.py -v)
  • AC13: All acceptance tests pass (pytest backend/app/test/acceptance/test_acceptance_phase8*.py -v -m acceptance)
  • AC14: Frontend tests pass (pnpm test in frontend/)
  • AC15: Existing tests still pass (no regression)

Risks & Mitigations

Risk Severity Mitigation
Vision LLM cost: Per-page image processing adds latency and token cost for large docs Medium Cache vision results per page hash. Skip vision for pages without detected tables. qa_vision_enabled toggle allows disabling vision and using heuristic-only extraction. Vision uses the same model/API key — no additional provider cost.
Q&A boundary false positives: Regex matches non-Q&A text as Q&A High Use detect_qa_style() first — only apply Q&A splitting if document has sufficient Q&A density. LLM verification pass for ambiguous boundaries. Fall back to section-based chunking if < 3 pairs detected.
Q&A boundary false negatives: Regex misses some Q&A pairs (O1-O4 style) Medium LLM verification pass catches these. Log missed pairs for manual review.
Table extraction quality: Vision LLM may misinterpret complex Chinese tables Medium Flag has_table=True so retrieval can surface context. Allow user to inspect raw chunk content. Provide config to disable vision and use heuristic-only extraction.
Performance: Vision LLM + LLM verification adds latency to ingest Medium Make vision extraction async (background task). Show progress in frontend. Cache results. Allow skipping verification for trusted documents.
Page boundary interference: Page footer numbers appear mid-Q&A text Low Preprocessing step strips known footer patterns. Test with all 4 LTT PDFs.
LegCo format drift: Future documents may use different Q&A markers Low Detection is regex-based — easy to add new patterns. LLM verification catches novel formats. Log format detection results for monitoring.
Chunk size: Some Q&A pairs are very long (7+ pages) Medium Apply max chunk token limit (configurable, default 3000). Recursive split on \n\n → \n with question text prepended to each sub-chunk for context.
DOCX/TXT Q&A: Non-PDF formats may have different Q&A markers Low Use same regex detection on concatenated text. Skip vision table extraction (text-based only).

Dependencies

  • Vision LLM API: The existing LLM_MODEL_NAME (qwen/qwen3.5-35b-a3b) is a native vision-language model that accepts base64 images via OpenRouter's standard OpenAI Chat Completions API. No separate vision model, API key, or endpoint needed. If vision is unavailable or disabled (QA_VISION_ENABLED=false), fall back to heuristic table detection (text-only).
  • New Python packages: Pillow (likely already installed for image rendering). pypdf already installed. No pymupdf needed — vision extraction sends raw page images (PNG) directly to the LLM; the LLM itself identifies table regions.
  • Existing codebase: No breaking changes. Strategy is additive — existing TokenChunkingStrategy unchanged.
  • ChromaDB: No schema change. Metadata fields are flexible (ChromaDB accepts arbitrary dict keys).

Cost Estimate (per document)

Component Model Cost Notes
PDF text extraction pypdf Free Existing parser
Q&A regex splitting N/A Free Pure Python
Vision table extraction (PDF) Same as LLM_MODEL_NAME Image token cost only Only pages with tables; no separate model cost
LLM boundary verification Same as LLM_MODEL_NAME ~$0.001/document Edge cases only
Embedding qwen3-embedding-4b ~$0.00002/page Existing pipeline

Single model, single API key. Vision table extraction and boundary verification both use the existing LLM_MODEL_NAME (qwen/qwen3.5-35b-a3b) via the same LLM_BASE_URL/LLM_API_KEY. The only incremental cost is image tokens for pages sent to the vision model — estimated at a few cents per document for typical LegCo papers.

References

Internal

  • backend/app/utils/chunking.py — Current chunking strategy
  • backend/app/routers/ingest.py — Ingestion pipeline
  • backend/app/utils/metadata.py — Metadata extraction
  • test materials/LTT/ — Test documents (4 LegCo PDFs)

External