diff --git a/.plans/debug_2026-05-15_vllm_highlight_failure.md b/.plans/debug_2026-05-15_vllm_highlight_failure.md new file mode 100644 index 0000000..b89d8e7 --- /dev/null +++ b/.plans/debug_2026-05-15_vllm_highlight_failure.md @@ -0,0 +1,241 @@ +# Fix Plan: vLLM Highlight Batch Structured Output Failure + +**Date**: 2026-05-15 (updated 2026-05-15 with external research) +**Source**: User report — highlight function breaks when deploying to local vLLM server, works fine on OpenRouter +**Status**: Root cause confirmed via internet research + +--- + +## Symptom + +When `VLLM_ENGINE=true`, the highlight feature fails silently: +- Citations appear as plain PDF links instead of highlighted chunk views +- Frontend console logs: `"Highlight batch computation failed"` +- Backend returns `{"status": "failed"}` from `POST /api/v1/v2/highlights/batch` + +When `VLLM_ENGINE=false` (OpenRouter), highlights work correctly. + +--- + +## Root Cause Analysis (Confirmed) + +### The Highlight Pipeline + +``` +SSE stream completes + → Frontend ResponsePanel.tsx calls POST /api/v1/v2/highlights/batch + → Backend chunks.py creates LLMClient(settings) + → ChunkHighlightService.compute_highlights_batch() + → self._llm.complete_structured(prompt, HighlightBatchResult) ← FAILS HERE +``` + +### Why It Works on OpenRouter + +`LLMClient._complete_structured_openai()` uses LangChain's `with_structured_output(pydantic_model, method="json_schema")`. LangChain sends OpenAI-native `response_format: {type: "json_schema", json_schema: {...}}` to the API. OpenRouter proxies this to the underlying model properly. Returns pre-validated Pydantic model. + +### Why It Fails on vLLM — CONFIRMED ROOT CAUSE + +`LLMClient._complete_structured_vllm()` (lines 140-191 of `llm_client.py`) tries two `extra_body` formats: + +| Attempt | Format | Status on latest vLLM | +|---------|--------|----------------------| +| 1 | `extra_body={"structured_outputs": {"json": schema}}` | ✅ Supported (v0.8+) — but may fail due to `chat_template_kwargs` merge conflict | +| 2 | `extra_body={"guided_json": schema}` | ❌ **REMOVED in v0.12.0** | + +**`guided_json` was deprecated in v0.11.0 and fully removed in v0.12.0.** This was confirmed via: + +- **[vLLM Official Docs](https://docs.vllm.ai/en/stable/features/structured_outputs/):** + > ⚠️ If you are still using the following deprecated API fields which were removed in v0.12.0: + > `guided_json` → `{"structured_outputs": {"json": ...}}` + +- **[vLLM RFC #19097](https://github.com/vllm-project/vllm/issues/19097):** The vLLM team confirmed the migration path and is moving all structured output constraints into `response_format`. + +When `structured_outputs` attempt 1 fails (e.g., due to the `{**base_extra, **body}` merge conflating `chat_template_kwargs` with structured output params), the `guided_json` fallback is **silently rejected** by vLLM v0.12+ — resulting in `LLMClientError` and `status: "failed"`. + +### What Latest vLLM Actually Supports + +Research confirmed three supported methods on latest vLLM (v0.12+): + +| # | Method | Format | Since | Evidence | +|---|--------|--------|-------|----------| +| 1 | `response_format` | `{"type": "json_schema", "json_schema": {"name": "...", "schema": schema}}` | v0.6.4 | [vLLM protocol.py](https://github.com/vllm-project/vllm/blob/master/vllm/entrypoints/openai/chat_completion/protocol.py) `to_sampling_params()`, [PR #7654](https://github.com/vllm-project/vllm/pull/7654), [PR #9530](https://github.com/vllm-project/vllm/pull/9530) | +| 2 | `extra_body` | `{"structured_outputs": {"json": schema}}` | v0.8+ | [vLLM test_chat.py](https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/openai/chat_completion/test_chat.py) (extensive test coverage), third-party repos: `encourage`, `mellea` | +| 3 | ~~`guided_json`~~ | ❌ | — | Removed v0.12.0 per official docs | + +### `$defs` / Nested Schema Support + +The `HighlightBatchResult` schema produces `$defs` from Pydantic's `model_json_schema()`: + +```json +{ + "$defs": { + "ChunkHighlights": { "properties": { ... } }, + "RelevantSentence": { "properties": { ... } } + }, + "properties": { + "results": { "items": { "$ref": "#/$defs/ChunkHighlights" }, "type": "array" } + } +} +``` + +**`$defs` are fully supported** on latest vLLM with the `auto` backend (default): +- **xgrammar**: Native `$defs` support via `xgr.Grammar.from_json_schema()` — `$defs` is NOT in `has_xgrammar_unsupported_json_features()` ([source](https://github.com/vllm-project/vllm/blob/main/vllm/v1/structured_output/backend_xgrammar.py)) +- **guidance**: Native support via `llguidance.LLMatcher.grammar_from_json_schema()` +- **`auto` backend**: Tries xgrammar → guidance → outlines, handling fallback transparently +- **[PR #15627](https://github.com/vllm-project/vllm/pull/15627)**: Specifically addressed `$defs` for the Outlines backend (closed when V0 removed; xgrammar/guidance already handled it) +- **vLLM test fixtures**: `sample_definition_json_schema` fixture tests `$defs` + `$ref` schemas in production + +### Why QueryDecomposer Works But Highlights Don't + +`QueryDecomposer` uses `LLMClientDP` (Deepseek), which has its own `complete_structured()` with Deepseek JSON mode (`response_format: {type: "json_object"}`) — entirely different code path, unaffected by vLLM's guided_json removal. + +`ChunkHighlightService` uses the main `LLMClient` and calls `complete_structured()` which routes to `_complete_structured_vllm()` → dead `guided_json` fallback. + +--- + +## Fix Plan (Revised — 2026-05-15) + +### Fix A (PRIMARY): Update `_complete_structured_vllm()` to use supported formats + +**Approach**: Replace the two-attempt `guided_json` loop with a two-tier approach using formats confirmed to work on latest vLLM. + +**Implementation** (`llm_client.py`, replace `_complete_structured_vllm()` lines 140-191): + +```python +async def _complete_structured_vllm(self, prompt, pydantic_model, step_name, start_time): + """Use vLLM-native structured output via response_format or structured_outputs extra_body.""" + schema = pydantic_model.model_json_schema() + model_name = pydantic_model.__name__ + + # Tier 1: OpenAI-native response_format (vLLM v0.6.4+, portable to OpenRouter/OpenAI) + # This is the format vLLM's own docs recommend and RFC #19097 confirms as future direction. + try: + response = await self._client.chat.completions.create( + model=self.model, + messages=[{"role": "user", "content": prompt}], + temperature=0.0, + response_format={ + "type": "json_schema", + "json_schema": {"name": model_name, "schema": schema}, + }, + ) + content = response.choices[0].message.content or "" + elapsed_ms = (time.perf_counter() - start_time) * 1000 + self.logger.info( + "[%s] vLLM structured succeeded via response_format in %.2fms", step_name, elapsed_ms + ) + return pydantic_model.model_validate_json(content) + except Exception as exc: + self.logger.warning( + "[%s] response_format failed: %s. Trying structured_outputs fallback.", step_name, exc + ) + + # Tier 2: extra_body structured_outputs (vLLM v0.8+, most battle-tested fallback) + try: + response = await self._client.chat.completions.create( + model=self.model, + messages=[{"role": "user", "content": prompt}], + temperature=0.0, + extra_body={"structured_outputs": {"json": schema}}, + ) + content = response.choices[0].message.content or "" + elapsed_ms = (time.perf_counter() - start_time) * 1000 + self.logger.info( + "[%s] vLLM structured succeeded via structured_outputs in %.2fms", step_name, elapsed_ms + ) + return pydantic_model.model_validate_json(content) + except Exception as exc: + elapsed_ms = (time.perf_counter() - start_time) * 1000 + self.logger.error( + "[%s] Both response_format and structured_outputs failed after %.2fms", + step_name, elapsed_ms, exc_info=True, + ) + + raise LLMClientError("vLLM structured output failed with all supported formats") +``` + +**What changes from current code:** +1. ❌ Remove `guided_json` loop (removed in v0.12.0 — confirmed dead code) +2. ❌ Remove `{**base_extra, **body}` merge (no more conflating `chat_template_kwargs` with structured output — `_build_extra_body()` remains for `complete()` text calls only) +3. ✅ Add `response_format` as Tier 1 (vLLM v0.6.4+, OpenRouter-compatible, future-proof per RFC #19097) +4. ✅ Keep `structured_outputs` as Tier 2 (battle-tested fallback for v0.8+ deployments) +5. ✅ Add per-tier success/failure logging for diagnostics + +### Fix B (DEFENSE-IN-DEPTH): Add markdown fence stripping + +**Approach**: vLLM models may wrap JSON output in markdown code fences even with structured output. Strip them before `model_validate_json()`. + +**Implementation** (`llm_client.py`, in both Tier 1 and Tier 2 of `_complete_structured_vllm()`, before `model_validate_json()`): + +```python +# Strip markdown code fences if present +import re +match = re.search(r"```(?:json)?\s*\n?(.*?)\n?```", content, re.DOTALL) +if match: + content = match.group(1).strip() +``` + +### Fix C (FALLBACK): Add plain complete() fallback to ChunkHighlightService + +**Approach**: If both structured output tiers fail, fall back to `complete()` with JSON format instruction. Only needed as last-resort defense if the vLLM server is misconfigured (e.g., `--structured-outputs-config.backend` not set). + +**Implementation** (`chunk_highlight_service.py`, in `compute_highlights_batch()`): + +Same as original Fix A in this plan. This is now demoted to Fix C since Fix A (updating the vLLM path) should resolve the issue at the LLM client layer for all structured output callers. + +--- + +## Files to Change + +| File | Changes | Priority | +|------|---------|----------| +| `backend/app/services/llm_client.py` | Fix A: replace `_complete_structured_vllm()` with two-tier approach; Fix B: add markdown fence stripping | **HIGH** | +| `backend/app/services/chunk_highlight_service.py` | Fix C: add JSON fallback in `compute_highlights_batch()` (defense-in-depth) | Medium | + +--- + +## Test Plan + +1. **Reproduce locally**: Set `VLLM_ENGINE=true` in `.env`, point to vLLM server, run a query, observe highlight batch failure +2. **Apply Fix A+B**: Update `_complete_structured_vllm()`, rebuild Docker image, redeploy to vLLM +3. **Verify highlights work**: Run a query against uploaded PDF, confirm citation links show highlighted chunk views +4. **Verify logs**: Check that `response_format` Tier 1 succeeds (should show `succeeded via response_format` in logs) +5. **Run existing tests**: `cd backend && pytest app/test/test_phase5_chunk_highlight_service.py -v` +6. **Run existing tests**: `cd backend && pytest app/test/test_phase5_highlight_endpoints.py -v` +7. **Run existing tests**: `cd backend && pytest app/test/ -v` (full suite — ensure no regressions) + +--- + +## Research Evidence + +| Source | Finding | +|--------|---------| +| [vLLM structured outputs docs](https://docs.vllm.ai/en/stable/features/structured_outputs/) | `guided_json` removed in v0.12.0; migration to `structured_outputs` | +| [vLLM protocol.py](https://github.com/vllm-project/vllm/blob/master/vllm/entrypoints/openai/chat_completion/protocol.py) | `response_format.type == "json_schema"` → `StructuredOutputsParams(json=...)` since v0.6.4 | +| [vLLM PR #7654](https://github.com/vllm-project/vllm/pull/7654) | Added `json_schema` support (Aug 2024) | +| [vLLM PR #9530](https://github.com/vllm-project/vllm/pull/9530) | Fixed `json_schema` not being passed to backend (Oct 2024) | +| [vLLM RFC #19097](https://github.com/vllm-project/vllm/issues/19097) | Future direction: move all structured output to `response_format` | +| [vLLM test_chat.py](https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/openai/chat_completion/test_chat.py) | `extra_body=dict(structured_outputs={"json": schema})` extensively tested | +| [vLLM backend_xgrammar.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/structured_output/backend_xgrammar.py) | `$defs` NOT in unsupported features list; xgrammar handles them natively | +| [vLLM PR #15627](https://github.com/vllm-project/vllm/pull/15627) | `$defs` fix for Outlines backend; xgrammar/guidance already supported it | +| [encourage](https://github.com/uhh-hcds/encourage) | Real-world use: `extra_body={"structured_outputs": {"json": model.model_json_schema()}}` | +| [mellea](https://github.com/generative-computing/mellea) | Real-world use: falls back from `structured_outputs` to `guided_json` for older vLLM | + +--- + +## Related Plans + +- `.plans/debug_vllm_structured_output.md` — Previous vLLM structured output diagnosis (2026-04-29) that fixed the QueryDecomposer path but did not address the highlight service + +--- + +## Decision Points + +| # | Decision | Rationale | +|---|----------|-----------| +| 1 | **Replace `_complete_structured_vllm()` rather than adding fallback to ChunkHighlightService** | The root cause is in the LLM client layer. Fixing it there benefits ALL structured output callers, not just highlights. `response_format` is the recommended future direction per RFC #19097. | +| 2 | Use `response_format` as Tier 1, `structured_outputs` as Tier 2 | `response_format` is portable (works on OpenRouter/OpenAI/vLLM), future-proof (RFC #19097 direction), and has extra OpenAI-layer validation. `structured_outputs` is battle-tested fallback. | +| 3 | Remove `guided_json` entirely | Removed in v0.12.0 — confirmed dead code. No reason to keep it. | +| 4 | Remove `{**base_extra, **body}` merge | `chat_template_kwargs` is independent of structured output. Merging them risks param conflicts. `_build_extra_body()` remains for `complete()` text calls only. | +| 5 | Keep `_complete_structured_openai()` separate for OpenRouter | LangChain `with_structured_output(method="json_schema")` has zero real-world usage evidence with vLLM. Keeping separate paths avoids thinking-control incompatibility (`_get_langchain_model()`'s `model_kwargs=None` for vLLM). | +| 6 | Fix C (JSON fallback) remains as defense-in-depth | Even with the fixed vLLM path, a plain `complete()` fallback protects against server misconfiguration or backend issues. |