docs: add plan for configurable SubQuestions format
This commit is contained in:
Woody
parent
76c3bec2ab
commit
63e4c1a385
|
|
@ -0,0 +1,178 @@
|
|||
# Plan: Configurable SubQuestions via System Prompt Page
|
||||
|
||||
**Source**: User request (2026-05-04)
|
||||
**Scope**: Make SubQuestions `description` and `max_length` configurable from the system prompt page. Rename "Step 1: Query Decomposition" → "Step 1.1: Query Decomposition". Add new "Step 1.2: Query Decomposition Format" step storing JSON config.
|
||||
**Status**: Draft
|
||||
**Depends on**: Package 6 (LLMClientDP) — already implemented
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Allow users to configure `SubQuestions.description` (the format instruction for how sub-questions should be structured) and `SubQuestions.max_length` (1–5) from the system prompt configuration page, rather than hardcoding them in `backend/app/models/decompose.py`.
|
||||
|
||||
## Design
|
||||
|
||||
### Step Split
|
||||
|
||||
The current "Step 1: Query Decomposition" (`step_name="decompose"`) stores the prompt template (e.g. "Given this question... break it down..."). This becomes **Step 1.1**.
|
||||
|
||||
A new step **Step 1.2: Query Decomposition Format** (`step_name="decompose_format"`) stores a JSON configuration string:
|
||||
|
||||
```json
|
||||
{"description": "<the field description text>", "max_length": 3}
|
||||
```
|
||||
|
||||
The `prompt_template` column in `system_prompts` is reused — it's a `TEXT NOT NULL` field that can hold any string, including JSON.
|
||||
|
||||
### Dynamic Pydantic Model
|
||||
|
||||
At runtime, `QueryDecomposer.decompose()` reads `decompose_format` from `PromptService`, parses the JSON, and creates a Pydantic model dynamically using `pydantic.create_model()`:
|
||||
|
||||
```python
|
||||
from pydantic import create_model, Field
|
||||
|
||||
def create_subquestions_model(description: str, max_length: int) -> type[BaseModel]:
|
||||
return create_model(
|
||||
"SubQuestions",
|
||||
questions=(list[str], Field(description=description, min_length=1, max_length=max_length)),
|
||||
)
|
||||
```
|
||||
|
||||
The static `SubQuestions` model in `decompose.py` is kept as a fallback and for the `_BUILTIN_DECOMPOSE_TEMPLATE` path.
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
system_prompts DB
|
||||
├── step_name="decompose" → "Given this question: '{question}'..." (Step 1.1)
|
||||
└── step_name="decompose_format" → '{"description": "...", "max_length": 3}' (Step 1.2)
|
||||
|
||||
QueryDecomposer.decompose(question):
|
||||
template = prompt_service.get_prompt_template("decompose") # Step 1.1
|
||||
prompt = template.replace("{question}", question)
|
||||
|
||||
format_json = prompt_service.get_prompt_template("decompose_format") # Step 1.2
|
||||
format_config = json.loads(format_json)
|
||||
DynamicSubQuestions = create_subquestions_model(
|
||||
description=format_config["description"],
|
||||
max_length=format_config["max_length"],
|
||||
)
|
||||
|
||||
result = await llm_client.complete_structured(
|
||||
prompt=prompt,
|
||||
pydantic_model=DynamicSubQuestions,
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
### 1. `backend/app/core/sqlite_db.py`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| Add `_SEED_DECOMPOSE_FORMAT` | Default JSON string with current description + max_length=3 |
|
||||
| Update `_SEED_STEPS` | Add `"decompose_format"` |
|
||||
| Update `_SEED_TEMPLATES` | Add `"decompose_format": _SEED_DECOMPOSE_FORMAT` |
|
||||
| Update `_SEED_DECOMPOSE` | Change "2-5" → "1-3" (match current defaults) |
|
||||
|
||||
### 2. `backend/app/services/prompt_service.py`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| `_VALID_STEPS` | Add `"decompose_format"` |
|
||||
|
||||
### 3. `backend/app/routers/prompts.py`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| `_VALID_STEPS` | Add `"decompose_format"` |
|
||||
|
||||
### 4. `backend/app/models/decompose.py`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| Add `create_subquestions_model()` | Takes `description: str`, `max_length: int`, returns dynamic model |
|
||||
| Keep static `SubQuestions` | As fallback with current defaults |
|
||||
| Add `parse_decompose_format()` | Parses JSON string into `(description, max_length)` tuple with validation |
|
||||
|
||||
### 5. `backend/app/services/query_decomposer.py`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| Read `decompose_format` | From `prompt_service.get_prompt_template("decompose_format")` |
|
||||
| Create dynamic model | Call `create_subquestions_model()` with parsed config |
|
||||
| Fallback | If `decompose_format` unavailable or parse fails, use static `SubQuestions` |
|
||||
| `_BUILTIN_DECOMPOSE_TEMPLATE` | Update "2-5" → "1-3" |
|
||||
|
||||
### 6. `frontend/src/components/PromptEditor.tsx`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| `STEPS[0]` | `label`: "Step 1: Query Decomposition" → "Step 1.1: Query Decomposition" |
|
||||
| Add new entry | After `decompose`, insert `{ key: 'decompose_format', label: 'Step 1.2: Query Decomposition Format', placeholders: [], type: 'format_config' }` |
|
||||
| `VALID_PLACEHOLDERS` | Add `decompose_format: []` |
|
||||
| Render `decompose_format` differently | Two fields: **textarea** for `description` + **number input** for `max_length` (min=1, max=5). Combined to/from JSON `{"description": "...", "max_length": N}` on read/write. Other steps render as single textarea (unchanged). |
|
||||
| No auto-sync | Step 1.1 and Step 1.2 are independent — user edits both manually. No `{max_length}` placeholder in Step 1.1 template. |
|
||||
|
||||
### 7. `frontend/src/test/components/PromptEditor.test.tsx`
|
||||
|
||||
| Change | Detail |
|
||||
|--------|--------|
|
||||
| Update assertion | "Step 1: Query Decomposition" → "Step 1.1: Query Decomposition" |
|
||||
|
||||
### 8. Tests (new/updated)
|
||||
|
||||
| File | Type | Coverage |
|
||||
|------|------|----------|
|
||||
| `test_phase6_decompose_format.py` | Integration | Dynamic model creation, DB config read, fallback to static model |
|
||||
| Update `test_phase1_query.py` | Integration | Mock `decompose_format` step in mock prompt service |
|
||||
| Update `test_phase3_query_history_integration.py` | Integration | Mock `decompose_format` step |
|
||||
| Update `test_phase4_integration_query_pipeline.py` | Integration | Mock `decompose_format` step |
|
||||
|
||||
---
|
||||
|
||||
## DB Migration
|
||||
|
||||
No schema changes needed. The `decompose_format` step uses existing `system_prompts` table columns. The existing `seed_default_profiles()` backfill logic (`INSERT OR IGNORE`) will automatically add the new step to existing profiles on next startup.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] User can configure `description` and `max_length` on the system prompt page under "Step 1.2: Query Decomposition Format"
|
||||
- [ ] Changing the format JSON updates sub-question output format in real decompose calls
|
||||
- [ ] `max_length` correctly limits the number of sub-questions returned
|
||||
- [ ] Existing profiles (A/B/C) get the new `decompose_format` step with defaults on restart
|
||||
- [ ] Import/export includes `decompose_format`
|
||||
- [ ] Fallback to static `SubQuestions` when `decompose_format` is missing or invalid
|
||||
- [ ] All existing tests pass
|
||||
- [ ] Step 1 label renamed to "Step 1.1" in frontend
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
```
|
||||
Task A: Backend model + service (decompose.py, query_decomposer.py)
|
||||
↓
|
||||
Task B: DB seed + valid_steps (sqlite_db.py, prompt_service.py, prompts.py)
|
||||
↓
|
||||
Task C: Frontend (PromptEditor.tsx, tests)
|
||||
↓
|
||||
Task D: Update all integration tests
|
||||
```
|
||||
|
||||
Tasks B and C can run in parallel after Task A.
|
||||
|
||||
---
|
||||
|
||||
## Risk: Existing DB Profiles Missing `decompose_format`
|
||||
|
||||
Existing SQLite databases won't have the `decompose_format` row. Mitigation:
|
||||
1. The `seed_default_profiles()` backfill loop adds missing steps on startup
|
||||
2. `PromptService.get_prompt_template("decompose_format")` may raise `RuntimeError` if missing
|
||||
3. `QueryDecomposer` catches this and falls back to static `SubQuestions`
|
||||
4. A manual profile save from the frontend will write the new step
|
||||
Loading…
Reference in New Issue