legco_ai_assistant/development_plan.md

RAG Video Q&A Web Application - Development Plan

Project Overview
Web-based application built in two phases.

• Phase 1: Text question → RAG retrieval → Point-form answer (strictly from database)
• Phase 2: Video upload + player → real-time audio streaming → ASR transcription → question extraction → Phase 1 RAG flow

Tech Stack

• Backend: Python + FastAPI (REST + WebSocket)

• Frontend: TypeScript + React 18 (Vite) + shadcn/ui + Tailwind CSS

• Server: Linux Ubuntu 22.04

• RAG Database: ChromaDB (persistent)

• LLM/ASR Integration: Dynamic via .env (supports local vLLM, OpenRouter, Alibaba Cloud)

  • Alibaba Cloud reference: https://modelstudio.console.alibabacloud.com/ap-southeast-1?switchAgent=101503&tab=doc&productCode=p_efm&switchUserType=3#/doc/?type=model&url=2989727

• Models:

  • Embedding: qwen/qwen3-embedding-4b
  • LLM: qwen/qwen3.5-35b-a3b
  • ASR: Qwen/Qwen3-ASR-1.7B

Deployment

• Development: Simple commands (uvicorn + npm run dev)
• Production: Docker + Nginx

---

Project Structure (Monorepo)

app/ ├── backend/ # FastAPI │ ├── app/ │ │ ├── main.py │ │ ├── routers/ # query.py, ingest.py, video.py, ws_asr.py │ │ ├── services/ # rag.py, llm_client.py, asr_client.py, video_service.py │ │ ├── models/ # Pydantic schemas │ │ ├── core/ # config.py, database.py │ │ └── utils/ # chunking, metadata extraction │ ├── uploads/ # video storage (max 300MB) │ ├── requirements.txt │ └── .env.example ├── frontend/ # React + TypeScript (Vite) │ ├── src/ │ │ ├── components/ │ │ ├── pages/ │ │ ├── lib/ # api.ts │ │ └── App.tsx │ ├── package.json │ └── vite.config.ts ├── chroma_db/ # Persistent vector store ├── Dockerfile ├── docker-compose.yml ├── nginx.conf └── deploy.sh

---

Key Requirements Incorporated

• LLM/ASR Configuration: Backend reads from .env for easy switching between development (OpenRouter / Alibaba Cloud) and production (local vLLM).
• RAG Database: ChromaDB with metadata support (filename + extracted content metadata).
• Embedding Model: qwen/qwen3-embedding-4b via sentence-transformers.
• Document Ingestion: Via UI (project-based demo, no user authentication).
• Video: MP4 and common formats, maximum 300MB.
• ASR Flow: Both automatic (on transcript updates) and manual “Ask from Video” button.
• UI Layout:
  • Top-Left: Video player
  • Top-Right: Real-time transcript + text input box
  • Bottom Half: RAG response (bullet points with source metadata)
• Authentication: Public demo (no login required).
• Mobile: Not required at this stage.

---

Phase 1: Text Question → RAG → Point-Form Answer (5-7 days)

Backend (FastAPI)

• Dynamic configuration via .env (LLM base URL, API key, model names).
• services/rag.py: Persistent ChromaDB + Qwen embedding + metadata extraction (filename, upload date, content summary).
• services/llm_client.py: OpenAI-compatible client for Qwen LLM with strict RAG prompt (only use retrieved context).
• Endpoints:
  • POST /api/v1/ingest – Document upload and ingestion with metadata.
  • POST /api/v1/query – Question → retrieve → LLM → bullet-point response.

Frontend (React + TS)

• Clean layout: Top-right input box, bottom response area.
• Type-safe API calls using TanStack Query.
• Display answer as clean bullet list with source metadata.

---

Phase 2: Video Upload + Real-Time ASR → RAG (8-10 days)

Backend Additions

• Video upload (POST /api/v1/upload-video) with size/format validation (<300MB).
• Static file serving for videos.
• WebSocket /ws/asr/{video_id} for real-time audio chunk streaming.
• ASR integration with Qwen/Qwen3-ASR-1.7B (file upload or audio content).
• Question extraction via LLM, then trigger Phase 1 RAG (auto + manual support).

Frontend Additions

• Drag & drop video upload + progress.
• Video player (<video controls>).
• Live transcript display (scrolling box).
• Top-Left: Video player | Top-Right: Live transcript + manual input.
• Bottom: RAG response panel.
• Support both automatic “Ask” on transcript updates and manual button.

---

Development Timeline

Phase	Duration	Key Deliverables
Setup + Phase 1 Backend	3-4 days	FastAPI + Chroma + Metadata + LLM client
Phase 1 Frontend	2-3 days	UI layout + text query flow
Phase 2 Backend	4-5 days	Video upload + WebSocket ASR + question extraction
Phase 2 Frontend	3-4 days	Video player + live transcript + auto/manual flow
Testing & Polish	1-2 days	End-to-end testing + deployment scripts

Total Estimated Effort: 13-17 developer days (2-3 weeks)

---

Deployment Strategy

Development:

• Backend: cd backend && uvicorn app.main:app --reload --port 8000
• Frontend: cd frontend && npm run dev

Production:

• Use docker-compose up -d (includes backend, built frontend, Nginx reverse proxy).
• Simple deploy.sh script for building and restarting.

---

File Information

• Filename: development_plan.md
• Last Updated: April 2026
• Status: Ready for implementation