Scaling RAG past a million documents is less about the model and more about the data pipeline. Most of the wins came from decisions made before inference.
Fixed token chunks break reasoning. I moved to segmenting by semantic structure — sections, lists and tables — attaching the parent heading to each chunk.
BM25 alone misses synonyms; vectors alone miss exact terms. Combining both with re-ranking solved most hard cases.
Retrieval quality sets the ceiling for answer quality. No prompt saves a bad context.
With pgvector and tuned HNSW indexes, I kept p95 under 400ms even at peak.