Prompt vs fine-tune vs RAG for a search-and-answer system

Reference answer

Architect the funnel: cheap embedding retrieval (top-50) -> rerank with cross-encoder -> top-5 chunks -> prompt with citations. Use a small open model for embedding/cross-encoder (~400M params) and a 7-70B LLM for answer generation behind a cache. Cache retrieved chunks by query rewrite; cache answers for high-frequency queries. Choose prompt + RAG over fine-tune because the corpus is large, evolving, and attribution is needed. Only fine-tune if user eval turns up consistent voice/format issues. Track cost: tokens/call * $/token * MAU; expose a guardrail that defaults to short answers for low confidence.

Then expect these follow-ups

When to switch to a smaller fine-tuned model on a slice?

Next question