MediumData

Prompt vs fine-tune vs RAG for a search-and-answer system

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

Architect the funnel: cheap embedding retrieval (top-50) -> rerank with cross-encoder -> top-5 chunks -> prompt with citations. Use a small open model for embedding/cross-encoder (~400M params) and a 7-70B LLM for answer generation behind a cache. Cache retrieved chunks by query rewrite; cache answers for high-frequency queries. Choose prompt + RAG over fine-tune because the corpus is large, evolving, and attribution is needed. Only fine-tune if user eval turns up consistent voice/format issues. Track cost: tokens/call * $/token * MAU; expose a guardrail that defaults to short answers for low confidence.

Then expect these follow-ups

When to switch to a smaller fine-tuned model on a slice?

Hiring now

Get the interview, not just the prep

landed applies to roles like these for you, on each company’s own site, with answers written for the role.

Get 5x interviews10 free auto-applies. No card.

Or practice this question first

Next question