Why rerank after retrieval?

Cross-encoder rerankers (Cohere Rerank 3, ColBERT, monoT5) score query-document pairs jointly, with full attention over both, which is much more accurate than bi-encoder similarity but ~1000x slower. The standard pipeline (2025-2026) is bi-encoder top-100 -> cross-encoder rerank -> top-5-10 to LLM. This recovers recall lost in fast retrieval without paying the latency cost. Empirical gains on BEIR, MIRAGE, RAGAS benchmarks are typically 5-20% in nDCG@10. Senior caveat: rerank quality is bounded by retrieval recall; if the ground-truth doc isn't in the top-100, reranking won't find it. So you tune recall@k first, then reranker. Late-interaction models like ColBERT retain per-token scores and let you trade off index size vs accuracy.