Lesson 6 of 8 · 50 min

Caching, pagination, and hot-path performance

Cache-aside and invalidation, stampede control, N+1 elimination, keyset pagination, HTTP caching altitude, and LLM-related cache choices.

Lesson 6 · Caching & perf

Cache-aside, invalidation, N+1, cursor pagination

Speed is a correctness budget under load

Caching without invalidation design is lying to users with confidence. N+1 queries pass unit tests and die in production. Pagination choices decide whether list endpoints scale. This lesson is hot-path performance for backends that serve product APIs and AI features with heavy metadata reads.
Measure first: p50/p95/p99 latency, DB time vs app time, cache hit ratio, rows examined. Premature Redis is theater if the real bug is an N+1 or missing index. Senior answers pick the bottleneck with evidence.

Cache-aside (lazy loading)

Read path: get from cache → miss → load DB → set cache with TTL. Write path: update DB → delete (invalidate) cache key (common) or update carefully. Cache-aside keeps the DB as source of truth. TTL bounds staleness when invalidation misses.
python
1def get_project(project_id):2	key = f'project:{project_id}'3	cached = redis.get(key)4	if cached is not None:5		return deserialize(cached)6	project = db.fetch_project(project_id)7	if project is None:8		return None9	redis.setex(key, 60, serialize(project))  # TTL seconds10	return project1112def update_project(project_id, fields):13	db.update_project(project_id, fields)14	redis.delete(f'project:{project_id}')  # invalidate

Invalidation is the hard part

Key design encodes invalidation: user:{id} is easy; feed:{user}:{page} explodes. Prefer primary-key caches + cheap recompute for lists, or version stamps (user:{id}:v{n}). Stampede: on expiry of hot keys, single-flight / mutex so one DB load fills many waiters.

What not to cache

Per-user secrets with shared keys, unbounded keys from raw user input (cache poisoning / memory blowup), and data that must be read-your-writes without versioning. Negative caching (cache 404s briefly) helps storm of missing ids but needs short TTL.

N+1 queries

ORM classic: load 50 runs, then each touches run.model → 50 extra queries. Fix: join/eager load, or batch WHERE id IN (...). In GraphQL, DataLoader. In hand-rolled SQL, select what the list view needs in one query. Detect with query counts in tests.
sql
1-- N+1 (bad): 1 + N queries2SELECT * FROM runs WHERE tenant_id = $1 LIMIT 50;3-- then per row: SELECT * FROM models WHERE id = $run.model_id;45-- Fixed: join or two-step batch6SELECT r.*, m.name AS model_name7FROM runs r8JOIN models m ON m.id = r.model_id9WHERE r.tenant_id = $110ORDER BY r.created_at DESC11LIMIT 50;

Cursor pagination performance

Keyset/cursor: WHERE (created_at, id) < ($c_ts, $c_id) ORDER BY created_at DESC, id DESC LIMIT n with a matching index. Avoid OFFSET 100000 — databases still walk rows. Stable sort ties need the id tiebreaker.
sql
1-- Keyset pagination (Postgres)2SELECT *3FROM llm_runs4WHERE tenant_id = $15  AND (created_at, id) < ($cursor_ts, $cursor_id)6ORDER BY created_at DESC, id DESC7LIMIT 50;8-- Index: (tenant_id, created_at DESC, id DESC)

HTTP caching (brief)

For public GETs: ETag/If-None-Match, Cache-Control. Private authenticated data: usually Cache-Control: private, no-store at the browser; use server-side Redis instead. CDNs help public docs, not tenant-private runs.

LLM hot paths

Cache embeddings for identical normalized text when product-safe. Cache auth session and feature flags. Do not cache non-deterministic completions as if they were immutable without an explicit content hash key. Prompt templates can be cached; user-specific PII outputs usually should not live in shared caches.
Embedding caches need a normalization contract (unicode, whitespace) or you thrash on trivial variants. Version the embedding model in the cache key — mixing vectors from two models is a silent product bug. Invalidate or namespace by model id on upgrades.

Connection pools and timeouts

API p99 often dies on pool exhaustion, not missing Redis. Size DB pools per instance × instances against Postgres max_connections. Set statement timeouts so one bad query cannot hold workers forever. Cache cannot fix a pool leak — measure wait time on acquiring connections.

Payload size and serialization cost

Returning 2MB JSON lists will dominate CPU and egress even with perfect SQL. Project columns for list views; put large artifacts behind separate GET. Prefer compact ids and avoid nesting huge blobs. Compression helps wire size; it does not remove parse cost on the client or server.

Read replicas — use carefully

Replicas scale reads but introduce lag. After a write, reading your own update from a replica can show stale data — route read-your-writes to primary or use session consistency tokens. Do not put strongly consistent billing reads on a lagging replica without saying so.

Interview answers — caching & perf

  1. 01Q: Cache-aside vs write-through? Aside is default for app objects; write-through keeps cache warm on write at higher write latency; pick via read/write ratio and staleness.
  2. 02Q: Invalidation strategy? Delete-on-write for PK keys; TTL safety net; versioned keys for complex graphs; avoid undelimited key explosion.
  3. 03Q: Stampede? Single-flight lock, probabilistic early refresh, or slightly staggered TTLs on hot keys.
  4. 04Q: Find N+1? Query counters in tests, APM spans, log SQL count per request id in staging.
  5. 05Q: OFFSET vs cursor? Cursor/keyset for large data; OFFSET only for small admin pages.
  6. 06Q: What to measure? Hit ratio, p99, DB time, rows examined, error rate after cache put — not only average latency.
  7. 07Q: Cache poisoning? Do not key raw untrusted input without normalization/allowlists; cap key cardinality.
  8. 08Q: Read-your-writes? Invalidate primary keys on write, sticky primary reads, or version tokens — pure TTL cache may show stale self-view.
  9. 09Q: Redis failure mode? Degrade to DB with load shed; do not let cache outage become full outage without a plan.
  10. 10Q: When not to cache? Tiny QPS already under budget; highly mutable strongly consistent paths where complexity exceeds gain.
docsAWS — Caching best practicesAWSdocsPostgreSQL — IndexesPostgreSQLdocsUse the Index, Luke — SQL performance explaineduse-the-index-lukedocsMDN — HTTP cachingMDN

Checkpoint

After updating a project name, users still see the old name for minutes. Cache-aside was used with TTL 10m and no invalidation on write. Fix?

AIncrease TTL to 24h so the cache is “more stable.”BDelete/update the project cache key in the write path; keep TTL as a safety bound.CRemove the database and serve only Redis as system of record.
Sign up free to answer and see why

Checkpoint

List endpoint does one query for runs then per-run query for owner email. 50 runs → 51 queries. Name and fix?

AN+1 query pattern — join or batch-load owners in one additional query / eager load.BCache miss storm — only Redis can fix relational fan-out.CCursor pagination bug — switch to OFFSET 0 always.
Sign up free to answer and see why

Checkpoint

Why add id as a tiebreaker in keyset pagination ordered by created_at?

ABecause UUIDs are required by the HTTP specification for all lists.BEqual timestamps would otherwise skip/duplicate rows across pages; (created_at, id) makes a total order.CSo clients can sort client-side without server ORDER BY.
Sign up free to answer and see why

Checkpoint

Hot key expires; 500 app servers stampede the DB. Mitigation?

ASingle-flight / request coalescing so one load refills cache while others wait; optional early refresh.BDisable the database when Redis expires so traffic stops.CSet TTL to 0 so keys never exist and stampedes cannot occur.
Sign up free to answer and see why

Checkpoint

Which data is a poor candidate for a shared Redis cache without careful design?

APublic documentation HTML with long Cache-Control freshness.BPer-tenant API responses keyed only by path without tenant id in the key.CFeature flag config documents with short TTL and versioned keys.
Sign up free to answer and see why

Can you design cache keys, invalidation, and a keyset list query for a multi-tenant API?

New to itGetting thereConfident

Takeaways

  • Cache-aside + delete-on-write + TTL safety net is the default story.
  • Invalidation and key design dominate cache correctness.
  • Kill N+1 with joins/batches; measure query counts.
  • Keyset pagination + proper indexes scales lists; OFFSET does not.

Next: logs, metrics, traces, SLOs, and on-call for backends.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with EnzoSign up free