← All questions
HardTechnical screen30 min to answer

Evaluate LLMs on a toy task and use LLMs to generate additional evaluation data.

Asked atAnthropic
1Give yourself 30 minutes
2Answer out loud, not in your head
3Then compare with the answer below
Stuck? Show a way to structure it+
  1. 01Clarify objective, users, constraints, and failure tolerance.
  2. 02Cover task definition, synthetic data, contamination, measurement in a causal structure.
  3. 03Compare at least two credible approaches.
  4. 04Specify evaluation, rollout, monitoring, and rollback.

Reference answer

Then expect these follow-ups

  • Which assumption is most likely to invalidate your approach?

    Tests: depth

  • What would you measure offline and in production?

    Tests: evaluation

  • How would your answer change with one-tenth the data or compute?

    Tests: adaptability

Hiring now

Anthropic has 552 open roles right now

landed applies to roles like these for you, on each company’s own site, with answers written for the role.

Get 5x interviews10 free auto-applies. No card.
  • Applied AI Engineer, Commercial

    San Francisco, CA

  • Account Executive, Quantitative Trading & Investing

    London, UK

  • Strategic Pursuits Lead, DACH

    Munich, Germany

Or practice this question first