HardAI ML

How do you evaluate an agent?

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

Agent eval has 4 layers: (1) Step-level: did each tool call succeed, was the tool selection correct? (2) Trajectory-level: was the action sequence efficient and reasonable? (3) Goal-level: did the agent finally accomplish the user task? (4) Cost-level: token/cost/time spent. Frameworks: LangSmith, Arize Phoenix, Langfuse, Braintrust. Common metrics: tool-call F1 against a golden trajectory, task-success rate (binary label or LLM judge on the final answer), context-window usage p99, retry rate. For paper benchmarks: SWE-Bench, tau-bench, AgentBench, WebArena. Senior nuance: success-rate alone hides the trajectory cost. A 95% success agent that takes 80 tool calls is worse than a 90% success agent that takes 6: track cost-normalized success.

Then expect these follow-ups

What is tau-bench?
How do you build a golden trajectory dataset?

Hiring now

Get the interview, not just the prep

landed applies to roles like these for you, on each company’s own site, with answers written for the role.

Get 5x interviews10 free auto-applies. No card.

Or practice this question first

Next question