← All questions
HardDataAI ML
An agent revision improves 180/200 task passes to 188/200 but creates three unauthorized writes. The old version had none in this set. Give a release decision and the smallest useful investigation.
1Give yourself 5 minutes
2Answer out loud, not in your head
3Then compare with the answer below
0
Reference answer
Then expect these follow-ups
What if all three failures share one tool version?
Hiring now
Get the interview, not just the prep
landed applies to roles like these for you, on each company’s own site, with answers written for the role.
Get 5x interviews10 free auto-applies. No card.
Next question
- How would you measure whether an LLM correctly executes a user’s requested actions?
- How would you launch an AI feature whose output cannot always be objectively graded?
- How would you evaluate a RAG pipeline end to end?
- Offline evaluations improved but production metrics fell. What happened?
- How would you build an evaluation dataset for an AI support agent?
- What is data leakage, and how can it invalidate an evaluation?