Lessons
1Reproduction triage: find the first disagreement35 min read
Build a comparison ladder for a failed reproduction.
- →Localize a failed reproduction with controlled comparisons
- →Use matched controls to isolate implementation changes
2Optimize a benchmark without changing the problem35 min read
Define protected correctness conditions for a performance task.
- →Protect benchmark correctness during optimization
3Spend experiment budget on discriminating tests35 min read
Choose runs that distinguish competing explanations.
- →Allocate a limited experiment budget to informative runs
4Explain your contribution and the claim's boundary35 min read
Present an engineering result with measured evidence and honest limits.
- →Communicate engineering evidence without overstating a research claim
- →Use matched controls to isolate implementation changes
- →Preserve an audit trail of failed and excluded runs
Skills in this course
- 01Localize a failed reproduction with controlled comparisonsLocalize a failed reproduction with controlled comparisons.
- 02Protect benchmark correctness during optimizationProtect benchmark correctness during optimization.
- 03Allocate a limited experiment budget to informative runsAllocate a limited experiment budget to informative runs.
- 04Communicate engineering evidence without overstating a research claimCommunicate engineering evidence without overstating a research claim.
- 05Use matched controls to isolate implementation changesUse matched controls to isolate implementation changes.
- 06Preserve an audit trail of failed and excluded runsPreserve an audit trail of failed and excluded runs.