Lessons
1Ablations need controls that answer the explanation35 min read
Use a two-factor comparison to identify an interaction.
- →Design ablations that distinguish mechanisms from confounds
- →Use factorial comparisons to detect interactions
2Equal steps may not mean a fair resource comparison35 min read
Select and report a resource comparison that matches the claim.
- →Compare methods under explicit resource constraints
3Keep the evaluation separate from the method's memory35 min read
Identify contamination paths and design a cleaner benchmark.
- →Construct an evaluation resistant to contamination and leakage
4A negative result can sharpen the next question35 min read
Distinguish absence of evidence, practical equivalence, and a failed mechanism.
- →Interpret a negative result and choose a next study
- →Document limitations and excluded runs transparently
Skills in this course
- 01Design ablations that distinguish mechanisms from confoundsDesign ablations that distinguish mechanisms from confounds.
- 02Compare methods under explicit resource constraintsCompare methods under explicit resource constraints.
- 03Construct an evaluation resistant to contamination and leakageConstruct an evaluation resistant to contamination and leakage.
- 04Interpret a negative result and choose a next studyInterpret a negative result and choose a next study.
- 05Use factorial comparisons to detect interactionsUse factorial comparisons to detect interactions.
- 06Document limitations and excluded runs transparentlyDocument limitations and excluded runs transparently.