Lessons
1Optimize the critical path you actually measured35 min read
Use timing evidence and Amdahl's law to choose an optimization.
- →Use a trace to identify the measured bottleneck
- →Report throughput under a fixed quality protocol
- →Separate overlap from additive timing
2Memory accounting before memory tricks35 min read
Build a first-order memory budget and choose a targeted mitigation.
- →Estimate memory before selecting a mitigation
3Mixed precision changes arithmetic, so test the update35 min read
Explain loss scaling and the correct order for gradient clipping.
- →Preserve optimization semantics with mixed precision
- →Report throughput under a fixed quality protocol
4Distributed training must preserve the intended average35 min read
Calculate the difference between rank means and a sample-weighted mean.
- →Check distributed gradient and sample-count semantics
Skills in this course
- 01Use a trace to identify the measured bottleneckUse a trace to identify the measured bottleneck.
- 02Estimate memory before selecting a mitigationEstimate memory before selecting a mitigation.
- 03Preserve optimization semantics with mixed precisionPreserve optimization semantics with mixed precision.
- 04Check distributed gradient and sample-count semanticsCheck distributed gradient and sample-count semantics.
- 05Report throughput under a fixed quality protocolReport throughput under a fixed quality protocol.
- 06Separate overlap from additive timingSeparate overlap from additive timing.