Lessons
1Diagnose a training slowdown across ranks25 min read
Locate a distributed training bottleneck from a timing trace.
- →Locate a distributed training bottleneck from a timing trace
2Design a checkpoint that can actually resume25 min read
Specify complete and verifiable training recovery state.
- →Specify complete and verifiable training recovery state
3Work a serving incident from observations25 min read
Prioritize measurements and reversible mitigations during inference overload.
- →Prioritize measurements and reversible mitigations during inference overload
- →Distinguish evidence from hypotheses in a performance incident
4Defend an inference platform design under failure25 min read
Build a capacity and recovery argument for a shared model service.
- →Build a capacity and recovery argument for a shared model service
- →Choose a failure-aware capacity plan
Skills in this course
- 01Locate a distributed training bottleneck from a timing traceLocate a distributed training bottleneck from a timing trace.
- 02Specify complete and verifiable training recovery stateSpecify complete and verifiable training recovery state.
- 03Prioritize measurements and reversible mitigations during inference overloadPrioritize measurements and reversible mitigations during inference overload.
- 04Build a capacity and recovery argument for a shared model serviceBuild a capacity and recovery argument for a shared model service.
- 05Distinguish evidence from hypotheses in a performance incidentDistinguish evidence from hypotheses in a performance incident.
- 06Choose a failure-aware capacity planChoose a failure-aware capacity plan.