Lesson 1 of 4 · 35 min

Optimize the critical path you actually measured

Use timing evidence and Amdahl's law to choose an optimization.

A training loop contains data loading, transfer, forward computation, backward computation, synchronization, and parameter updates. Some stages overlap. Others wait on each other. A list of operator durations is not automatically a wall-clock timeline because nested operations and concurrent work can be counted more than once.

Measure a representative steady-state window. Warm-up, compilation, cache population, and initial data loading can dominate the first steps. Report whether these costs are included. For a short experiment, startup may matter to total time; for a long run, steady-state throughput may matter more. Neither metric is universally correct. Match the measurement to the workload being optimized.

Use profiler ranges to identify phases and inspect both CPU and accelerator activity. A GPU can be idle because the CPU is preparing data, because a transfer is blocking, or because a synchronization call waits for earlier work. Low utilization alone does not distinguish these causes. Change one suspected bottleneck and remeasure the whole step.

Amdahl's law provides a useful bound. If fraction f of serial wall time becomes s times faster, ideal total speedup is one divided by 1 minus f plus f divided by s. The calculation assumes the rest of the system is unchanged and the fractions describe non-overlapping time. It is a ceiling for the simplified model, not a replacement for a trace.

Worked example

This invented serial step takes 100 ms.

Phase	Time
Data wait	40 ms
Transfer	10 ms
Forward/backward	40 ms
Update	10 ms

Halving compute reduces the step to 80 ms, a 1.25x speedup. Eliminating half the data wait also gives 80 ms. Making compute infinitely fast cannot beat 60 ms, or 1.67x speedup, under this serial model. The result suggests that a heroic kernel optimization may have limited value while data waiting remains large.

Now suppose transfer overlaps compute in a real trace. Adding their durations would overstate wall time. The measured critical path, not this toy table, must guide the final conclusion. The table teaches the calculation; the profiler establishes the actual execution relationship.

Exercise and solution

A step has 30 ms non-optimizable work and 70 ms computation. A proposed kernel makes computation twice as fast. Calculate the ideal new time and speedup. Then name a reason the measured gain may be smaller.

The new time is 65 ms and speedup is about 1.538x. Possible limits include overhead added by the kernel, a changed bottleneck, or incorrect attribution of overlapping time. Award one point each for time, speedup, a concrete measurement limit, and rechecking end-to-end throughput.

Lab artifact: an overlapping timeline

The serial calculation is useful only when its intervals do not overlap. Consider this invented steady-state trace for one step. All times use the same clock boundary, and the accelerator work is synchronized at the end for elapsed-time measurement.

code

1time in ms:     0       10      20      30      40      502CPU prepare:   [----------20-----------]3H2D transfer:          [----10----]4GPU compute:                  [------------30-------------]5CPU logging:                     [---8---]6step end:                                                   50

The exact brackets are illustrative; the defined intervals are CPU preparation [0,20], transfer [10,20], GPU compute [20,50], and CPU logging [25,33]. Summing all recorded durations gives sixty-eight milliseconds, while elapsed time is fifty. The logging interval is not on this step's critical path as drawn. Optimizing it from eight milliseconds to four would not necessarily shorten the step. It might still matter across steps if it blocks a later operation, so inspect the longer trace before concluding it never matters.

A profiler table's inclusive CPU time can include child operations, while device time may overlap other streams. Read the tool's column meanings. A long operator duration is a hypothesis generator, not proof of the largest reducible end-to-end delay. Draw dependencies and inspect gaps between kernels as well as kernel duration.

A second failure case: asynchronous timing

Suppose a host clock measures only the time to enqueue an accelerator kernel. It reports one millisecond, while the actual device operation takes twenty. Comparing that enqueue interval with a synchronized twenty-millisecond baseline produces a false twentyfold speedup. Timing boundaries must represent the same completed work.

python

1# Conceptual accelerator timing protocol2warm_up_with_representative_inputs()3synchronize_device()4start = monotonic_clock()5for batch in fixed_measurement_batches:6    run_complete_training_step(batch)7synchronize_device()8elapsed = monotonic_clock() - start9# Report completed steps / elapsed, plus workload and startup policy.

This protocol measures a bounded group of steps and includes host overhead inside the interval. It is conceptual, not a substitute for the framework's event-timing API. Device events may answer a narrower question about device execution. Choose the boundary based on the claim and report it.

Exercise: startup can reverse the winner

Method A compiles for thirty seconds and then takes one second per step. Method B has no startup and takes 1.2 seconds per step. Ignore all other costs. For fifty steps, A takes eighty seconds and B takes sixty, so B finishes sooner. For one thousand steps, A takes 1,030 seconds and B takes 1,200, so A wins. Solve the crossover: 30 + n = 1.2n, giving n = 150 steps.

Award one point for each total, one for crossover, and two for explaining why the reported metric must match run length. If the intended use restarts frequently, amortized throughput from a long warm run can be misleading. If compilation artifacts are safely reused, the effective startup boundary changes and should be recorded.

Misconceptions to correct

“Highest accelerator utilization means fastest useful training” fails when the device does redundant work, processes extra padding, or uses an objective that needs more updates. “Adding operator times gives wall time” fails with nested and overlapping work. Use end-to-end completed work and inspect phase attribution as supporting evidence.

Repeat measurements under controlled workload and report a distribution, not only the fastest observed step. Keep batch shapes, sequence lengths, precision, and completed updates fixed for a local performance comparison. For a scientific time-to-quality comparison, also report whether the optimized procedure reaches the same target and how the target was selected. A fast step that changes convergence is a different tradeoff, not automatically an improvement.

The final interview answer should name one proposed intervention, the trace evidence that makes it plausible, and the end-to-end measurement that could disprove its benefit. This turns profiler reading into an experiment rather than a list of slow operators.

Interview probe

Original practice: The largest operator becomes twice as fast, but training improves only 10%. Why? A strong answer inspects its fraction of the critical path, overlap, overhead, and the new bottleneck. Follow up with startup cost. A weak answer assumes operator speedup must equal whole-run speedup.

Sources

docsPyTorch: profiler guidedocs.pytorch.org docsLLNL: parallel computing and Amdahl's lawhpc.llnl.gov

Checkpoint

A serial 100 ms step has 40 ms compute; compute duration halves. Ideal new step time?

A50 msB60 msC80 msD40 ms

Checkpoint

Preparation [0,20], transfer [10,20], compute [20,50], logging [25,33] share a clock. Elapsed step time is?

A68 msB50 msC58 msD30 ms

Checkpoint

A host timer stops after queuing a device kernel. What does that interval establish?

ACompleted device execution time.BEnd-to-end training throughput.COnly enqueue-side elapsed time unless completion is synchronized or otherwise measured.DA valid comparison with any synchronized baseline.

Checkpoint

A needs 30 s startup and 1 s/step; B needs no startup and 1.2 s/step. At 50 steps, which is faster?

AB: 60 s versus A: 80 s.BA: 50 s versus B: 60 s.CThey tie at every run length.DA because steady-state throughput alone determines total time.

Checkpoint

A logging interval lies entirely inside device compute and blocks no later step. What should an optimization claim say?

AHalving logging necessarily halves step time.BLogging consumes no resources.CIts duration should be added again after compute.DA local logging reduction may leave this critical path unchanged; remeasure the full workload.

Can you draw the elapsed-time boundary, distinguish overlap, and calculate a proposed speedup? Rate confidence from 1 to 5 and state a measurement that could disprove the proposed gain.

Not yetGetting thereConfident

Wrap-up

Measure representative execution and optimize the critical path. Keep operator speedup separate from end-to-end gain.

Sources

Free to read · better with Enzo

Learn it with Enzo

Save your progress, answer the checkpoints, and let Enzo quiz you on what you just read.

Continue with Enzo Sign up free