Selection changes the meaning of the winning score

A validation score is partly signal and partly noise. When many candidates are tried, the best observed score tends to benefit from favorable noise. This happens even when every individual evaluation is correctly computed. Reporting the winning validation score as an unbiased estimate of future performance ignores the selection process.

Separate development from final estimation. Training fits parameters. Validation guides model and hyperparameter choices. A final test estimates the selected procedure after those choices are fixed. If you inspect the final test and change the method, that test has entered development. It can still provide information, but it no longer has the same untouched role.

Nested cross-validation formalizes this separation when data is limited. The inner loop selects settings. The outer loop evaluates the selection procedure on data not used by that inner selection. The outer score estimates the procedure, not one universally fixed hyperparameter configuration. After evaluation, the deployment model may be refit under a defined selection rule.

Multiple comparisons also affect scientific claims. Trying many endpoints, subgroups, and analysis variants creates opportunities for attractive findings. Predeclaring primary outcomes and reporting exploratory analyses helps distinguish confirmation from discovery. Statistical correction methods have assumptions and goals; they do not replace a transparent account of what was tried.

A teaching researcher tries twenty learning rates on one validation set. The winner scores 84%, while most score around 80%. They then claim the method achieves 84% on unseen data. That claim is not supported by the selected validation score alone.

A valid next step freezes the chosen training procedure and evaluates it on an untouched test set. Suppose it scores 80.5%. The report includes the selection process and test result. It does not hide the 84% development score if that history matters, but it clearly labels its role. If the researcher now changes the method based on test errors, a new confirmation design is needed for a fresh final claim.

A team checks test performance every day and chooses the best day's checkpoint. They call the test set "held out" because no gradient uses it. Explain the flaw.

The test influences checkpoint selection, so information from it guides the final model choice even without gradients. It functions as validation. Award one point each for recognizing selection leakage, distinguishing parameter fitting from model selection, relabelling the set honestly, and proposing a fresh evaluation or nested procedure. "No backpropagation on the test set" is necessary but insufficient for independent estimation.

A valid evaluation design tracks which decisions each dataset can influence. The following schematic describes nested selection in a limited dataset.

1Outer fold:2  outer-training data3    -> inner train/validation splits4    -> choose hyperparameters using only inner results5    -> refit chosen procedure on outer-training data6  untouched outer-test data7    -> estimate that fold's selected procedure89Repeat outer folds under the declared protocol.10Report outer estimates of the selection procedure.11Do not use outer-test scores to choose among procedures and still call them untouched.

The last line is easy to miss. If a team tries ten different search spaces and selects the one with the best outer score, outer data has participated in a higher-level selection. Nested cross-validation does not make unlimited adaptive choices unbiased. The information boundary applies to the whole procedure being selected.

Preprocessing belongs inside the relevant fitting boundary. A scaler fitted on all data before outer splitting leaks outer information even if hyperparameter selection is nested. Group and time constraints also apply to both levels when required by the deployment target. A nested random split cannot repair a target that requires future-time or unseen-group evaluation.

A researcher measures accuracy, calibration, latency, memory, and ten subgroup metrics, then writes the hypothesis around the one favorable result. The individual statistic may be correctly calculated, but the report's confirmatory framing is misleading. It should describe the analysis as exploratory and preserve the set of investigated outcomes. A follow-up study can predefine the promising endpoint.

Multiple-testing procedures can control particular error criteria under assumptions, but they are not a universal cure for unrecorded adaptive research. The lesson's main control is procedural transparency: identify what was tried and which evidence remains independent of the final choice.

A team uses five outer folds. In each outer-training set it chooses among four learning rates using inner cross-validation, then scores the refitted winner on that outer test. Three folds select rate A and two select rate B. What does the combined outer result estimate?

It estimates the specified training-and-selection procedure across the outer partitions, not the performance of a globally fixed rate A chosen afterward because it won most often. The deployed model can apply the defined selection rule to all permitted development data and refit, but its exact parameter choice is a new fitted instance. If the team changes the rule to “always choose A” after seeing outer results, distinguish that new procedure and its evidence. Award two points for the target procedure, one for rejecting a fixed-rate interpretation, and two for the final refit and changed-rule boundary.

“No gradients touch the test set, so it is untouched” ignores checkpoint, feature and endpoint selection. “Nested cross-validation is inherently pessimistic” mistakes correction of selection optimism for a directional bias guarantee. Estimates still have sampling variation and depend on the design.

A clean final record names the candidate family, search budget, stopping rules, score used for selection, data identities, and excluded or failed candidates. The best observed development score can be useful as development history, but it should not replace independent estimation. If no independent evaluation remains, report the limitation directly and avoid presenting a selection score as a confirmed generalization result.

This distinction also protects negative results. If a promising candidate was selected from many trials and then failed independent evaluation, the failed confirmation is informative. Deleting it and returning to the next best development winner without tracking the sequence repeats the same selection problem.

Original practice: Why does nested cross-validation sometimes give a lower score than ordinary tuning on the same folds? A strong answer explains selection optimism and the outer evaluation of the full selection procedure. Follow up with daily test inspection. A weak answer blames nested CV for being pessimistic by definition.