Pair the evidence and respect the sampling unit

Two methods evaluated on the same tasks can be compared through paired differences. Pairing removes some variation caused by task difficulty because each task acts as its own comparison. But the unit of independence still matters. Ten answers from one user are not automatically ten independent users, and ten checkpoints from one training run are not ten independent training runs.

Choose the resampling unit to match the estimand and dependence structure. If the claim concerns variability across trained models, resample runs rather than individual predictions alone. If the data contains user clusters, a user-level procedure may be needed. A bootstrap that resamples dependent rows as if independent can produce intervals that are too narrow.

A permutation test also needs a valid null and exchangeability assumptions. In a paired sign-flip test, the signs of differences are randomized under a suitable symmetry or exchangeability condition. The resulting p-value describes how unusual the observed statistic is under that null procedure. It is not the probability that the research hypothesis is true.

Small samples limit resolution. With four paired observations, there are only sixteen possible sign assignments in an exact sign-flip enumeration. No amount of confident prose turns four independent units into forty. Report the sample size and the test's assumptions alongside the result.

The following invented paired differences are method B minus A on four independent experimental units.

The mean is 1.5. Under an exact sign-flip null with all sixteen assignments equally valid, only all-positive and all-negative assignments have an absolute mean at least 1.5. The two-sided p-value is 2/16, or 0.125. All observed differences favor B, but this small test does not meet a conventional 0.05 threshold.

This does not prove there is no effect. It shows that this design has limited evidence under the stated test. The effect estimate and its practical meaning remain useful, and additional independent units may improve precision.

A researcher evaluates two models on 100 messages from each of five users. They report n = 500 independent observations for a user-level claim. What should be checked?

Check within-user dependence and the target population. A user-level resampling or hierarchical analysis may be appropriate, and the effective number of independent users is only five for that target. Award one point each for identifying the cluster, linking unit to estimand, avoiding a false independence claim, and reporting the resulting uncertainty. Do not select a procedure solely because it gives a smaller p-value.

The four-unit example is small enough to inspect without an asymptotic approximation. Under the stated sign-flip null, keep absolute differences [1,2,1,2] fixed and enumerate all sixteen sign vectors.

1from itertools import product2magnitudes = [1, 2, 1, 2]3observed = 1.54null_means = [5    sum(s * d for s, d in zip(signs, magnitudes)) / 46    for signs in product([-1, 1], repeat=4)7]8p_two_sided = sum(abs(v) >= observed for v in null_means) / len(null_means)9# 2 / 16 = 0.125, for this complete exact enumeration and stated null.

This uses an absolute-mean tail definition. Other software conventions for a two-sided p-value can differ in discrete or asymmetric settings, so state the statistic and tail rule. The enumeration includes the observed assignment among all possibilities. It is different from drawing a limited Monte Carlo sample, for which the estimator and any plus-one convention need their own specification.

The sign-flip procedure is justified only when the sign assignments are valid under the null model or randomization design. Paired observations alone do not automatically establish that assumption. For observational algorithm scores, symmetry of paired errors around zero may be an assumption rather than a consequence of randomized assignment. State that assumption and consider whether the design supports it.

Two users produce different numbers of messages. User A has one hundred messages with a method difference of plus one on every message. User B has ten messages with a difference of negative one on every message. The message-weighted mean is ninety divided by one hundred ten, about 0.818. The equal-user mean is zero.

If the target is a randomly selected user, the equal-user quantity is relevant. If the target is a randomly selected message from this traffic mixture, the message weighting may be relevant, but dependence still affects uncertainty. Choosing the resampling unit and choosing the metric weights are related but distinct decisions. A cluster bootstrap can resample users while retaining their messages and recomputing the desired weighted statistic. It does not automatically force equal-user weighting unless the statistic does so.

There are fifty independent documents, with two model scores per document. The claim concerns the mean within-document score difference for this document population. Describe one paired bootstrap and a mistake it avoids.

Form the fifty paired differences or keep the score pairs intact. Draw document indices with replacement, use the same sampled indices for both methods, and calculate the mean difference for each replicate. Use an interval method appropriate to the analysis plan, noting that bootstrap intervals can behave poorly with very small or degenerate samples. Sampling method A's documents independently from method B's would destroy the observed pairing. Award two points for the unit and paired resampling, one for the statistic, and two for the interval assumptions and avoided error.

If those documents are actually chapters from five books, the independence claim needs review. A book-level cluster procedure or a model of the hierarchy may be more appropriate. Resampling fifty chapters as independent books does not create information about fifty independent source populations.

“A p-value of 0.125 means a 12.5% probability the null is true” reverses the conditional interpretation. The calculation concerns data extremeness under the specified null procedure. “A non-significant result proves equivalence” ignores effect size, uncertainty and power. Equivalence needs a defined margin and suitable analysis, not merely failure to reject zero difference.

For an interview, write the independent unit, pairing rule, statistic, null or sampling assumptions, and conclusion before discussing software. A correct library call with the wrong unit gives a precise answer to the wrong question. The small table is valuable because it lets you see the target and dependence before trusting a large output report.

Original practice: Why can paired evaluation be more informative than comparing two separate means? A strong answer uses shared task difficulty and then states the correct independence unit. Follow up with repeated outputs from one model run. A weak answer treats every logged row as independent.