Propose a study that can fail informatively

A proposal interview tests whether you can turn an interesting question into useful experiments. Start with the problem and why the current evidence is insufficient. Then specify a hypothesis that can fail, a baseline, and the observations that distinguish success from alternative explanations.

Keep the first study small enough to execute and strong enough to answer a real question. A tiny experiment that cannot exhibit the proposed mechanism is not useful merely because it is cheap. A huge experiment with no discriminating control is not useful merely because it is impressive. Choose the scale where the necessary behavior can appear and the main confounds can be controlled.

Write a decision table before execution. If the method improves only the target slice, what does that suggest? If it improves every slice equally, could extra capacity explain the result? If it harms the target slice, what hypothesis is weakened? These predictions need not cover every possible outcome, but they should prevent the interpretation from changing opportunistically.

A proposal should include data provenance, evaluation isolation, resource accounting, and a plan for reporting failures. If the study needs human annotations, define the rubric and disagreement process. If it touches sensitive use cases, define the relevant safeguards and limits. Do not add broad claims about societal benefit without a path from the measured outcome to that benefit.

A fictional proposal asks whether a retrieval filter reduces irrelevant-context errors. The baseline uses unfiltered retrieved passages. All methods start from the same retrieved pool and use the same answer-generation limit. The candidate filters passages; a random-removal control retains the same number and matched total token length as the candidate. The unfiltered baseline may use more input tokens within a declared maximum, so its resource difference is reported. Questions are grouped by underlying task family before development and test separation.

The predicted pattern is candidate improvement over both baseline and random removal on distractor-heavy tasks, without a major loss on tasks needing multiple sources. If candidate and random removal perform similarly, less context may explain the gain. If the candidate loses on multi-source tasks, the filter may remove necessary evidence. The study reports all three outcomes rather than only the favorable slice.

Propose a first study for "Longer context improves answers". You can vary context length and whether added material is relevant. Include a control and an informative failure.

A strong design crosses short versus long context with relevant versus irrelevant additions, keeps model and answer rubric fixed, and evaluates held-out question families. An informative failure is worse performance with longer irrelevant context, which limits the claim that length alone helps. Award one point each for factors, controlled budget interpretation, valid split, predicted outcomes, and a stop rule. If longer context increases compute, report that tradeoff instead of pretending length changed without cost.

The retrieval-filter proposal needs separate comparisons for selection quality and context quantity. Candidate versus matched random removal addresses whether semantic selection helps beyond merely shortening the input. Candidate versus unfiltered context measures the combined effect of selection and the changed retained context, under the stated resource record.

These interpretations are hypotheses, not automatic causal proofs. The methods must see comparable question sets, retrieval pools, and answer-generation conditions. A filtering method that secretly sees the final answer label has an invalid advantage even if its retained token count is matched.

A proposal claims to improve handling of conflicting evidence but screens only on tasks with one unambiguous fact. A null result on that screen does not refute the claimed conflict-resolution mechanism because the necessary condition is absent. Conversely, success may measure ordinary extraction rather than conflict handling. The first study should include verified conflict cases and a no-conflict control, while remaining small enough to inspect.

1Minimum informative study:2task families: conflict, distractor, multi-source, simple extraction3development: tune filter only on designated families/examples4final evaluation: held-out structures and verified evidence requirements5primary comparison: candidate versus length-matched random removal6secondary comparison: unfiltered baseline with actual input cost reported7stop: reference or answer-rubric failure blocks method ranking8report: all conditions, failed runs, excluded cases and reasons

The primary comparison addresses semantic selection under matched retained length. The secondary comparison addresses a different practical choice. Keeping both is useful as long as their interpretations are not merged into one mechanism claim.

Two reviewers independently score forty proposed evaluation answers. They disagree on eight. The study author wants to keep only the thirty-two agreements to produce a clean benchmark. What should a better plan do?

First inspect whether disagreement reflects ambiguous questions, unclear rubric, or reviewer error. Use a documented adjudication process and preserve the original labels and reasons. If ambiguous cases are excluded, state the exclusion rule and how it changes the target population. Excluding difficult or ambiguous cases can make the final benchmark unlike intended use. Do not resolve disagreements by selecting the label that favors the candidate. Award one point for the ambiguity audit, one for independent/adjudicated records, one for consistent exclusions, and two for target-population and method-blinding concerns.

A revised rubric should be checked on development examples before final scoring. If final outcomes guide the rubric revision, acknowledge that the evaluation has become partly adaptive and seek a new confirmation set where needed.

“A small study is useful because it is cheap” fails when it cannot distinguish the hypothesis from alternatives. “A control is fair because its name says random” fails when it retains different token lengths, removes a different number of sources, or samples from a different pool. Inspect the actual intervention.

A strong proposal ends with branches rather than a promise of success. If the semantic filter matches random removal, investigate context quantity and stop claiming unique selection benefit. If it preserves needed sources and wins on independent structures, scale the study with a predeclared protocol. If answer verification fails, repair measurement before ranking methods. Each branch should retain enough budget and time to make its next result interpretable.

This makes failure informative. The project can discover that a simpler random reduction is sufficient, that a method harms multi-source tasks, or that the original benchmark never tested the target skill. Those outcomes can all improve the research decision without producing the hoped-for leaderboard gain.

Original practice: What is the cheapest experiment that could change your mind? A strong answer preserves the mechanism and controls while reducing unnecessary scale. Follow up with a result where random removal matches the new filter. A weak answer proposes a leaderboard run with no alternatives.