← Advanced course and all videos

Video 39 of 53 · 3:06 · Applications, benchmarking and evidence

Preserving the correction trail

In this video

  1. 0:00 · Introduction
  2. 0:13 · Correct the claim that depended on the baseline
  3. 1:10 · Core explanation
  4. 2:15 · A new task creates a new claim
  5. 2:55 · Direct takeaways

Caption transcript

Show transcript
This overview explains how a corrected comparison changes the claims supported by historical evidence. A later improvement on a different task cannot retroactively validate an earlier mismatched comparison. Imagine a candidate taking six units against a weak ten-unit control. The observed comparison favors the candidate. These fictional values illustrate how a correction changes a claim; they do not replace retained research measurements. A stronger control completes the same required task in five units. The candidate still produces the same exact answer, but its earlier advantage disappears. The relevant performance claim must change because its comparison premise changed. Keep the mathematical identity, the implementation description, and the speed claim as separate cards. A baseline correction can invalidate the speed comparison without disproving a separately checked identity. It also does not establish that the implementation is otherwise flawless. Keep the old evidence and mark the wording it no longer supports. A correction creates a traceable successor; it does not make history disappear. The audit replaced a weak generic comparator story with sharing-aware CSE-flat, checked exact outputs and execution order, and separated kernel from wrapper. The corrected B2/B4 compiled-kernel advantage survived in its stated scope. EPFL remained near parity against CSE-flat. The wrapper comparison remained slower. Three conclusions remain separate: a scoped B2/B4 kernel advantage, EPFL near parity, and a slower public wrapper. The withdrawn wrapper crossover and dramatic mismatched-task SymPy claim must stay withdrawn. Later counting or packed-mask improvements answer their own questions; they do not retroactively validate an unsupported historical headline. A claim registry needs both supersession edges and the retained source identities. Suppose the corrected result concerned a fresh full-vector request. Its conclusion belongs to that contract, including the preparation and output work the caller required. A later scalar evaluation experiment can be valuable, but it asks for one answer rather than all outputs. A win on that new task cannot retroactively restore the old full-vector claim. Preserve the old record, attach the correction, and state the new task as a separate experiment. This gives the reader a trail they can follow without forcing them to infer which conclusions survived. Preserve the old evidence and its supersession record. Support a new claim with a new matched comparison rather than reviving a withdrawn headline.