← Advanced course and all videos

Video 31 of 53 · 4:32 · Applications, benchmarking and evidence

What replication adds

In this video

  1. 0:00 · Introduction
  2. 0:18 · Count independent questions, not just rows
  3. 1:10 · Keep formula weighting distinct from row weighting
  4. 2:21 · Core explanation
  5. 3:34 · Make replication reproduce the actual job
  6. 4:22 · Direct takeaways

Caption transcript

Show transcript
This overview explains how the retained B2/B4 cloud runs extend the evidence across separately recorded environments. Observed ranges across machines do not automatically form a confidence interval or replace the original aggregation design. Suppose one formula is timed one hundred times. Those rows can describe timing variability for that formula and setup. They do not create one hundred independently selected formulas or establish coverage of an entire input family. Now run the same frozen job on three hosts and imagine ratios zero point eight, one, and one point two. The host transfer changes the observed result. These three points are illustrative host results, not automatically an uncertainty interval with a declared statistical interpretation. Within-host repetitions and across-host changes address different questions. A statistical analysis must specify what its units represent. Collapsing the two axes into one pile of rows can make evidence look broader than it is. Replication asks whether an observation persists under separately recorded environments. Formula balancing gives each formula equal influence despite different numbers of repeated timing rows. Row weighting gives every timing row equal influence. Consider a teaching example: formula X has ratio zero point five measured three times, and formula Y has ratio one point five measured once. For an arithmetic mean in this teaching example, the four rows average to zero point seven five. Giving each of the two formulas equal weight gives one. Repeating X changes the row-weighted result but not the equal-formula result. These teaching arithmetic means are not a reconstruction of the historical estimator. Formula-cluster uncertainty keeps each formula and its repeated rows together when resampling. It describes formula variation within the recorded run, not a guarantee across machines. The companion links the exact retained aggregation and ratio protocol; use that protocol for the historical numbers. Replication asks whether an observation persists under separately recorded environments. These pod summaries use row weighting, whereas the local headline uses formula balancing. The aggregation procedures differ. The retained three-pod bare CM-over-CSE-flat time ratios span 0.9026 to 0.9126 overall. At support 16 they span 0.9746 to 0.9766. Each value belongs to its own recorded machine and run. Each of the three estimates retains its own machine and run identity, under the compiled-kernel timing boundary. These are descriptive replication values, not endpoints of a predeclared pooled confidence interval. They support the scoped direction on those runs. They neither erase wrapper overhead nor establish that the current implementation will repeat them. These are historical retained runs, rather than a new benchmark. A replication should keep the relevant job identity explicit: source, inputs, dependencies, wrapper, output contract, and timed boundary. A change in one of these can turn replication into a different experiment. Check that both hosts produce the required exact outputs before interpreting their timing ratios. A faster run that omitted output work or changed the assignment universe is not a successful replication of the original task. A matching replication strengthens evidence about the declared task and reveals sensitivity to the tested environments. It does not by itself establish a result for unrelated formula families or for a different requested artifact. Retain each run's scope and weighting method. Separate descriptive replication ranges from inferential uncertainty.