WEBVTT

1
00:00:00.000 --> 00:00:09.033
This overview explains how correctness checks, execution order,
clustered sampling, and uncertainty protect a benchmark.

2
00:00:09.033 --> 00:00:16.592
Precise timing cannot repair an incorrect output,
a weak comparator, or an invalid dependence model.

3
00:00:16.592 --> 00:00:16.600
Precise timing cannot repair an incorrect output,
a weak comparator, or an invalid dependence model.

4
00:00:16.900 --> 00:00:22.600
Suppose the required and vector is zero zero zero one, but a fast candidate produces zero

5
00:00:22.600 --> 00:00:24.300
zero one one.

6
00:00:24.300 --> 00:00:29.767
At input one zero, the candidate returns one where the rule requires zero.

7
00:00:29.767 --> 00:00:34.533
That mismatch is enough to reject its exactness claim.

8
00:00:34.533 --> 00:00:39.233
The invalid output cannot be celebrated as a faster exact solution.

9
00:00:39.233 --> 00:00:45.033
Keep its failed record in the audit trail, but separate it from valid timing comparisons.

10
00:00:45.033 --> 00:00:50.133
A timing improvement does not compensate for changing the answer.

11
00:00:50.133 --> 00:00:55.200
A matching output hash can confirm that two stored byte sequences match.

12
00:00:55.200 --> 00:01:00.600
It does not prove that the bytes represent the right variable order or the right problem.

13
00:01:00.600 --> 00:01:06.133
Check the contract and compare with an independent exact reference.

14
00:01:06.133 --> 00:01:06.143
Correctness, execution order, dependent
sampling and uncertainty are separate problems.

15
00:01:06.143 --> 00:01:13.433
Correctness, execution order, dependent
sampling and uncertainty are separate problems.

16
00:01:13.433 --> 00:01:17.567
A precise timer does not solve them all.

17
00:01:17.567 --> 00:01:23.167
First compare each arm's exact output with the frozen reference.

18
00:01:23.167 --> 00:01:27.800
A mismatched output is not an eligible speed result.

19
00:01:27.800 --> 00:01:34.200
Next counterbalance arm order so one arm
does not always inherit the warm position.

20
00:01:34.200 --> 00:01:39.933
Group repeated observations by the formula or circuit that generated them.

21
00:01:39.933 --> 00:01:45.433
Finally construct an interval with that cluster as the resampling unit.

22
00:01:45.433 --> 00:01:51.433
Ten repeated runs of one formula are still one formula identity.

23
00:01:51.433 --> 00:01:58.600
A digest binds an output to an identity
but is not an independent semantic oracle.

24
00:01:58.600 --> 00:02:04.833
A small-domain scalar check supplies
independent correctness evidence here.

25
00:02:04.833 --> 00:02:11.167
A within-run clustered interval does not
establish between-machine variation.

26
00:02:11.167 --> 00:02:17.395
Record invalid rows and refusals instead
of dropping them after seeing timings.

27
00:02:17.395 --> 00:02:17.400
Record invalid rows and refusals instead
of dropping them after seeing timings.

28
00:02:17.700 --> 00:02:22.633
If the first method always runs cold and the second always runs warm, order

29
00:02:22.633 --> 00:02:25.000
can bias the comparison.

30
00:02:25.000 --> 00:02:31.367
A protocol should declare how order is controlled or varied and what state is reset.

31
00:02:31.367 --> 00:02:37.133
The correct design depends on the intended cold or resident task.

32
00:02:37.133 --> 00:02:42.433
Repeated runs of the same formula are useful for timing variability, but they remain

33
00:02:42.433 --> 00:02:46.100
clustered around one input identity.

34
00:02:46.100 --> 00:02:53.467
An analysis that treats them as ten unrelated formulas exaggerates the range of evidence.

35
00:02:53.467 --> 00:02:59.400
Do not silently remove failed or refused rows and then describe the surviving timings as

36
00:02:59.400 --> 00:03:01.867
if every case succeeded.

37
00:03:01.867 --> 00:03:07.500
The audit should show the original population, the reasons for exclusions, and the precise

38
00:03:07.500 --> 00:03:11.533
set supporting the reported result.

39
00:03:11.533 --> 00:03:11.538
Check exact outputs and freeze the comparison protocol.

40
00:03:11.538 --> 00:03:16.267
Check exact outputs and freeze the comparison protocol.

41
00:03:16.267 --> 00:03:23.854
Keep dependent repeats within their formula
clusters and report refusals and invalid rows.