Video 27 of 53 · 3:33 · Applications, benchmarking and evidence
The boundaries of a timing result
In this video
Caption transcript
Show transcript
This overview explains how preparation, kernel execution, wrappers, and delivery contribute to a completed request.
A faster kernel can belong to a slower overall path when costs outside the timer are omitted.
Use abstract cost units.
A request takes one unit to parse, eight to prepare, one to execute, and
two to deliver its output.
The complete total is twelve.
The numbers are a worked accounting example, not a measurement from the research results.
An execution-only timer reports one.
A timer including preparation and execution reports nine.
The complete request reports twelve.
These numbers can all be correct while answering different questions.
A ratio is uninterpretable unless both methods use the same boundary.
For a control with costs one, two, three, and two, the full total is eight.
The candidate wins the execution-only comparison, one against three, but loses the
complete request, twelve against eight.
No contradiction exists once the boundaries are visible.
A completed request runs from source preparation through delivery of the answer.
Compilation prepares an artifact; the kernel evaluates it; wrappers, extraction and delivery add other work.
These timer brackets specify which costs are counted: preparation, the kernel, or the complete request.
The following costs are invented teaching values, not benchmark measurements.
Method M takes 8 milliseconds to prepare and 1 per evaluation.
Method C takes 2 to prepare and 2 per evaluation.
One completed request takes 9 for M and 4 for C, even though M's kernel is faster.
With 10 evaluations and preparation charged once, the totals become 18 and 22.
Those totals describe only this toy lifecycle.
Cold construction, a warm resident artifact, retained state across queries and reload in a new process each impose different obligations.
Complete vectors, scalar counts and projected counts also need separate endpoints.
Later retained measurements carry their own boundaries.
Return to the opening four-stage ledger, using abstract cost units.
Candidate parsing costs one, preparation eight, execution one per use, and delivery two per use.
The control costs one, two, three and two respectively.
With one prepared object and ten uses, the totals are one plus eight plus ten times three, or thirty-nine, and one plus two plus ten times five, or fifty-three.
These costs differ from the intervening eighteen-versus-twenty-two example.
Ten independent cold requests cost one hundred twenty and eighty instead.
The reuse assumptions changed the answer.
Calling both experiments ten runs without describing object lifetime would hide the reason for the difference.
Report the output contract, residency, and timer boundary together.
Repeated measurement does not itself imply reuse, and reuse does not imply that output delivery has disappeared.
delivery has disappeared.
State exactly where the timer starts and stops.
Compare identical delivered outputs under the same cold, warm, resident, or reload lifecycle.

