← Advanced course and all videos

Video 35 of 53 · 3:08 · Applications, benchmarking and evidence

Why there is no universal winner chart

In this video

  1. 0:00 · Introduction
  2. 0:14 · One formula can require different deliveries
  3. 1:05 · Core explanation
  4. 2:14 · Keep three kinds of absence distinct
  5. 2:56 · Direct takeaways

Caption transcript

Show transcript
This overview explains why method rankings require a common requested artifact and lifecycle. A fast scalar count and a complete truth vector answer different requests, so their bars cannot establish one shared winner. A point evaluation and a witness search can both use A or B, but they deliver different objects. The point query fixes the input and asks for one bit. The witness search asks the method to find an input making the formula true. A full vector supplies four outputs. Counting all satisfying assignments gives three. Counting distinct selected A values gives two. These are three more tasks, and a time from one should not be inserted into a ranking for another. Organize evidence by requested artifact and eligible method. A comparison is meaningful only after the methods agree on the delivered object, input universe, lifetime, and boundary. That grid is more informative than one universal winner label. A ranking needs a common job. Start with the answer required, then the workload, lifecycle, comparator and timed stages. A single podium often hides these conditions. The grid separates one-bit evaluation, full truth output, scalar count, and distinct projected count into different rows. Its columns separate cold construction, warm resident queries, and reload. The same engine can occupy different positions across this grid. The retained studies already include a CM kernel gain, an EPFL parity result, slower public wrappers and strong counting controls. A scalar count may avoid constructing the truth vector entirely. That is a legitimate advantage for a counting request, not proof that it won a full-output task. Preserve negative cases, refusals and unpromoted routes in the grid. A blank cell means the comparison has not been established. Unknown means there is no matching retained evidence for that cell. It does not mean the method is slow or incorrect. The result remains an open comparison. Refused means the method or API did not accept this input or contract. That may be an expected eligibility boundary. It is different from completing the correct job and losing on elapsed time. A measured loss means the method completed the same task correctly but had worse measured cost within the declared scope. Keeping unknown, refused, and measured loss separate prevents missing data from becoming a fabricated ranking. Match the deliverable, workload, and lifecycle before ranking methods. Keep negative results and unestablished comparisons visible.