How we verify
One machine, one protocol. Numbers we measured, numbers contributors measured on their own GPUs, and numbers the authors reported are stored and shown apart, and never mixed.
- 01
It has to run
A model is listed only if it has public code and public weights, and inference finishes on the whole shared test set at its default input size within 24 GB. If we had to patch something, the page says "runs with fixes" and links the patch.
- 02
Same inputs for everyone
Every model of a task runs on the same shared inputs, with one fixed setting per task. For vision tasks those are our own or CC0 photos and one score threshold. We look at every output and mark the failures by hand.
- 03
Timing
We time one call to predict(): preprocessing, the forward pass and postprocessing, which is what your code experiences. Batch 1, fp16 when the official code supports it, otherwise bf16 if it runs that, otherwise fp32. Three separate runs, each with 10 warm-ups and 100 timed calls; we show the median and p90 of the three, and discard a measurement whose runs disagree by more than 10% or that something else on the machine disturbed.
- 04
Memory
A background thread polls device memory every 10 ms or faster while the model loads and runs. VRAM peak is the device maximum minus the idle baseline taken before loading.
- 05
Accuracy
We score each checkpoint ourselves on the full standard benchmark, such as COCO val2017 for detection, with the reference scorer and at the same precision we time it. Our score is shown first; the paper's number for the same benchmark sits next to it, labelled. Benchmark images are never published, only the scores.
- 06
Licenses
The code license comes from the repository LICENSE file and the weights license from the model card, separately for each checkpoint, with the date we read it. The verdict is the strictest of the two. It is a summary, not legal advice.
Community results
The measuring tool is open. Anyone with an NVIDIA GPU on Linux can run the same protocol on their own machine and send the run record. It goes to a review inbox, and once we accept it, it shows on the model's page under "Community results", with the GPU, the contributor's GitHub name and the date.
- Community numbers never replace ours. Our RTX 4090 card, the task charts and the rankings use only what we measured.
- A result is Reproduced once a second contributor on the same GPU, size and precision agrees within 10%, the spread our protocol allows between runs. Until then it says "Unverified".
- Every record carries the tool's commit, the hashes of the adapter, its lock file, the input and the weights, and every timed call, so we can recompute its numbers and spot odd ones. That makes it checkable, not proven: we can't see the contributor's machine.
- A GPU running below its default power limit is marked "power-limited", because it will be slower.