How Graphometer measures
Graphometer measures large language models on hardware you can own, and publishes what the records support, nothing more. This page is the method behind every other page: what the labels mean, what counts as evidence, what happens when we get something wrong, which machine the numbers come from, and why you can check all of it yourself.
The rules
Every figure on this site carries a label naming its basis, and the labels have fixed meanings.
| measured | From a recorded run on our machine, with the date given. |
|---|---|
| vendor | The maker's claim, printed as the maker's claim, never restated as ours. |
| arithmetic | Reasoning from file sizes and headers that we consider sound but have not run. |
| stated | A value a maker states that we did not measure. It is marked in amber wherever it appears, like this row's name. |
| transferred | A number measured in one configuration and carried, labeled, into advice about a different one. |
Behind every measured number is a recorded run: a dated file the run itself produced, kept, so that every figure has a home that can be produced if questioned. Where no such record exists, no number appears. The Workbench states the rule for its own context meter, and it is the house rule everywhere: a number we cannot stand behind is a number we do not display.
What counts as evidence
Our studies ship their evidence as a downloadable data package: the
files the runs actually produced, not a summary. In each package,
data/README.md maps every file, states its provenance, and
states the redactions plainly, and data/NUMBERS.md maps
every figure on the page to its file. The standing offer printed on
those pages holds here too: if a number on a page disagrees with a file
in its package, the file is right and the page is wrong;
tell us and we will fix the page.
Packages ship today with Empty Answer, Slower by default, The wrong flag, Reasoning effort is a sentence, not a dial, the Qwen3.8-27B day-zero field card, and the Muse Glimmer 30B field card.
Corrections, published in full
This site publishes corrections of our own records as a genre. The worked example is The wrong flag: for six days our records blamed warmup for a decode-speed gap on Mistral Large 3 675B, and a pre-registered three-arm A/B then refuted our own attribution and traced the gap to llama.cpp's default thread count instead. The page does not quietly fix the records. It names what we got wrong and why the wrong story survived, and the superseded entries are annotated rather than rewritten. A correction published in full is worth more than a claim that was never tested.
The bench
Every local number on this site comes from one desktop computer: one NVIDIA GeForce RTX 5090 with 32,607 MiB of VRAM, an Intel Core Ultra 9 285K (24 cores: 8 performance + 16 efficiency, no hyperthreading), 188 GiB of system RAM, and the weights on a consumer 4 TB NVMe SSD.
One machine is a boundary, and we treat it as one: speeds are field measurements, one configuration each, not benchmarks, and they do not transfer across configurations, which is rather the point of measuring.
No scripts, no tracking, no cookies
This site is static. Not one script tag exists on any page, the styles are inline in the page itself, the fonts are the ones your system already has, and your browser makes no request to anyone's server but this one.
You do not have to take that on trust, which is the point: open any page's source and look. A site about verifiable numbers should itself be verifiable in under a minute.