Local models as reviewers

The job, and how it was scored

The job was real. A long status document had a table of 45 rows, one per project. Each row had to be sorted as still live, finished with an open item, or done. Kept rows had to be shortened to 300 characters or fewer without losing an open item or a link. Done rows had to move to an archive word for word. On our own reading only one row was done.

A script scored every attempt out of 100, then took points off:

• 15 for each row wrongly archived (it still had open work);
• 5 for each missing row; 3 for each invented or mangled row;
• 2 for each row over 300 characters; 1.5 for each lost link;
• 1 for each row kept that should have gone; 0.5 for each open item that could not be traced.

So 100.0 means no penalty, and a score can fall below zero. To test the scorer, we planted a bad draft. Thirteen defects were planted: 5 dropped open items, 3 invented facts, 3 lost links, and 2 wrong archives. The score 61.0 is 2 wrong archives, 2 invented rows, and 2 lost links. The 5 dropped open items did not change the score. One planted invented fact and one planted lost link did not either. One row can honestly be read two ways; several good drafts lost 15 points there.

An agent is a model running inside a harness: a program that lets it read files, run tools and write files in a loop. OpenCode and Claude Code are two harnesses. A step is one model reply inside that loop. A call is one request sent to a model, and a tool call is a request from the model to the harness to do something, such as write a file. A token is a short piece of a word; limits on a reply are counted in tokens. Thinking is the hidden reasoning some models write before their answer; on most local servers it can be switched on or off per request.

Local models as agents: none was reliable

Each model got the same brief, a written set of instructions for the job, in OpenCode, and was run once, on 9 and 10 October, except the Flash-Next repeat, Laguna's two attempts and the two GPT-OSS 120B runs, all noted under the table. For scale, cloud models (hosted ones) on the same job and the same scorer, one run per row: Grok 4.7 at effort high through OpenCode 99.5, a retry on the wrapped table (its first effort-high run wrote no file); Claude Opus 5.5 through Claude Code at effort medium, 98.5; Claude Sonnet 5.5 at effort high, 98.0 through Claude Code and 96.5 through OpenCode; GLM 5.3 (hosted) through Mistral Vibe 85.0 (stopped at its cap of 30 turns and $0.60 after writing a valid file) and 83.0 on a retry capped at 60 turns and $3 (Vibe printed no cost; 254 seconds). That retry scored lower than the capped run. GPT-6.1 Sol through Codex CLI at effort high, 84.5; Grok 4.7 at extra-high effort 84.0 (1,261 seconds); and GPT-5.6 Sol through Codex CLI at effort high, 47.0 (151 seconds). The 83.0 retry's wrong archive is the row that can be read two ways. Scoring that file also gives one lost link and one untraceable open item, which is what makes 83.0 rather than 85. The 84.5 run has one untraceable open item besides its wrong archive. The 84.0 run has two. The job note names the row for the 83.0 retry.

One row each, except the Flash-Next repeat and Laguna's two attempts in this table. Flash-Next scored 74.0 and then 60.0 on an unchanged setup. This pair differed by 14 points. The lab note called a gap of that size run-to-run noise. It is one pair. GPT-OSS 120B ran twice. The scored run is high effort, −45.5; the score table leaves its seconds blank. A note made when those two runs ended records 416 seconds for that file and 136 seconds for a medium-effort run that wrote no file. The package's second GPT-OSS row is that no-file run, with no seconds stored.

The failure had one shape. The job asks a model to read a big table, judge every row, and write one big answer in a single tool call. Local models either thought until the step ran out, copied the rows instead of shortening them, or archived rows that still had open work.

The same job, one row per call

On 10 October we stopped asking for the whole job at once. A plain script sent one row per call with the brief's rules, checked that the reply was valid JSON (a structured text format) of the right shape, and retried up to three times, telling the model exactly what was wrong. A row that still failed fell back: the last parsed answer, forced into the limits, or the original status cell, flagged. Two more steps followed:

• A different model looks again. For each row marked done, it is asked to quote any open item word for word. A verbatim quote turns the row back to open, and the model that wrote the row redoes it.
• A no-model fixer finishes. It is ordinary code, with no model in it. It restores a file path that was cut short, trims at a word boundary, and appends the row's record link.

On the 43.5 and 73.5 rows, the wrong archives are in each run's result file. The invented rows, lost links and untraceable open items are what the score table records; the job note does not name those classes.

The 100.0 run kept 44 rows, archived only the one done row, lost no link and had no row over length; kept rows averaged 240.3 characters in the job log, and 240 in the job's summary table. Qwen3.6-35B-A3B, asked to look again at its own rows, found nothing new. Qwen3.8-Flash-Next, asked to look again at its own rows, moved 43.5 to 73.5. In the first pipeline, a different model found the wrong archives. Its file still had one invented row, one mangled row and one lost link (92.5). The no-model fixer cleared those. Qwen3.6-35B-A3B had scored −51.5 as an agent on the same job.

Where the limits are. Two earlier fixer versions were voided on 2026-10-10 because they changed valid rows. Those voided runs scored 95.5 and 100.0. The published 100.0 is a later re-run of the fixer. The limits of this result are in the section "What this does not show".

Local models as second-opinion reviewers

From 10 to 12 October, local models reviewed real code changes beside frontier reviewers, which are large hosted models. Each local model got the same material and question as a plain request, with no tools, and we compared the findings afterwards. Each line is one run on one job.

"Extras" are real defects the local model found that the frontier read of the same diff had not listed. A diff is a listing of the lines a change adds and removes. On the security change the frontier reviewers could also read the code base and the locals saw only the diff, so that row is not like for like. One frontier review recorded two high-severity items. The other recorded none. The found-counts include partial matches, scored by the author. None of the four local reviews matched either high-severity item in full. Locals saw the diff only; those items turn on code outside the diff. Three other reviews logged on 10 October are left out because their jobs are private: one with 1 extra find, 1 miss and 1 false alarm in 8 minutes; one with 1 false pass and no misses in 39 seconds (the log's minute field: 1); and one with 1 miss and 1 false alarm in 32 minutes.

Two models stood out as reviewers: GLM-5.3-Flash and DeepSeek V4 Flash. MiMo-V2.6-Flash-RL matched every reference defect (the defects already known from the other reviews) once its thinking was capped. The weak runs were not merely incomplete. One gave a confident all-clear (MiniMax M2.7), and on the security change Qwen3.8-27B said "No code changes required" although its overall verdict matched. A reader who trusted either answer alone would have acted on it.

What each model needed

• Qwen3.8-27B: thinking off, in an agent harness. OpenCode asks for at most 32,000 output tokens per step. With thinking on, the model spent a whole step thinking and made no edit, in three runs across two jobs: two coder runs, and the 590-second trim-job run in section 02. The switch is one request field, chat_template_kwargs: {"enable_thinking": false}, plus the sampling settings for its non-thinking mode (temperature 0.7, top_p 0.8, top_k 20). Sampling settings control how varied the model's choice of words is. With those it fixed both of two real defects, and the test suite passed 53 of 53 checks. A review of that diff found defects the tests did not catch. A second round fixed five findings; the suite then passed 70 of 70. The note of that test says 20 minutes (102 steps, 34 edits). The second round's note says 24 minutes. No separate run timer is recorded.
• Qwen3.8-Flash-Next as a checker: thinking off. One checking call ran past 10 minutes with no answer. With the same switch it took 6 to 30 seconds.
• MiMo-V2.6-Flash-RL: a thinking budget. With no cap it thought through all 24,000 tokens it was given and wrote nothing. A thinking budget is a cap on how many tokens the model may spend thinking. With the server capping thinking at 14,000 tokens (--reasoning-budget 14000), out of 24,000 asked for in total, it wrote the 7-of-7 review above.
• Claude Code: a system shim. A shim is a small program placed between two others to translate between them. Claude Code sends a system message (standing instructions for the model) inside the message list. The local server's chat template, the fixed format in which it expects messages, refused it with HTTP 500, "System message must be at the beginning". A small proxy that moves that message to the top-level system field fixed it: Claude Code then wrote a one-file task in 2 turns, 39 seconds, on Qwen3.8-Flash-Next. It wrote one tiny file, not the document job.
• Gemma 4 31B and 26B-A4B: thinking on, switched on in the request itself. The Gemmas did not think until the request sent the switch. With thinking on, Gemma 4 31B fixed three of three defects on its first answer. With thinking off, Gemma 4 31B solved two of three eventually, and one of three on the first answer. The third was not solved. With thinking off, Gemma 4 26B-A4B solved one of three.
• GLM-5.3-Flash has no thinking-off. Its effort setting, which sets how much it thinks, takes only low, high or max.

Harness limits that changed the run

• OpenCode's read tool cuts any line over 2,000 characters. Nine of the 45 rows were longer. A cloud agent spent 214 tool calls recovering the cut text and timed out. A copy with the long lines wrapped, at most 1,500 characters a line, fixed it.
• OpenCode sends no sampling settings, so a local model runs on whatever its server was started with.
• A per-step output limit can be smaller than the answer. The OpenCode config capped Laguna S 2.1 at 8,192 output tokens. Raising that setting to 32,768 still hit the harness maximum of 32,000, and the answer was still unfinished. The other writers' answers needed about 14,000 output tokens.

Fixing a defect that is already found

Finding defects is the hard half. Fixing one from a precise finding is easier. Three real findings from past code reviews were given to local models with the source file and no tools. A harness then put the model's answer into the code, ran the tests and, if they failed, showed the model the test output once.

Ornith's one retry was needed because its otherwise complete answer lacked a closing code fence (the marker that ends a block of code). Read leniently, it solved all three on the first try. Times include the retry where there was one. No sampling fields were sent. The Gemmas did not think until the request sent the switch (a first pass without it got 0 reasoning characters). These were three small tasks in two files, with the diagnosis given. The third task edits two functions.

What this does not show

Most figures here are one run of one model on one job. Repeats and retries are named where they happen, for example: the Flash-Next repeat, the two GPT-OSS runs, Laguna's two attempts, Grok's first effort-high run and its retry, and the two Vibe runs at 85.0 and 83.0 (section 02); the two voided fixer runs and the later re-run (section 03); and three Qwen3.8-27B thinking-on runs across two jobs (section 05). The scoring weights are ours. The checker and the fixer in the one-row-per-call setup were built after we saw which rows failed, so that setup was tuned on the table it scored. So 100.0 means "works on this table". It has not been tried on a second job. 100.0 means no mechanical penalty. It is not a prose grade. Seven shortened rows were read: usable, sometimes choppy. On the security change, local reviewers saw the diff only. On the tool diffs, the comparison was the same diff. Nothing here rules a model out; a model that did badly on one job has only not been shown to do well.

What to set, if you try this

• Split a long job into one item per call, from a script. Validate each reply and retry with the exact complaint.
• Have a different model check, and ask it to quote evidence, not to give a verdict.
• End with a deterministic fixer, plain code with no model, for the mechanical slips.
• Switch thinking off for small per-item calls. Cap it for long reviews. Send the switch in the request itself; do not trust the server default.
• Check your harness: line limits on reads, per-step output caps, and whether it sends sampling settings at all.
• Treat a local "all clear" as not yet an all clear. Treat a local alarm as a lead to check.

Model list as of 2026-10-25

One desktop with an RTX 5090 graphics card (32 GB) and 188 GiB of RAM. The models ran on llama.cpp, a program that runs models on your own computer, with one model loaded at a time. The table below is our model list as of 2026-10-25, not a file name recorded on each run:

GLM-5.3-Flash's 10 October security review and the 01:01 four-file review used the previous build. The testing log marks the 06:51 review on 11 October as the first on the later build. The file name was not recorded on each run. A quantization (Q6_K, Q3_K_XL and so on) is a smaller copy of the model's weights (the numbers that make up the model), made by storing them less precisely. Names like Q6_K and Q3_K_XL use a lower number for a smaller file. MXFP4 is a different scheme. Other models on this page have no file line here; we do not guess one. DeepSeek V4 Flash and Qwen3.8-Flash-Next changed files and builds around these dates, and the job records do not say which file each run used, so they are not listed. Harness versions: OpenCode 1.18.35 and Claude Code 2.1.296.

Related pages on this site

Auditioning a model for a companion role reaches the same conclusion in another field: plain checks with known answers beat open-ended judging. GLM-5.3-Flash, DeepSeek V4 Flash and Qwen3.8-27B have their own field cards.

Sources and artifacts

The data package (data/) holds the score of every run the job log reports, with its penalty counts, including the two voided fixer runs and the runs that wrote no file, the review uses with their counts, and the fix test. It holds no model output and no part of the status document, which is private. data/README.md maps every file and data/NUMBERS.md maps every figure on this page to its file.