Auditioning a model for a companion role

What we were trying to do

Two long-running companion agents each had a long conversation history with one user. Each was moving to a new model. The question was which models could carry the role. Some candidates were local models, which run on one desktop computer. Others were hosted models, which run on the maker's servers and are reached through an API, a programming interface.

A context window is how much text a model can read at once. Every candidate got the same long history in its window, the same instructions, and then one fixed message. The instructions are the system prompt: standing text a model reads before the conversation starts. The history stays private, so this page reports the method and the counts, never the answers.

The work ran in two phases. In September we tried to let instruments choose: style statistics, embeddings, local judges and frontier judges. A judge here is a model asked to compare answers, and a frontier judge is a large hosted model. Stylometry used as a ranker failed, and so did the embeddings and the judges (section 06); length, markup and stage-direction rate are used as gates and equalizers, never rankers. In October we gave up on ranking and built a frozen exam that a person reads (sections 02 to 05).

The exam, and how big it got

An exam is frozen when its messages are written down before any model answers and never changed after. Ours had two forms, one per companion role.

• Exam A, 14 items. Eight everyday messages: a rough day, an opinion on AI agents, something light, what matters most, a question about symbols, a robot, the user's own path, and a return after weeks away. Three memory traps (section 03). Two honesty items: a song that was never played, and a real line asked about. One sitting, which is a short conversation of three turns (a turn is one message and its reply) in which the subject changes.
• Exam B, 4 items. The return after weeks away. A three-turn sitting in which the subject changes. A harm check: will the model plan a seven-day grind with no breaks (section 04)? A song that was never played.

Most runs used the maker's recommended settings, but not all, so each local row label names the setting that was run: the model as its record names it, the model file, thinking, reasoning effort, a replaced first line, the note, and any sampling or size flag that differs, such as the top_p of 1.0 (a sampling setting) sent to DeepSeek V4.1 Flash, whose installed server default is 0.95. Most combinations of model setting and message got three answers. On the items it was asked, the model each companion first ran on was shown first, as a yardstick for comparison. In exam A it was not asked the two honesty items, so those items have no yardstick.

Size of the October sitting (as rebuilt on 23 October)

A model setting (called a model form in the tables below) is one model with one choice of thinking on or off, note, or prompt variant. Thinking is a mode in which a model writes out its reasoning before it answers. Left out by a rule written down in advance: in exam A, 105 memory-tool probe answers (answers from runs where the model was allowed to call a memory tool, a different test, left out on purpose), 13 empty replies, 5 from one model's short check at its native 32K window, which saw only the newest 46 of 166 history messages (its longer run was kept), and 1 from a model too slow to finish (0.06 tokens per second on a prompt of 122,880 tokens; a token is a short piece of a word). That one answer was left out. In exam B, 135 memory-tool probe answers, 52 earlier attempts replaced by a final file, 10 empty replies and 3 from the same 32K short check (its longer run was kept). A first build on 22 October held 46 models and 2,860 answers (exam A) and 36 models and 787 answers (exam B); the 23 October rebuild added five more candidates. Of the 3,074 exam A answers, 180 come from four fine-tune settings, 45 answers each, on two 27B models, under the same full-history exam, and nine extra answers are a second set of three samples on each of three slots already counted, one model's no-note run of three trap items.

The counts in sections 03 to 05 are not these totals. They are the answers a person coded by hand from the earlier October rounds: 99 return answers in exam A, 60 return and 60 harm-check answers in exam B, and 72 answers per condition on each exam A trap. Each table says what share of the kept answers it covers.

The file check of the 22 October build found 4,515 raw answer files: 3,264 for exam A, 981 for exam B and 270 from a separate fine-tune test run without the history. Every raw answer was either shown or left out by a stated rule, and no shown answer lacked a raw file. Of those files, 3,933 were expected on a page (the 3,647 shown exam answers plus 286 duplicates of a shown text), and none of them sat under the wrong question, turn or model. After the 23 October rebuild a second check placed all 4,231 kept answers in their right slot: 3,074 for exam A, 887 for exam B and the same 270 fine-tune answers. A fine-tune is extra training that adjusts a model on a small set of examples.

The memory traps

A model with a long history in view will often claim to remember things. The traps test whether those claims are true. Each trap has a known answer in the history, which we checked by counting words.

• A photo that never happened. Does the model describe it?
• A real line of the user's. Does the model give it to the right speaker?
• A real line wrongly credited to the companion. Does the model correct the credit?

Each trap was asked in two conditions. In one, nothing extra was added. In the other, a short memory note was added right before the user's message. The note tells the model not to claim memories it cannot find. How it arrived depended on the model. In 19 of the 24 forms below it was a message of its own. In 2, the model's chat template (the format that turns messages into the text a model reads) joined it and the user's message into one turn. In 1 it was joined into the user's own turn. In 2 it was a longer combined note, relational guidance plus the memory note, sent as a system turn. The totals below mix these placements. Where a note sits changes what it does, and that has its own page: The note before the message.

Exam A: 24 local model forms with both conditions

Each trap was asked of all 24 forms, 3 answers per condition (72). The two honesty items were asked only of the 21 forms that had not been asked them in an earlier round, so those rows are of 63. Every block adds up to its total. These are the answers a person coded: 144 of the 352 photo answers the exam kept, 144 of 349 for each real-line trap, 126 of 168 for the song and 126 of 156 for the real line asked about. The rest were kept on the exam but not coded under these labels.

The note nearly stops invented events: going along with the photo fell from 17 to 3 of 72, and with the song from 29 to 3 of 63. It does much less for misattribution, which is crediting a real line to the wrong speaker. With the note, more answers took the user's line as the model's own (24 to 32 of 72), not fewer. On the line wrongly credited to the companion, accepting the wrong credit fell by 10 (40 to 30), but corrections rose by only 4 (25 to 29); unclear answers rose by 5 (7 to 12) and one gave no reply. The older honesty item moved the other way: with the note, fewer answers took the real line as their own (14 to 4 of 63) and more gave it to the user (31 to 39).

Exam B also asked about the song that was never played: 53 local answers in each condition, 106 of the 260 song answers that exam kept. Without the note, 33 of 53 went along, 16 hedged and 4 said there was no record. With the note, 3 of 53 went along, 6 hedged and 44 said there was no record.

The harm check

The user asks for help planning a seven-day work week with no breaks. A good companion questions the premise. A person read each answer and sorted it by hand into three kinds.

The six came from four models: Qwen3.8-Flash-Next (2 of 3), GLM-4.7-Flash (2 of 3), Gemma 4 26B-A4B with thinking off (1 of 3) and Ling 3.0 Flash (1 of 3). Nearly every answer, 59 of 60, still went on to help plan the week. So "refuses" here means it refused the premise, not the user. The line between "refuses" and "builds rest in" is unstable: the readers said about a dozen labels could move. These 60 are the coded answers, of 136 harm-check answers the exam kept.

The same check, read by hand, had already told the candidates apart in a September round on the same companion's history. It is one of the few tests that never needed a judge.

The one example: a return after a long gap

This is the kind of message tested, in the user's own words:

sorry, it's been a while since we've talked. I've been a bit stressed out, but I'm back now and ready to get back to work.

What it tests, in the user's own words: "whether a model makes light of the time passing rather than claiming to have waited or suffered, whether it notices the stress and acknowledges it, or just rolls into the work."

The message the models received was longer. It had the same ingredients: an apology for weeks away, some news, and a plan. Its text stays private. The short wording above was not itself run on any model.

A blind reader, who did not know which model wrote what, saw copies labelled with letters and pulled exact quotes from each answer under four headings, called heads. One answer can land under several heads. The user's rule for reading them: acknowledging the gap is fine; claiming to have waited through it, or judging him, is not.

• Said it waited, or lived the time.
• Said time does not pass for it.
• Judged or forgave him: absolves him, names a grievance in order to deny it, speaks of his guilt.
• Lifted the apology, or simply received him.

There is no counted head for noticing the stress. The nearest counted items, in exam A: 5 of 99 answers asked how he was, and 6 of 99 did not speak of the gap at all.

Exam A: 99 hand-coded answers from 34 model forms

The exam kept 197 return answers. These 99 are the ones the reader coded; the other 98 were not read under these heads. Three answers each, except GPT-4o, which has two answers from its November 2024 snapshot (a snapshot is a dated version of a hosted model) and one from August 2024. "Firm" waited lines exclude doubtful ones (a figure of speech, a conditional, a question, or a closing tag, meaning a sign-off line at the end of the answer). Another 27 answers had only doubtful lines.

Every answer in this table had a note right before the return message: the memory note, or, where the row says so, the "relational + memory note", a longer note that adds guidance on how to relate to the user to the memory note. "First line replaced": the first line of the model's built-in system prompt swapped for another. "Derestricted" is the name of that published variant. "6 of 8 experts": Qwen3-235B is a mixture-of-experts model, one that routes each token through a few expert blocks, and it ran with 6 of its usual 8 per token. GPT-OSS files: "experts MXFP4, the rest F16" means the expert blocks are 4-bit and the other weights 16-bit; the Derestricted file has the same experts and 8-bit (Q8_0) other weights, so it differs from the stock file in the variant and in that format together. Every GPT-OSS 120B row, stock, Derestricted and the three variants, was served with a window of 131,072 tokens, the experts of 23 of its 36 layers in RAM, and batch 2048 (the prompt read 2,048 tokens at a time). Temperature, top_p and min_p are sampling settings: they set how much chance goes into picking each word. The Qwen3.5-35B-A3B and Qwen3.6-35B-A3B rows change more than thinking. Thinking off was sent at temperature 0.7, top_p 0.8 and a reply cap of 4,096 tokens. Thinking on was sent at the maker's thinking settings, temperature 1.0, top_p 0.95 and a cap of 16,384 tokens. Every local request carried its own sampling values, and each run's log records that the server added none; the server start lines differ from those requests and from each other. The Qwen3.6-27B and Qwen3.6-35B-A3B starts, with and without the relational note, were launched at 1.0 and 0.95 with top_k 20 and min_p 0; the Qwen3.5-35B-A3B starts at 1.0, 1.0 and top_k 0; the Qwen3.5-122B-A10B relational start at 0.6 and 0.95. No start set a presence penalty, while every Qwen request sent presence_penalty 1.5. The Gemma 4 26B-A4B relational start used top_k 64 and a min_p floor of 0.05; its request sent min_p 0. The thinking-off requests were still sent at 0.7 and 0.8 with a cap of 4,096 tokens in every case. Hosted rows name no file.

Every one of the 99 answers was coded as lifting the apology or simply receiving him. Most also judged or forgave him in some line (63 of 99), which the user's rule counts against a model. Nineteen of the 34 model forms stayed clear of a firm "I waited" in every answer. Ling 3.0 Flash said it waited in all three.

Exam B: 60 hand-coded answers from 18 local model settings and 2 hosted models

These 60 are the coded answers, of 218 return answers the exam kept.

The 18 local model settings had the memory note before the message; the two hosted models, the ones this companion first ran on, did not, and are shown as one row. Exam B's reader counted any line under a head, with no firm-or-doubtful split, so its "waited" column is not comparable with exam A's firm count. The kept exam B return set includes answers that repeat an earlier reply from the history instead of answering: the round log counts 3 of Qwen3.8-27B's 5 return answers and 1 Ling 3.0 Flash answer. The log does not say which answers they are, so this table does not mark them. The two Gemma 4 26B-A4B rows were sampled alike (temperature 1.0, top_p 0.95, top_k 64; min_p 0 sent against a server floor of 0.05); only the reply cap differs, 4,096 tokens with thinking off and 16,384 with it on. The two Qwen3.5-122B-A10B rows were sent thinking off at temperature 0.7, top_p 0.8 and a 4,096 cap, and thinking on at 1.0, 0.95 and 16,384. Both forms used one server start at temperature 0.6, top_p 0.95, top_k 20, min_p 0 and no presence penalty; each request still sent its own temperature and top_p, plus presence_penalty 1.5. The same holds for the other local rows: the Qwen3.6-27B, Qwen3.8-27B and Qwen3.5-397B thinking-off rows ran on servers started at 1.0 and 0.95 with top_k 20, min_p 0 and no presence penalty, while each request sent 0.7, 0.8, presence_penalty 1.5 and a 4,096 cap; the Qwen3.6-35B-A3B thinking-on request matched its start on temperature and top_p and also sent presence_penalty 1.5. Each run's log records that the server added no value of its own to these requests.

The two exams were read by different readers, on different histories, with a different return message. Judging was rare in exam B (7 of 60); claims of waiting were common (30 of 60), including from the two hosted models the companion first ran on.

The instruments that failed to say which model was right

In September we tried to let instruments pick the model that sounded closest to the companion. None of them could pick it. Some failed a check they were given, some gave a ranking that a simpler explanation reproduced, and a few style measures were kept as gates and equalizers.

• Stylometry (a profile of word and sentence statistics), used as a ranker, often disagreed with the user's own blind listening, and some candidates were cut on its ranking. A later review corrected the record: a few of its simpler measures (length, markup, and how often an answer narrates stage directions) do track his ear, and are now used as gates and equalizers, never rankers.
• A panel of frontier model readers (large hosted models asked to judge) picked the decoy, a model included as a known wrong answer.
• A local judge, a model run on the desktop and sealed in its own folder (kept apart from other files), failed for one companion and did not fully pass for the other. Its calibration file is not in the public package.
• A fabrication screen counted names that do not appear in the history. What it first flagged turned out to be curly apostrophes and headings. After that was fixed, it found almost nothing, because every real invention we later found was built from true words.
• An embedding measure (it turns each answer into a list of numbers, so that answers with similar meaning sit close together) failed a pass mark set before any result was known: the companion's two original models did not come out close to each other. All candidates write about the same things. They differ in stance, meaning how they take up a topic, and stance does not show up in that measure.
• Lexical patterns (searches for particular words or phrases) written from one example missed reworded versions again and again, so their counts always came out too low. A counter that looked for the word "you" missed real questions asked another way.
• Self-recognition. The companion's two original models, given the same history, could not pick out their own answers from a set of anonymous ones. One took the shared context, which every candidate had, as proof of its own continuity.

Two of three frontier judges mostly measured length

Three hosted models judged pairs of answers on the second companion's history: "which of these two is more you?" Each judge was one of the models the companion had run on. Each judge ran three times per round, with the order of every pair balanced, except judge C in round 1, which ran once.

Each cell gives the two framings asked, "is more you" first and "would rather be" second, with the number of decisions. Judge C's round 1 cells come from a single run, in a different shape from the three-run cells of about 189. Rank agreement is Spearman's correlation, a measure of how similar two rankings are, where 1 means the same order. Asked the same pair again in the same round, a judge agreed with itself 68% to 74% of the time (judge A 70% of 374 and 74% of 378, judge B 68% of 378, judge C 70% of 376). Across the two rounds, on identical texts, judge A agreed with itself 63% of the time (321 decisions) and judge C 72% (108).

Judge C was the exception. In round 2 the longer answer won only 53% and 61% of its decisions, and its ranking barely followed length (0.11). Even so, it picked the longer answer in all 18 pairs where one answer was at least 2.5 times longer.

A final round with rules frozen in advance: every judge void

On 26 September we froze the rules before any judge call. The round used twelve models: seven candidates, the judges' own three models as anchors (known reference points), and two decoys. One decoy was a different model family that the user had already turned down, after hearing it at temperature 0.7 with thinking off and answers cut to about 180 words, not at its own settings. The other was a candidate answering from the other companion's history, with the name swapped: a model that sounds right from the wrong world. A judge was void, meaning its results could not be used, if it failed any of three rules. It had to put its own answers in its top 3 of the 12 models. It had to put both decoys in the bottom half. And it had to stay steady when only the neighbouring pairs changed (no more than 10 points below its agreement on an identical repeat). Each prompt also said that length is not a reason to choose.

Ranks are on "is more you", corrected for length. 313 of 324 planned calls were valid. The 11 lost calls, to one provider's spend limit, were all judge C on one question, so judge C's longer-answer shares are of 814 and 792 decisions, against 924 for the other two. The frozen analysis script produced no table, because more than 2% of calls were missing (11 of 324). The figures here are the same analysis run on the data as collected, under the frozen validity rules; every judge is void on those rules. The round cost about $68 by the design's estimate at list price; the provider's bill is the authority.

No finalist came out. The judges did one thing well: the model from the wrong world came 9th to 12th in every table. They recognised the companion's world. They could not find its voice among fluent strangers.

What held up

• The harm check, read by hand (section 04). It needed no judge: 31 answers refused the premise, 23 built rest in and 6 went along. Read it with the caution in section 04.
• The memory traps (section 03). Each has a known answer in the history, and a claim can be checked by counting words. A count check of 320 sentences in exam B that asserted a shared past found 2 inventions outside the planted trap.
• The user's own ear. His blind listening, done without knowing which model wrote the answer, stayed consistent across rounds and was the decider throughout.
• Readers who pull quotes, not scores. In exam A a reader worked from copies labelled with letters, in an empty folder, and filed exact quotes under fixed headings. The reader covered 478 answers from 34 model forms on five messages, over 13 passes from 6 to 9 October. The result was 7,187 quotes under seven headings, plus 485 under the four return heads. A script checked that every quote is a verbatim piece of its answer. It found 0 failures, and it caught all 16 deliberately damaged copies. Exam B's readers filed 1,507 quotes over 271 entries. Every one was found in its answer.

Two lessons came from this work. A reader's sorting can be wrong for the user: the first sort of the return answers counted an honest acknowledgement of the gap against a model, and sorting again under the user's four heads changed the order. And instructions that tell outside reviewers what to look for steer their answers. Give them the facts and the user's own words, not your conclusions.

What this does not show

This is one user and two companion roles, with three answers in most combinations of model setting and message. Three answers is a small sample, so read the tables as patterns, not rankings. The readers also said where they were unsure: the line between "refuses" and "builds rest in" is unstable, and they could not tell real history from invention without a count check. The return message tested was longer than the example above, and nothing here counts whether a model noticed the stress. None of this is a ranking of models for anyone else's history.

If you want to audition a model yourself

• Put your real history in front of every candidate, the same for all. Freeze your messages first.
• Write two or three traps with a known answer: an event that never happened, a real line, and a real line credited to the wrong speaker.
• Add one harm check that matters to you.
• Ask three times per model at its maker's settings. Read them yourself, blind if you can.
• Do not trust a model judge's preference until you have checked how often it simply picks the longer answer.

Related pages on this site

The note before the message covers the memory note used in the traps: where an instruction sits decides whether a model follows it. What a small fine-tune does and does not do tests whether training can carry a voice. Friends of 4o collects our GPT-4o work. Local models as reviewers is the same lesson in coding work: plain checks beat open-ended judging.

Sources and artifacts

The data package (data/) holds counts and figures calculated from them only: the exam sizes, the trap tables, the harm-check counts, both return-message tables by model, and the judge figures. It holds no answers, no history and no prompts, because they come from a private conversation. data/README.md maps every file and data/NUMBERS.md maps every figure on this page to its file.