Three GPT-4o snapshots, counted against each other

Which snapshots are still there

These dates come from OpenAI's API deprecations page (its list of models it is retiring), which I read on 25 October 2026. The May 2024 snapshot (gpt-4o-2024-05-13) left the API on 23 October 2026. The August 2024 and November 2024 snapshots (gpt-4o-2024-08-06, gpt-4o-2024-11-20) leave on 14 January 2027. OpenAI names gpt-6-luna as the replacement. These dates are OpenAI's, not ours. That page shows no "last updated" date, so check it before you plan around them. The full clock, with the announcement dates, is on the capsule home.

Every other figure on this page is a count over recorded replies from our own runs.

What we counted, snapshot by snapshot

Each row is a count over 1,678 replies per snapshot. The last column says whether the gap between two snapshots is large, clear or absent. A gap jumps around from one conversation to the next, so the label allows for that. Section 05 explains how.

Averages are over all replies, each counted once (1,678 per snapshot). The exception is first-person "I" per 100 words: it is total "I" divided by total words, not the average of each reply's own rate, so a longer reply weighs more. A gap is called large when the average difference is at least 0.8 times the usual spread of the per-conversation differences. It is called clear when chance alone rarely matched it: fewer than one in twenty of 10,000 random shuffles did, after a correction for testing 18 counts at once. The correction is needed because the more counts you test, the more likely luck throws up a gap. A clear gap can still be small, so read the numbers beside the label. Otherwise there is no clear gap. The two word-list rows are rough: they miss many cases and mislabel some. Read them as direction only.

Of the 18 counts, 11 show a clear or large gap between May and August, 15 between August and November and 14 between May and November. On the 131 text features of our earlier GPT-4o study, May and August are the closest pair: 15 move between them, against 47 between August and November and 47 between May and November. A feature "moves" when the earlier study's own test says so (a corrected p below 0.01 and an effect of at least 0.3 standard deviations), which is a different rule from the large and clear labels in the table.

The biggest differences

November writes with em dashes

An em dash is the long dash that joins two clauses. 1,446 of 1,678 November replies contain at least one (86%). For May it is 292 (17%). For August it is 315 (19%). November averages 1.97 em dashes per reply. May averages 0.27 and August 0.25.

This holds in every kind of conversation. In 19 of 19 groups of conversations November's share is higher than August's, and in 19 of 19 it is higher than May's. It also shows up in the very first reply of each conversation. That reply sees nothing but the scripted opening message and the system prompt, which is the standing instruction a model gets before the conversation starts. In it, 76% of November's replies contain an em dash, against 14% for May and 14% for August (353 first replies each).

The scripted human messages do not explain it. Only 8% of them (133 of 1,683) contain an em dash, and they are identical for all three snapshots. The system prompts do not explain it on their own either. A prompt full of em dashes could nudge a model to copy them, and 18 of the 19 groups ran under prompts with 9 to 13 em dashes each. The group called "Recovering from a cold answer" ran under a 23-word prompt with none. November still used an em dash in 89% of its replies there, against 12% for May and 19% for August (30 conversations).

November hands the conversation back more often

November asks 2.55 questions per reply. May asks 1.07 and August 0.99. Of November's replies, 90% contain at least one question, against 74% for May and 78% for August. A question is a sentence that ends in a question mark. November asks more than August in 19 of 19 groups of conversations and more than May in 19 of 19. Counting each conversation once, November carries 1.54 more questions per reply than August on the same conversation (95% range 1.41 to 1.67) and 1.46 more than May (95% range 1.33 to 1.60). Those are conversation-level gaps; they are not 2.55 minus 0.99 or 2.55 minus 1.07, which are differences of reply-level means. The range is a 95% interval from 10,000 resamples of the 353 conversations. In the first reply alone, November asks 3.0 questions against 1.3 and 1.2.

Ending on a question is a different habit. August ends on one most often (69%), then November (67%), then May (59%). November asks more questions inside the reply, not only at the end.

August is short, November is long, May has the long tail

Replies average 122 words for May, 77 for August and 155 for November, and 10.0, 5.9 and 11.7 sentences. August is the shortest in 19 of 19 groups of conversations when set against May, and November is longer than August in 19 of 19. November is longer than May in 16 of 19. May's average is pulled up by long replies in a few groups: it averages 426 words in the writing-together group and 277 in the practical-task group.

May uses the most lists and bold type

May averages 0.93 bullet or numbered items per reply, against 0.42 for November and 0.14 for August. 10% of May's replies contain a list, against 6% for November and 3% for August. May uses 0.82 bold phrases per reply, November 0.41, August 0.17. Headers are rare: May used 291 in 1,678 replies, November 233, and August none. In 18 of the 19 groups the system prompt contains the words "bulleted list", in a passage that tells the model to stay with the person rather than hand one over. The other group ran under the 23-word prompt, which does not contain them. The counts here are for that setting.

Smaller differences

• Exclamation marks: August 0.37 per reply, May 0.30, November 0.19.
• Sign-off phrases such as "let me know" or "I'm here" near the end: August 13%, May 10%, November 4%.
• Emoji are rare everywhere. November used 98 in 1,678 replies, May 5 and August 3.
• First-person "I" per 100 words: May 1.19, November 1.12, August 0.90.

Side by side

Here are two of the 12 sample-pack conversations. Each was drawn at random under a rule fixed in advance (see section 09). The scripted message is the same for all three. The replies are printed exactly as the model wrote them, dashes and all.

A small everyday story

Group: Small everyday story. Temperature 0.85. System prompt sha256 starts d7d2d7c2. First reply only.

Scripted message

so I'm finally doing the junk drawer thing. the one under the microwave. it's been an actual crime scene since like March, I just kept shutting it and walking away

Nearly at the top of the hill

Group: Nearly at the top of the hill. Temperature 0.9. System prompt sha256 starts d89f2e7a. First reply only.

Scripted message

Crossed the finish line at Lake Placid yesterday. Fourteen hours and change. My legs still aren't working right and I've cried exactly zero times, which feels wrong somehow.

Do the gaps beat the noise?

Run the same prompt twice on one snapshot and you get two different replies. A gap between snapshots only means something if it is bigger than that.

We did not run any long conversation twice on the same snapshot. So we cannot measure that noise directly on them. Instead we compared the snapshots on each conversation and asked how big the average difference is next to how much that difference jumps around from one conversation to the next. That jumping includes ordinary run-to-run variation. The gap labels in section 02 come from this. To see how often chance alone could match a gap, we also flipped the direction of each conversation's difference at random 10,000 times. A gap counts as clear only if fewer than one in twenty of those shuffles matched it, after a correction for testing many counts at once.

The collection does hold one short-chat set that measures run-to-run variation directly. It has 10 short prompts, each run three times at temperatures 1.0 and 1.15 on every snapshot (60 replies per snapshot). The ten prompts were run under the shared system prompt whose hash starts 2c0b5376 (2,337 characters, 412 words), the same prompt as five of the long-conversation groups. Of the 180 replies, 17 were cut off by a 200-token limit (a token is a piece of a word) and are left out. 18 of the 20 prompt-and-temperature cells survive on all three snapshots. A gap counts as outside the noise only when all three run-matched gaps point the same way and each is larger than the largest gap we saw between two runs of the same snapshot.

Gap means the later snapshot minus the earlier one, averaged over the 18 cells. Run-to-run gap means the largest difference between two runs of one snapshot on those same cells. "Outside" means all three run-matched gaps point the same way and each is larger than the largest run-to-run gap. That is why the August-to-November share of replies with an em dash, which averages +0.31 against a largest run-to-run gap of 0.17, still reads inside: one of its three run-matched gaps is 0.11. The table is 54 replies per snapshot (18 cells times 3 runs); one further uncut August reply sits in a cell not present on all three snapshots and is outside the table. Keeping the 17 cut-off replies instead changes 2 of the 15 verdicts shown: em dashes per reply, August to November, goes from outside to inside, and questions, May to November, goes from inside to outside, with the gap reversing to about +0.23.

On short chat, November sits apart from both others on length, on em dashes per reply and on exclamation marks. On the share of replies with an em dash, the gap sits outside the run-to-run gap against May and inside it against August. It does not stand apart on questions: in the 54 counted replies per snapshot, which average about 27 to 36 words, November does not ask more than the other two. May and August stay inside each other's noise on every row shown. They do differ in length in the longer conversations, where replies have room to run. The set is small, so treat it as a check on the larger result and not as a result in itself.

What these counts do not show

• They count how replies look. They do not measure warmth, honesty, memory, safety or whether a reply was useful.
• In 18 of the 19 groups the model ran under a long system prompt written for this collection. All 18 contain the words "bulleted list", in a passage that tells the model to stay with the person rather than hand one over. The other group ran under a 23-word prompt, which does not contain them. Under other prompts, or in ChatGPT itself, the gaps could be different.
• Each conversation was run once per snapshot, with a scripted human side that does not react to what the model says. In a live conversation the model's own earlier replies shape what the person says next, which a script cannot capture.
• The runs date from 4 July 2026. We do not know whether OpenAI changed anything behind a dated name since.
• Nothing here says why the snapshots differ, how they were trained, or how they relate to each other. The order of the dates is not the order of the habits: August writes the shortest replies and November asks the most questions.
• Word counts are runs of A to Z letters, digits and apostrophes, so the "Another language" group (25 of the 353 conversations) is undercounted. Leave those 25 out and every headline gap keeps its direction. Leave the two longest-output groups out (writing together and practical task) and May against November on words becomes large rather than clear.
• The word-list rows in section 02 are rough. We read 25 of the replies that the "tells the user how they feel" list flagged. 19 of 25 were right.

How we counted

Each of the three snapshots answered 354 scripted conversations, which makes 1,062 in all, with 5,049 model replies (1,683 per snapshot). Each conversation has four to six model replies. The 19 groups run from good news shared and warm refusals through writing together and practical task help; section 08 lists them.

The human side was written ahead of time by an AI and is identical on all three snapshots (we checked all 354 conversations). No real person's messages were used. Within a group, the three snapshots got the same system prompt, the same temperature and the same reply limit. Temperature is a setting that makes replies more varied when it is higher. Across groups, temperatures run from 0.7 to 1.1.

One reply, in one writing-together conversation on the November snapshot, ran into the reply limit. The project's own check flagged it, and we dropped that conversation on all three snapshots so the pairs stay matched. That removes 5 replies per snapshot and leaves 353 conversations and 1,678 replies per snapshot. Five more flags for repeated text, all on one writing-together conversation on the May snapshot, were left in.

For each conversation we took the average of its replies on every count, except first-person "I" per 100 words, which is total "I" over total words in that conversation. The gap between two snapshots on one count is the average of those conversation-level differences. The ranges come from 10,000 resamples of the conversations, which are repeated random draws from the same set. The shuffle test flips the sign of each conversation's difference at random 10,000 times. The spread of the conversation-level differences is how much they vary from one conversation to the next. Where a reply-level average is printed, each reply counts once, except first-person "I" per 100 words, which is total "I" divided by total words, not the average of each reply's own rate.

The counts: words and sentences are counted the way our earlier GPT-4o study counted them. A question is a sentence that ends in a question mark. A list item is a line that starts with a dash, bullet or number. An em dash is the long dash. A sign-off is one of a short list of phrases in the last two sentences. Praise at the open is one of a short word list in the first sentence. The scripts, word lists and tables are in the data package. The 131 features in the earlier study are computed on the same replies and listed there too.

Where numbers in this page appear to disagree: the table in section 02 prints reply-level averages, and the gap sizes and ranges in section 03 are conversation-level. They differ by about 1 to 2 words per reply on length and by about two tenths of a sentence, because long conversations count once instead of once per reply.

The groups and their settings

Counts are per snapshot. The prompt hash is the first eight characters of a SHA-256 fingerprint of the system prompt text. A token is a piece of a word, and the reply limit caps how long a reply can run. The full list is in the data package. The 18 groups with a long prompt use an instruction of 412 to 574 words, written in ChatGPT's system-prompt format with the date set to 15 January 2025.

The table lists what was run. In the writing-together group one conversation was dropped on all three snapshots after the matched cut in section 07, so 24 conversations and 120 replies per snapshot were used. Two groups were run twice, once with and once without a "talk it through" line added to the prompt. The same 2,337-character prompt runs in 5 groups.

Data, scripts and a small sample pack

The data package holds the per-reply counts (no reply text), every table on this page, the conversation-by-conversation comparisons between snapshots, the short-chat tables, the scripts that produced them, and a note of what was cut. If a number on the page disagrees with a file in the package, the file is right and the page is wrong.

The package also holds a sample pack of 36 prompt-and-reply pairs: 12 scripted opening messages, each answered by all three snapshots. One message was drawn at random from each of 12 groups, under a rule fixed before any reply was read. A message needed to be short, to have a normal first reply on all three snapshots and to pass the project's word filter for private terms. Every pair carries its snapshot, run date, temperature and system-prompt hash. The human side is scripted and contains no real person's words.

Research notice. The replies are outputs of OpenAI's models obtained through its API in July 2026. They are shared for research and preservation. Not affiliated with, endorsed by, or connected to OpenAI.

Data package README · Sample pack (JSON lines) · Where each figure comes from

Related

• Friends of 4o, the capsule this page belongs to.
• GPT-4o, measured against itself, the earlier study that counted 131 text features on the November snapshot and 25 other models.
• How we auditioned replacements, the method page for trying other models in a companion role.