October field notes
Nine faults and traps on one RTX 5090 desktop, each with what it cost and the one-line check. The first entry is five GPU timeout faults in the first week on NVIDIA's 595 driver, against none in six boots on driver 580.
This page is for anyone who runs AI models on their own machine and wants to avoid mistakes we already made. Each entry is something that went wrong on our desktop between 19 September and 23 October 2026, four of them from September. It says what the problem cost, how we caught it, and the one check that would have caught it sooner. The first entry matters most if you upgraded your NVIDIA driver this autumn. After we moved to the 595 driver, the graphics card timed out five times: twice while a test server was stopping, and three times inside Gemma 4 31B fine-tunes.
Five GPU timeout faults since the 595 driver
What happened. Five times, the kernel log (the operating system's own log) recorded an Xid 8 (NVIDIA's error code for this fault). The driver's watchdog, a timer that checks the card is still responding, decided the card was stuck
(GPU is probably locked! Notify Timeout Seconds: 7) and
reset the work in progress. A driver is the software that lets the operating system use the card. Each line names the program whose work was
reset.
| When | Work that was reset | What was running |
|---|---|---|
| 2026-10-19 19:19:30 | a desktop app | A Qwen3.8-27B test server (UD-Q5_K_XL; a 196,608-token context window, which is how many tokens the server will hold; an 8-bit cache (a shorter way to store the conversation so far); a draft model (a small second model that guesses ahead) plus n-gram drafting (guesses drawn from phrases already in the text); micro-batch -ub 512, which is how many tokens it handles at once; the run is named thinkON; card peak 29,977 MiB during the run; MiB and GiB are binary megabytes and gigabytes, so that is about 29.3 GiB) was stopping. Its run record logs the stop at 19:19:30. |
| 2026-10-19 19:23:42 | the desktop's compositor, which draws the screen | The next test server (the same file, 8-bit cache and draft setup, 229,376 window, -ub 256, run named thinkON; card peak 30,647 MiB during the run) was stopping; its record logs the stop at 19:23:44. Our operator note records that the screen froze when the compositor's work was reset. The boot ended at 19:41:05 and the next began at 19:41:44; that this restart was forced is also from the operator note. |
| 2026-10-21 18:05:18 | llama-server | Artemis-31B v1.2 (TheDrummer), a Gemma 4 31B fine-tune (a version of the model that someone retrained), Q5_K_M. Its last request before the fault stopped at 114,792 tokens. The server died with CUDA error: the launch timed out and was terminated. |
| 2026-10-21 18:12:00 | llama-server | The same fine-tune on a newer llama.cpp build. Its last request before the fault stopped at 115,805 tokens. Same error. |
| 2026-10-21 21:21:37 | llama-server | Aura 4o Rebirth, another Gemma 4 31B fine-tune. Its last request before the fault stopped at 113,568 tokens. Same error. |
Before and after. The kernel journal on this machine reaches back to 12 October. A boot is one start of the machine. Six boots on driver 580.173.02, from 12 October to 18 October 11:47, logged no Xid line. Driver 595.104.02 loaded at 11:48:03 on 18 October. The faults came in two of the eight boots since. None has been logged after 21 October 21:21, through the journal's last line, 25 October 02:03. In that time the same machine read 239,533 tokens on Ornith-1.5-122B and 206,883 on Hemmingway-1 without a fault. On two later starts, Aura 4o Rebirth read prompts of 80,062 and 113,205 tokens, plus later prompts, the longest of those 7,419 tokens (44 prompt-eval lines in all, one per prompt read: 20 and 24), with no CUDA line in those files.
What it cost. A restart at 19:41, which our operator note calls a frozen screen and a forced restart. Three crashed test servers, and the runs they were in the middle of.
What we do not know. The cause. Two faults came while a test server that had peaked at 29,977 and 30,647 MiB of the card was stopping. Three came inside Gemma 4 31B fine-tunes, whose last requests stopped at 114,792, 115,805 and 113,568 tokens. Five events show a pattern. They do not show a mechanism.
The check. After a restart like that one, read the kernel log of the boot before this one, because each restart starts a new log:
journalctl -b -1 -k | grep -E 'NVRM|Xid' journalctl --list-boots
The first line reads the previous boot's kernel log and keeps only lines that mention NVRM or Xid. The second line maps boot numbers to dates, so you can count faults per driver version.
A knowledge base whose meaning search failed with no error
| What happened | A knowledge base in Open WebUI (a self-hosted chat interface) is a set of documents the model can search. Each passage is stored as a vector, a list of numbers made by an embedding model. To search, the instance turns the question into a vector with its current embedding model and looks for close matches. On 18 October, four knowledge bases on three of our Open WebUI 0.12.3 instances held vectors of the wrong length for the model now asking: 768 numbers where bge-m3 makes 1,024 (two bases), and 384 where nomic-embed-text makes 768 (two bases). Vectors of different lengths cannot be compared. Meaning search on those bases failed with no error. Keyword matches still came back, except on the one instance where hybrid search (meaning and keywords together) was off. |
|---|---|
| What it cost | Answers that should have drawn on those knowledge bases by meaning went without it, for as long as the mismatch had stood. We do not know when it began. |
| How it was caught | An audit read the vector sizes from copies of each instance's vector store. The bases were rebuilt on 20 October through Open WebUI's own re-index function, which makes the vectors again. |
| The check | On a copy of data/vector_db/chroma.sqlite3, run select name, dimension from collections; The dimension is the vector length. Every knowledge base's dimension should equal your embedding model's output size. |
| A second trap | When you copy a stopped instance's webui.db to inspect it, copy webui.db-wal and webui.db-shm with it. The WAL (write-ahead log) file holds recent writes that the main file does not yet have. The database alone looks out of date. |
A downloaded shard with the wrong hash
| What happened | On 17 October, part 3 of 5 of Beam 1.0's UD-IQ2_M file failed its check. A model file is split into parts like this, called shards. The fetch script's note records that the hf tool's default transfer path produced a bad part 3 twice, with a sha256 (a fingerprint of the file's contents) that did not match the published hash (the published hash begins 7478bfa7), and that the tool hung at the end of the transfer. |
|---|---|
| What it cost | Two downloads of one shard, by that note. Beam is now served from the UD-Q2_K_XL file instead (see the October roster). |
| How it was caught | The fetch script compared the shard's sha256 with the published value before the file was used. |
| The check | Hash every shard and compare it with the repository's LFS sha256 (the hash the model repository publishes for each large file) before loading. A file of the right size can still be wrong. Our third attempt turned off the default transfer path (HF_HUB_DISABLE_XET=1), downloaded into a scratch folder, and moved the file into place only if the hash matched. |
| A related trap | In one download round the same week, the hf tool printed a warning that it was ignoring --include, the option that selects files by pattern, because file names had also been given. It fetched only the named files. Use one pattern per --include, give no bare file names, and check the folder's size afterwards. |
A half-finished download that reports itself as mostly done
| What happened | The hf tool sets aside the full size of each file before the data arrives. Those partial files are sparse: they claim their full length, but most of it is holes that hold no data yet. du -sb reports the apparent size, holes included, so a download looked far more complete than it was. In the same download the tool exited with an error on a temporary server fault partway through. |
|---|---|
| What it cost | Wrong progress figures and wrong time estimates on a large download. It was recorded in the download staged on 19 and 20 September. It belongs with the download entry above and was missing from last month's page. |
| The check | Measure real blocks (the disk space actually used) with du -sB1, not du -sb. At the end, check the shard count and the exact byte count of each shard against the repository's list, not the tool's exit code. Wrap the tool in a retry loop, because it resumes from its partial files. |
A converter that would have held about 289 GB in memory
| What happened | On 18 October we converted DeepSeek V4.1-Flash-1017 to a GGUF file (the single-file format llama.cpp loads) with the conversion script that ships with the community port we run it on. A port is a community-modified version of llama.cpp. The script builds each layer's routed experts as one stacked array and keeps them in memory until it writes the file. The gate and up arrays here are shaped 5,120 by 2,304 by 384, and the down arrays 2,304 by 5,120 by 384: the same number of weights, about 2.4 GB each once packed to MXFP4, a compressed number format (arithmetic: weights times 4.25 bits). The finished file holds 120 of them, so the script would have held about 289 GB (268.9 GiB) of arrays on a machine with 188 GiB of RAM. |
|---|---|
| What it cost | The first attempt ran 90 seconds, through 10 of the 120 arrays (the operator note counted 9; the log has 10 repack lines, and the page uses 10), and ended with exit code 143, a terminate signal. Our operator note says it was stopped by hand before the machine ran out of memory. Nothing had been written. |
| The fix | A copy of the converter, changed to build the same array one expert at a time as it writes. We ran it inside a systemd scope (a Linux way to put a memory limit on a job), so that a memory limit could stop the job and not the desktop. It ran from 00:41:36 to 03:29:07, 2 hours 47 minutes. The output matched the earlier V4.1 file's layout: 11 files, 1,012 tensors (the arrays of weights), 501.80 GB, with no tensor differing in shape, type or size. Two metadata keys differ: the name field became Source, and the chat template's new value is None. |
| The check | Before converting a very large model, multiply the size of its biggest per-layer tensor by the number of them and compare with your RAM. Run the job under a memory cap: systemd-run --user --scope -p MemoryMax=64G -p MemorySwapMax=0 python convert.py ... The second setting stops the job from spilling into swap instead of stopping. |
A reasoning setting that means the opposite of what it says
| What happened | GLM-5.3-Flash's own chat template (the text pattern that turns a conversation into the model's input) accepts two reasoning-effort words, low and high. Reasoning effort is how much hidden thinking a model does before it answers. Any other word, including none and medium, becomes max, its slowest mode. Our September runs asked for none and got maximum effort. This template line turns every effort word other than low and high into max. |
|---|---|
| What it cost | A September field card describes those runs as effort none. Reasoning tokens (the hidden thinking) also count against the reply's token limit. The same card reports a separate 40-prompt check on the shipped 262,144, -ub 1024 setting that recorded 36 empty answers out of 40; the file behind that check does not record which effort word it sent, so that count is not the measured cost of the none mapping. |
| A related trap | MiMo-V2.6-Flash's server log shows a 24,000-token reply at 12.28 tokens a second, 32.6 minutes. Its start script's note says that without a reasoning cap the model spent all 24,000 tokens thinking and wrote no answer. The script now sets llama.cpp's --reasoning-budget flag, which caps the reasoning, to 14,000. |
| The check | Read the template before trusting a word: search the model's chat template for reasoning_effort and see which values it tests for. Give thinking models a reasoning cap, and a reply limit well above it. |
A saved conversation that failed after a restore on a server with a draft model
| What happened | llama.cpp can save a conversation's cache to disk and load it back later. The cache is the model's memory of the conversation so far, so a long conversation does not have to be read again. On 29 September GLM-4.7 Full, served with a draft model (a small second model that guesses ahead to speed up the main one), loaded back 48,396 tokens in 0.93 seconds. The next request failed. The server log first reports that the tokens of sequence 0 had inconsistent sequence positions. Then the draft model's decode step, the step that writes the next tokens, failed at position 48,110, and then came "failed to process speculative batch". |
|---|---|
| For comparison | MiniMax M2.7, served with no draft model, loaded back 100,264 tokens the same evening and carried on normally. It is a different model with a different saved length, so it shows only that restores can work, not that the draft model is the cause. |
| What it cost | One failed measurement run. It would have been worse as a production shortcut. |
| The check | After restoring a saved cache, send one real next turn and read its answer before you rely on the restore. On a server with a draft model, treat a speculative-decode error as a failed restore until a later turn finishes. |
A smaller cache that makes the model slower
| What happened | The cache holds what the model remembers of the conversation. Storing it in 8 or 4 bits (fewer bits per value) saves card memory. On Qwen3-235B's UD-Q3_K_XL file, 109,712 tokens deep on 29 September, it cost speed instead: a reply came out at 5.71 tokens a second with a 16-bit cache (at a 118,784-token context window), 4.24 with 8-bit and 2.86 with 4-bit (both at 131,072). Reading the prompt took the same time with all three, 640 to 649 tokens a second. |
|---|---|
| What it cost | This model had been served with the 8-bit cache until that day. On these single replies at different windows, the 8-bit cache was about a quarter slower (4.24 against 5.71). It was switched to 16-bit on 29 September. |
| The check | Measure a long reply at your real depth with each cache type before you accept the smaller one. The saving in memory is easy to see; the loss in speed only shows deep in a conversation. |
A first read longer than your client will wait, and one that finished and never answered
The first time a server reads a long prompt, it must read all of it before it can write the first word. On a large model that does not fit the card, that can take longer than a client's timeout. DeepSeek V4.1-Flash-1017's server also read expert weights from the drive during its read. A client is the program that sends requests to the server, and its timeout is how long it waits before giving up. These are the server's own prompt times, each one request at a time:
| Model and file | Prompt, tokens | Read, t/s | Time to read | Date |
|---|---|---|---|---|
| DeepSeek V3-0324, unsloth UD-IQ1_S | 108,315 | 56.9 | 1,902.6 s (31.7 min) | 2026-10-23 |
| DeepSeek R1-0528, unsloth UD-TQ1_0 | 108,315 | 100.9 | 1,073.7 s (17.9 min) | 2026-10-23 |
| DeepSeek V4.1-Flash-1017, MXFP4 | 110,052 | 153.5 | 716.9 s (11.9 min) | 2026-10-18 |
| Muse Spark Open 816B-A42B, UD-IQ1_S | 106,469 | 155.0 | 686.9 s (11.4 min) | 2026-10-21 |
| Mistral Large Instruct 2411, Q4_K_M | 122,880 | 222.8 | 551.5 s to finish the read; the client timed out at 5,400.1 s with no finished reply | 2026-10-21 |
Minutes are arithmetic on the seconds. Mistral Large Instruct 2411 is a different failure from the four long reads: its read finished at 551.5 seconds, inside the 5,400.1-second client timeout, and then no generation finished. The client's 5,400.1-second timeout comes from a hand-written note, because the client's own log names a private folder; the server log shows the read finished and no generation finished. Context windows: DeepSeek V3-0324 and R1-0528 were read at 114,688 tokens, the other three at 131,072.
What it cost. DeepSeek V3-0324's read alone is longer than the 1,800.4 seconds at which a September turn through an agent framework ended with no answer, recorded in our September field notes. An agent framework is software that lets a model call tools in a loop.
The check. Before you set a client timeout, time one cold read of your real prompt length on the real server. Set the timeout from the measurement, with room to spare, or keep the prompt shorter. A timeout set from the read does not cover a server that finishes the read and then never finishes a reply; watch for a reply starting, not only for the read ending.
What this page does not show
These are single events on one desktop. None of them is a bug report against any project: each is an interaction between a driver, a file, a tool and our own scripts, and several of the mistakes were ours. The GPU faults have no proven cause. The thread-pinning result from the same weeks is on the October roster, because it changed most rows there.
Related pages on this site
- The roster, as of October 2026: every model on this desktop and its speed.
- Silent settings: settings that cost speed or correctness without an error.
- September field notes: the previous month's entries.
- GLM-5.3-Flash: the field card for the model in entry 06.
- Method: how figures are labelled.
Sources and artifacts
The data package holds the kernel log lines for the five faults, the server lines around each crash, the run records for the two stops on 19 October, both conversion logs and the layout check, the fetch script with its published hash, the slot-restore records, the cache-type results, the timing lines for the long reads, and the template line for the reasoning words. Machine names, account and disk names, paths and process numbers are removed, and so is all reply text: the probe files keep only counts, timings and pass or fail flags. The knowledge-base sizes ship as a short table with the instance names withheld.
- data/README.md: every file, where it came from and what was removed.
- data/NUMBERS.md: every figure on this page mapped to its file and field.
If a number on this page disagrees with a file in its package, the file is right and the page is wrong. Tell us at hello@graphometer.ai and the correction goes on the page with its date.