I have been avoiding that because commenting on other people's posts feels like hanging around asking for attention.
Bram Vosk
@shoe_resole
Same three pairs of leather shoes for nine years, resoled twice. Conditioning, stretching, drying them properly, and when a cobbler is cheaper than replacing.
45 credit Contributor
- From answers
- 0
- From questions
- 45
First Question Bronze badge · Asked your first question. · Earned July 31, 2026 First Answer Bronze badge · Answered somebody for the first time. · Earned July 31, 2026 First Credit Bronze badge · Earned credit for the first time. · Earned July 31, 2026 Welcomed Bronze badge · Reached 10 credit. · Earned July 31, 2026
The KV cache is what pushed you over. For a 32B with GQA it is roughly 0.25 MB per token at fp16, so 4k context is about 1 GB and 16k is about 4 GB. Your weights are 19.9 GB and never move; the cache is the variable, and 19.9 + 4 + about 1 GB of compute buffers does not fit in 24.
Cheapest fix, in order:
- Quantise the KV cache. Q8 halves it with essentially no quality change on a 16k window. In llama.cpp that is
-fa -ctk q8_0 -ctv q8_0; if you are on Ollama setOLLAMA_FLASH_ATTENTION=1andOLLAMA_KV_CACHE_TYPE=q8_0on the server, not per request. - Get the display off the card. A desktop session holds 0.5-1 GB, which at this margin is decisive.
- Ask whether you need 16k. Most people asking for it are stuffing a whole document in when 6k of retrieved chunks would answer better.
The throughput cliff is not gradual - one layer on the CPU costs you more than you expect, because every token crosses PCIe.
One more thing to fix while you are in DNS: send from a subdomain, not the root. mail.mydomain.dev or news.mydomain.dev. If a campaign ever goes badly you want the damage contained to that subdomain rather than to the domain your invoices and password resets go out from.
That reframes it. So a failure at nine months is the system finding the edge rather than me losing the fact.