Sasha

It is being unloaded - the default keep-alive is 5 minutes and yours matches exactly. Do not buy a drive, change a setting.

Per request: "keep_alive": -1 in the JSON body keeps it resident indefinitely.

Server-wide, which is what you want for a shared tool: set OLLAMA_KEEP_ALIVE=-1 in the environment of the server process. If it runs under systemd that means a drop-in with Environment="OLLAMA_KEEP_ALIVE=-1" and a daemon-reload, not an export in your shell - that trips up everyone at least once.

Then preload on boot by sending one request with an empty prompt so the first real user is never the one who pays.

Worth knowing where the ceiling is: q8 KV is fine, q4 KV is where people start reporting the model losing track of things early in a long context. If you ever go there, test it on your actual task rather than trusting a benchmark someone posted.

Blunt version: a 14B at Q6 that fully fits will beat your 32B at Q4 that does not, for almost any real workload, because you are comparing 30 tok/s against 3.

People anchor on parameter count because it is the number on the model card. The number that decides whether your tool is usable is whether every layer is on the card. Below that line nothing else matters.

Take the 14B that fits, and separate two things you are currently treating as one.

JSON validity is not a model capability question - it is a decoding question. Constrain the output with a grammar or a JSON schema and the shape is guaranteed at every quant in your list, including Q4. You will never see a missing brace again regardless of which model you pick, so stop weighting that.

What quantisation actually degrades is judgement: which field a value belongs in, whether an absent value should be null or inferred, whether it noticed the second date on the page. That degradation is real but it is gradual, and it is nothing like the 8x throughput cliff you get from spilling 14 layers to CPU.

So: build a 50-document eval set with hand-written expected output, score exact-match per field, and run both. It is an afternoon and it replaces this entire argument with a number. For what it is worth, on my extraction set 14B Q6 scored within 2 points of 32B Q4 and ran seven times faster.

Splitting it by table is the answer I did not consider. Invoices and member management keep the join, everything read-heavy moves to claims.

The index was there but on (user_id) alone, so the subquery still had to visit the heap for org_id. Adding the second column took it to 2.1ms before I changed anything else.

The other half of why everything scores 4.6: your judge has never seen a bad answer, so it has no idea where the bottom of the scale is. Put two examples in the prompt, one clearly poor with the reasoning spelled out and one clearly good, and the distribution opens up immediately. Anchoring beats instruction here.

Many holes, so at least that one is not it.

The puck had a little crater in the middle, so I clearly needed to look at it weeks ago. Paperclip stirring tonight.

Related and worth knowing before it bites: with mmap the first generation after a cold start is slow because the pages get faulted in as they are touched. If you are benchmarking and your first run is always terrible and every later run is fine, that is this, not the model warming up in any meaningful sense.

Big pours also drop the bed temperature fast. Four or five smaller pours cost you 40 seconds and change the cup a lot.

Reframe the goal. At $700 the question is not "how do I run a 70B", it is "what is the best code review I can get from 24GB".

A 32B coding model at Q4_K_M on one 3090 runs at 28 tok/s with 8k of context and is genuinely useful on real diffs. A 70B at 3 tok/s is a thing you demo once and never open again, because reviewing a 400-line diff takes eleven minutes and you will just read it yourself.

Speed is a quality feature for interactive tools. Nobody iterates against something that slow.

Wait. Beans that fresh are full of carbon dioxide, and the gas physically pushes water out of the way, which is why the puck gushes and the bloom erupts. Nothing is wrong with the coffee or your technique.

Rough windows I use: espresso is best from about day 7 to day 21 off roast, filter is usable from day 4 and best around day 10 to day 25. Dark roasts degas faster and are ready earlier, light roasts often want the far end of that range. Write the roast date on the bag with a marker and just leave it alone.

Also decant. If you leave the coffee in the press it keeps extracting against the grounds and the last cup is both silty and bitter. Everything goes into a preheated flask here the moment it is done.

Two things worth confirming anyway, both take a minute:

1. Check the server log actually says it is using Metal and not falling back. It is rare but it happens after an update.
2. If you ever load something above about 32GB on that machine, macOS caps what the GPU may wire and you will get a mysterious slowdown rather than an error. sudo sysctl iogpu.wired_limit_mb is the knob. Not your problem at 9GB, but remember it exists.

It also tells you what a bigger quant costs before you download it. Going from Q4 to Q8 doubles the bytes read per token, so expect roughly half the tok/s. The maths holds up well enough for planning.