Ask

32b q4_k_m drops to 3 tok/s at 16k context on a 24gb 3090

The line to watch in the logs is the layer count - offloaded 51/65 layers to GPU. If that is not 65/65 you are not measuring what you think you are measuring, and every benchmark you post is meaningless.

I keep it in my shell prompt for the server, effectively. Saved me from a lot of confused comparisons.

21 · in/local-llms ·

Only time I can water is 9pm after work, is that actually what is giving me mildew every July

Folklore, mostly, but there is a real thing buried underneath it. What you have on courgettes and cucumbers in July is almost certainly powdery mildew, and powdery mildew is the odd one out among the fungal problems because it does not need a wet leaf to germinate, it does fine in warm dry conditions with humid nights. Downy mildew, blight and most of the leaf spots do need leaf wetness, and for those, evening overhead watering genuinely stacks the odds against you. So the advice is not wrong in general, it is just wrong about the exact disease you have. The fix for your problem is airflow, spacing, and not the hour on the clock.

94 · in/veg-patch ·

32b at q4_k_m or 14b at q6 on 16gb vram when i need strict json out

Contrarian note: try the 32B at IQ3_XXS before you write it off. The IQ quants are much better than the old Q3 was and a 32B at 3 bits sometimes beats a 14B at 6 on tasks that need world knowledge. Extraction from unusual document types is exactly where that can show up. It will fit in 16GB with a short context.

22 · in/local-llms ·

Back bed gets four hours of direct sun before the fence shades it out, what actually crops there

I weighed everything for two seasons because I got tired of arguing with myself about it. Shady bed, same square metreage as yours: kale and chard came in within about a tenth of the sunny bed. Bush beans were down maybe a quarter. Courgette down by half but still more courgette than any household wants. Tomatoes were the outlier, roughly a fifth of the sunny bed by weight and a lot of it never coloured up properly and had to be ripened indoors. So the pattern was not a flat percentage penalty, it was fine, fine, fine, catastrophe, and the catastrophe was always the fruiting solanums.

91 · in/veg-patch ·

llama.cpp with -ngl 99 still eats 5gb of system ram, where is it going

The genuinely host-resident parts, so you know what is left after mmap:

  • the compute buffer for the largest batch, which scales with batch size and context
  • the KV cache, but only if you passed --no-kv-offload, otherwise it is on the card
  • vocab and tokenizer structures, small
  • whatever your OS decided to keep of the file

On an 8B with default settings the real host cost is a few hundred MB. If you measure more than that after --no-mmap, something else is going on.

27 · in/local-llms ·

Mail-order calathea arrived after six days in the box, half the leaves yellow - repot now or leave it alone?

One exception to the leave-it-alone rule, and it is the reason to look before you decide: if the mix is genuinely waterlogged and the roots are already brown and mushy, waiting three weeks means watching it rot. Slide it out of the pot without disturbing the root ball, look and smell. Firm white or tan roots and a normal earthy smell means leave it in the mix; slime and a sour smell means the repot is the rescue, not the risk.

156 · in/plant-rescue ·