Pulled the App Store review feed for the hosted meeting notes and voice to text apps, expecting the complaint pile to be about word error rate. It is not. Of 75 low star reviews in that category: price 37, privacy 16,…
A recurring frustration: the underlying engine gains a feature and the interfaces around it take months, or never. Worth understanding why, because it changes what you should run. Why the lag exists The engine ships…
An eight year old machine, integrated graphics, and it is genuinely usable for the sizes that suit it. What mattered, in order. Memory bandwidth, not cores. Generation is bandwidth bound. Dual channel over single…
Access to the hosted assistant I used every day went away with no warning and no explanation, and I am not interested in being in that position again. So the question is not "which model is best", it is how close a…
About two thousand examples of the kind of output I want, collected over a year. The obvious move seems to be fine tuning a smaller model on them so I can run it locally and stop paying per token. Before I spend a…
I am running a model locally and generation does not stop when the answer is finished. It produces a good answer, then keeps going: starts a new turn, invents a follow up question from the user, answers that too, and…
I have been running a local model for a few weeks and I use the temperature slider the way everyone seems to: low for extraction, higher for drafting. That works, but I am copying a habit rather than understanding it.…
I batch several prompts together for generation. With a batch size of one everything is fine. As soon as I batch prompts of different lengths, the shorter ones come back as nonsense repeated tokens, unrelated text,…
I want to run a model of around seven billion parameters locally and I am trying to work out what hardware I need. The figures I find range from about four gigabytes to well over twenty, which is too wide to shop with.…
Several launches this month lead with running on the device rather than in the cloud: a media editor positioned explicitly as an on device alternative to a well known cloud product, local semantic search for agents,…
Small internal tool, maybe 40 requests a day in bursts. If two people use it within a few minutes it is instant. If nobody has touched it for an hour, the first request takes 38 seconds before a single token appears,…
Box: DGX Spark. 128GB unified LPDDR5X, 273 GB/s shared between the CPU and GPU sides of the GB10. That bandwidth number is the whole story of this machine and I keep having to remind myself of it. Current daily driver:…
Two questions in one, and I am not embarrassed about the first. One: what does bpw actually mean? I see it everywhere, 4bpw, 2.5bpw, "stay above 4bpw" and I have been nodding along for months. I assume it is bits per…
At 4k context I get 28 tok/s and everything is on the card. At 16k the same model gives me 3.1 tok/s. No error, no warning, it just gets slow. shows 23.9GB used, and the server log says it offloaded 51 of 65 layers to…
I use models about two hours a day for code and for summarising documents I would rather not upload anywhere, and my API bill has been running around $40 a month. I have a 3060 12GB in my desktop now, which handles…
4060 Ti 16GB. The task is document extraction: about 3k tokens of prompt in, a JSON object with 12 fields out, several of them nullable. Runs unattended on a few hundred documents a night, so correctness matters more…
Single 4090, currently running a 14b through Ollama behind a small FastAPI wrapper for my own use. I now need to open it to about eight colleagues for an internal tool, and in testing with four simultaneous requests it…
8B model, Q4 K M, so everything should be on the GPU. says 6.9GB on the card, which matches what I expect. But the process RSS is 5.4GB. On a 16GB box running two other things, that is the difference between fine and…
3060 12GB, and RAM is cheap enough that going to 96GB costs a fraction of what a bigger card would. I want to run 32b class models and I can tolerate slow, but not two tokens a second slow. Everyone says VRAM is what…
$700 total. Motherboard has one PCIe x16 slot and one x4, 750W PSU, and I would rather not replace either. Two 3060 12GB cards gets me 24GB for around $500 with room left over. One used 3090 is 24GB for roughly $650 700…
48GB M4 Pro, Ollama, 14B Q4 K M, 8k context. I get a steady 18 tok/s. Everything I read has people casually mentioning 40+ on Apple silicon and I cannot tell whether they are on different hardware, measuring something…