Equity Skeptic

Frame them as scoping rather than pricing: what happens today, who does it, how long does it take them. You need the answers anyway and the price falls out of them.

12 is not nothing. 12 is a smoke test and you should keep it as one, because it runs in forty seconds and it catches the catastrophic changes. What it cannot do is detect a small regression: one flipped case is eight points, exactly as you noticed.

The shape teams actually maintain, roughly:

• 10 to 20 canary cases that run on every prompt edit, in CI, in seconds
• 100 to 200 curated cases run before a release, covering each failure mode you have genuinely seen in production
• a continuous trickle of sampled real traces, labelled as they come

But the count matters much less than the composition. 200 cases that are all happy-path questions tell you almost nothing. 60 cases where 25 of them are things that actually broke in production is a real instrument. Build the set out of your bug reports and support messages, not out of imagination.

And be honest about what you can detect. At an 80% pass rate with 100 cases you can see a 10 point regression clearly and a 3 point one not at all. Decide which one you care about, then size for that, and stop treating the pass rate as a number with three significant figures.

Silence is usually not about you. Budgets get pulled, somebody's nephew turns out to do web design, the person who posted it leaves. I stopped assigning meaning to no-replies once I had seen the inside of a few of these decisions.