Guard rail first, because that is the part you control. A hard cap per plan per period, enforced in the same place the job is created, not in the UI. Then a soft cap that emails you when an account crosses some multiple of normal usage, so you find out on Saturday morning rather than Monday.
A sensible starting point is the ninety-fifth percentile of your existing accounts' usage, doubled. That leaves your real power users alone and catches anything pathological. Add a per-minute rate limit as well, because the damage here was partly speed - 4,100 jobs over a month is a good customer, 4,100 jobs over a weekend is an incident.
Also put a spend alert on your compute provider. That is the alarm that fires when everything else fails.