every generated test mocks the thing it is supposed to test, 40 files in tests
Inherited a service with no tests. Asked for coverage, got 40 Vitest files, 380 tests, all green, 84% coverage. Felt great for about a day.
Then I deliberately broke a validation rule: changed a >= to a > in the middle of the pricing logic, and every test still passed.
Looking properly, most files mock the dependencies of the module under test so thoroughly that the assertions are checking the mocks. There is a test literally called calculates tax correctly that mocks calculateTax and asserts it was called once.
Is there a way to ask for tests that don't do this, or was 380 tests just the wrong ask?
@coworking_cass · 4mo ago · 3 replies
380 tests was the wrong ask. Coverage as a target produces exactly this, the cheapest way to execute a line is to mock everything around it, and you asked for lines executed.
Two changes fix it.
Ask for behaviour rather than coverage: "write tests for the pricing rules in this file, table-driven, real inputs and expected outputs, no mocks except the HTTP client." Naming what may be mocked is the important part; the default is to mock everything reachable.
Then verify with mutation testing rather than coverage. Stryker pointed at the pricing module will do automatically what you did by hand: flip a
>=, run the suite, report that nothing failed. Start with one module, it's slow. A mutation score is the only coverage-adjacent number that can't be gamed by mocking.Also: delete the 40 files. Seriously. They're 380 tests of assurance you do not have, and keeping them means every future failure is ambiguous.
Reply
Report
@drainfield_dez · 4mo ago
Ran Stryker on the pricing module overnight. 71% line coverage, 9% mutation score. Deleted the lot and rewrote 22 tests by hand with the model filling in table rows.
Reply
Report
@heirloom_hank · 4mo ago
9% is a great number to have. It turns 'these tests feel useless' into something you can put in front of a colleague.
Reply
Report