This shouldn’t be a hot take in big 2026, but I’ve seen lots of people presenting the number of tests in a codebase as a measure of software quality and confidence in their code.

These posts and replies are recreated from examples I read almost daily. The original wording and profiles are omitted because the point is not to criticise individual people.

Test count is a vanity metric

Test count is a very easy metric to inflate. When prompted with “add more tests”, an agent can generate a large number of redundant tests simply to satisfy the instruction. A team can also split one scenario into many methods or test a private implementation detail from several directions. The count grows without necessarily protecting more behaviour or making the code more reliable when it changes.

Goodhart’s law describes what happens when a measure becomes the target people optimise for: it stops being a reliable measure of the thing it was meant to represent. Coverage is useful when it helps us notice code that has not been reached by the tests, and test count can describe the size of a project, but targeting these numbers can turn both into output to manufacture. Set a coverage threshold and people will add tests that execute code paths without necessarily asserting anything useful. Set a test-count target and coding agents will generate tests. Neither number can demonstrate confidence in the behaviour that matters.

What coverage can tell us

Coverage tells us which code was executed while tests ran, not whether their assertions were useful or whether they would catch a meaningful mistake. A test for a data-access class might mock the database client, request a record through the class, and verify that it returns the record configured in the mock. It may execute every line in the method and show 100% coverage, even though in practice it has mostly tested that the mock returns the value it was told to return.

Martin Fowler makes a useful distinction in his explanation of test coverage. Coverage is valuable because it can reveal code that tests never reach. Once the number is high enough, however, it says very little about the quality of the assertions or the relevant failures the tests can catch. Research reached a similar conclusion: after controlling for the number of tests, coverage had only a low to moderate relationship with fault detection.

These limitations existed before coding agents, but AI makes them easier to overlook and much easier to scale.

AI can couple tests to an implementation

When an agent writes tests after implementing a feature, it will often verify what the code already does. A mistaken assumption or an already present error can become an expectation in the tests, while implementation details become part of what those tests protect. Changing the implementation then requires changing the tests too, even when the externally visible behaviour remains the same. This adds friction to development and refactoring without increasing confidence in the codebase.

The order here matters. A test written before the implementation describes the expected behaviour without being influenced by how the feature happens to be implemented. It gives the agent an independent target and a clear feedback loop: the test should fail first, then the implementation should change until it passes.

When tests are generated afterwards, the implementation becomes the agent’s main source of truth. The agent infers the expected behaviour from the branches, return values, and errors already present in the code. This can produce coverage while preserving incorrect assumptions and implementation details. The tests pass, but they may only confirm that the code behaves exactly as it was written.

None of this means that a large test suite is inherently bad. The important question is why those tests exist and which risks they address.

When a large test collection makes sense

SQLite is a good example of a serious project where an enormous test collection is entirely justified. Its public testing documentation describes 51,445 distinct TCL test cases. Its separate TH3 harness adds more than 50,000 cases and runs about 2.4 million instances for full coverage.

That scale exists for a reason. SQLite runs on billions of devices, stores persistent data, supports a huge range of platforms, and maintains decades of compatibility. Its tests exercise failures that would be irrelevant for a typical CRUD service, including memory exhaustion, disk I/O errors, power loss during a transaction commit, malformed databases, different encodings, page sizes, journal modes, and concurrency. It also compares behaviour against other database engines and uses fuzzing. SQLite’s testing documentation is an unusually good explanation of what a large collection of tests is actually for.

SQLite’s test suite is valuable not because it is large, but because its size follows from the number and severity of the risks the project must handle. Each category of tests exists to protect a specific behaviour or failure mode. That is the useful lesson to take from its scale: tests should be chosen in response to real risks, not added to improve a metric.

What to test

Most projects face a narrower set of risks than SQLite, but the same principle applies. Tests should begin with the consequences of failure rather than with a target number. The first step is to identify the behaviours that would cause real harm if they broke, and then choose the simplest mechanism that can protect them.

Production failures provide the clearest examples. When a bug reaches production, write a test that reproduces the failure and confirm that it fails before fixing the code. Once the fix makes it pass, keep the test so the same bug cannot return. Whenever practical, a production incident should leave a scar in the test suite.

Not every guarantee needs to come from a test. Some invalid states can be excluded more effectively by the type system. Testing what happens when a number is passed to a function that only accepts strings adds little value when a typed language already rejects that call. Types cannot protect runtime boundaries, however. Input from users, files, databases, and external services still needs validation because external data does not become trustworthy when it enters a typed application.

For the behaviours that do require tests, test them at the simlest level that still provides meaningful confidence. Pure business logic can usually be tested without infrastructure, but a data-access class is often better tested against a real temporary database because its important behaviour includes queries, constraints and transactions. A mocked client cannot provide the same confidence.

Mocks are most useful when the real dependency is expensive, unreliable, slow, or outside our control, like an external API. Their value decreases as they replace more of the behaviour the test is supposed to verify. A test adds little confidence when it only proves that a mocked object returns the value configured a few lines earlier.

Following these principles may produce a small test suite or a large one. What matters is the risks it covers, the failures it can detect, and the freedom it gives us to change the implementation. The number of tests is only a consequence of those decisions, not evidence of their quality.

Further reading