▲ | yojo 5 days ago | |
"Brittle" here means either: 1) your test is specific to the implementation at the time of writing, not the business logic you mean to enforce. 2) your test has non-deterministic behavior (more common in end-to-end tests) that cause it to fail some small percentage of the time on repeated runs. At the extreme, these types of tests degenerate your suite into a "change detector," where any modification to the code-base is guaranteed to make one or more tests fail. They slow you down because every code change also requires an equal or larger investment debugging the test suite, even if nothing actually "broke" from a functional perspective. Using LLMs to litter your code-base with low-quality tests will not end well. |