Remix.run Logo
▲ shoo 2 hours ago

> Running the suite in CI on every build would have caught a lot of this

yeah, provided:

  1. test cases are being designed to cover functionality
  2. test cases are implemented correctly - i.e. they actually fail if the implementation is defective
  3. the tests aren't all commented out
  4. the test suite fails if a test fails
  5. CI actually detects test suite failure
  6. failing tests block changes from being merged (apart from initial triage to stabilise the suite)

Presumably a few of these things weren't being done -- I'd bet 1 & 2 were skipped or implemented so poorly that they were worse than having no tests at all -- checking nothing but giving a false illusion that some kind of testing had happened.

Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.