Remix.run Logo
westurner an hour ago

Like becoming a manager?

When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.

Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax

Managers and developers are responsible for setting a test coverage threshold for merge approval.

If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification?

How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?

pjc50 39 minutes ago | parent | next [-]

Formal verification is always more valuable than mere testing, but it's hitherto more expensive. The thing that ultimately matters is closing the loop: how well do the tests match the requirements, both as written and as unwritten in the mind of the customer?

Working with an LLM has given me a real eye opener on unwritten requirements. It's like outsourcing. "Yes, you've given me what I wrote down, but I never expected you do to it in that way"

tablarasa an hour ago | parent | prev [-]

> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?

This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.