| ▲ | westurner an hour ago | |
Like becoming a manager? When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce. Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax Managers and developers are responsible for setting a test coverage threshold for merge approval. If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification? How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? | ||
| ▲ | pjc50 39 minutes ago | parent | next [-] | |
Formal verification is always more valuable than mere testing, but it's hitherto more expensive. The thing that ultimately matters is closing the loop: how well do the tests match the requirements, both as written and as unwritten in the mind of the customer? Working with an LLM has given me a real eye opener on unwritten requirements. It's like outsourcing. "Yes, you've given me what I wrote down, but I never expected you do to it in that way" | ||
| ▲ | tablarasa an hour ago | parent | prev [-] | |
> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure. | ||