I think the kind of judgement required here is to design ways to test the code without inspecting it manually line by line, that would be walking a motorcycle, and you would be only vibe-testing. That is why we have seen the FastRender browser and JustHTML parser - the testing part was solved upfront, so AI could go nuts implementing.

▲

northfield27 6 hours ago | parent [-]

I partially agree, but I don’t think “design ways to test the code without inspecting it manually line by line” is a good strategy.

Tests only cover cases you already know to look for. In my experience, many important edge cases are discovered by reading the implementation and noticing hidden assumptions or unintended interactions.

When something goes wrong, understanding why almost always requires looking at the code, and that understanding is what informs better tests.

▲

visarga 5 hours ago | parent [-]

Another possibility is to implement the same spec twice, and do differential testing, you can catch diverging assumptions and clarify them.

	▲	northfield27 5 hours ago \| parent [-]
		Isn't that too much work? Instead, just learning concepts with AI and then using HI (Human Intelligence) & AI to solve the problem at hand—by going through code line by line and writing tests - is a better approach productivity-, correctness-, efficiency-, and skill-wise. I can only think of LLMs as fast typists with some domain knowledge. Like typists of government/legal documents who know how to format documents but cannot practice law. Likewise, LLMs are code typists who can write good/decent/bad code but cannot practice software engineering - we need, and will need, a human for that.