If you don't push your system to failure, you can't really say you understand it. And anyways the precise failure modes under various conditions are important characteristics for stability/resiliency. (Does it shed load all the way upto network bandwidth of SYNs; allocate all the memory and then exit; freeze up with deadlocks/disk contention; go unresponsive for a few minutes then recover if the traffic dies off; answer health check pings only and not progress on actual work).

If you write your tests the Test-Driven Development way in that they first fail before production changes are introduced, you will be able to trust them a lot more. Especially if they are well-written tests that test behavior or contracts, not implementation details. I find that dependency injection helps a lot with this. I try to avoid mocking and complex dependencies as much as possible. This also allows me to easily refactor the code without having to worry about breaking anything if all the tests still pass.

When it comes to agentic coding. I created an open source tool that enforces those practices. The agent gets blocked by a hook if it tries to do anything that violates those principles. I think it helps a lot if I may say so myself.

https://github.com/nizos/tdd-guard

Edit: I realize now that I misunderstood your comment. I was quick to respond.

We can't really property test most code. So it comes down, as with everything, to good judgement and experience.

The difference with proofs (whether formal or informal) is that they quantify over all possible cases, whereas testing is always limited to specific cases.

It helps some. There are plenty of errors, a large majority I'd say, where types don't help at all. Types don't free up memory or avoid off-by-one errors or keep you from mixing up two counter variables.

Acceptance criteria are often buggy themselves, and require more context to interpret and develop a solution.

If your acceptance criteria state something like “produces output f(x) for any inout x, where f(x) is defined as follows: […]”, then you can’t possibly test that, because you can’t test all possible values of x. And if the criteria don’t state that, then they don’t cover the full specification of how the software is expected to behave, hence you have to go beyond those criteria to ensure that the software always behaves as expected.

You can’t prove that something is correct by example. Examples can only disprove correctness. And tests are always only examples.

Because AC don’t cover non-functional things like maintainability/understandability, adherence to corporate/team standards etc.

I've been tasked with doing a very superficial review of a codebase produced by an adult who purports to have decades of database/backend experience with the assistance of a well-known agent.

While skimming tests for the python backend, I spotted the following:

    @patch.dict(os.environ, {"ENVIRONMENT": "production"})
    def test_settings_environment_from_env(self) -> None:
        """Test environment setting from env var."""
        from importlib import reload

        import app.config

        reload(app.config)

        # Settings should use env var
        assert os.environ.get("ENVIRONMENT") == "production"

I have plenty of anecdata that counters your anecdata.

LLMs can generate code that works. That much is true. You can generate sufficiently complex projects that simply run on the first (or second try). You can even get the LLM to write tests for the code. You can prompt it for 100% test coverage and it will provide you exactly what you want.

But that doesn't mean OP isn't correct. First, you shouldnt be remembering everything. If you are finding yourself remembering everything your project is either small (I'd guess less than 1000 lines) or you are overburdened and need help. Reasoning, logically, through code you write can be done JIT as you're writing the code. LLMs even suffer from the same problem. Instead of calling it "having to remember to much" we refer to it as a quantity called "context window". The only problem is the LLM won't prompt you telling you that it's context window is so full it can't do it's job properly. A human will.

I think an engineer should always be reasoning about their code. They should be especially suspicious of LLM generated code. Maybe I'm alone but if I use an LLM to generate code I will review it and typically end up modifying it. I find even prompting with something like "the code you write should be maintainable by other engineers" doesn't produce good value.

My jaw hit the table when I read that. Just checking here but, are you being serious?

This approach sounds like a great way to get a lot of security holes into your code. Maybe your competitors will be faster at first, but it’s probably better to be a bit slower and not leaking all your users data.

How often do you buy stuff that doesn't work, and you are OK with the provider telling you "we had a reasonable belief that it worked"?

How are we supposed to use software in healthcare, defense, transportation if that's the bar?