> Yes, that's the kind of contract we take on regularly. My point is that the exact thing the customer wants/needs is the one part that can't be formalized meaningfully

I think that's fine though. The exact fuzzy thing the customer wants is typically still built on top of a ton of stuff that can be verified. I'm not sure it needs to be either-or. You formalize what you can, and do your best with what you can't. Ideally (if we could find a way to make formalized software cheaper/easier to build) it would reduce time wasted on the stuff that can be formalized, and allow spending more time on the stuff that can't be.