This is why a test suite and mock application running on the host is so important. Tools like valgrind can be user to validate that you won't have any memory errors once you deploy to the platform that doesn't have protections against invalid accesses.

It wouldn't have caught your issue in this case. But it would have eliminated a huge part of the search space your embedded engineers had to explore while hunting down the bug.

Custom OS, cross compiling from Windows, using Arm's old C compiler so tools like valgrid weren't available to us.

Since it was embedded, no malloc. Everything being static allocations made the search possible in the first place.

This wasn't the only HW bug we found, ugh.