This is pretty cool, I wasn’t aware of these types of challenges. How does one even approach this?

Feels to me like it’s similar to dumping a binary with an image, the format being entirely custom.

And/or trying to decode a language or cipher, trying to recognize patterns.

Study math/statistics/ML at a graduate level, to start.

TFA details a solution, it's pretty interesting. Basically the problem was to reverse engineer an absurdly obfuscated and slightly defect MD5 algorithm.