I wonder if the dnhkng results could be correlated to reasoning in first order logic / set theory notation?

There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.