There's an earlier paper [0] involving the same authors which explains this a bit better.

AIUI, they use the 3x3 neighbourhoods to capture local directional and curvature (i.e. gradient) information in the distance matrix. They then apply two heuristics (reduction to an 8-bit binary number and binning into sextiles) to reduce the floating point gradient information to coarse integers to aid pattern recognition.

The more recent paper adds another heuristic (empirically chosen similarity threshold) to aid finding starting points of recurring patterns.

[0] https://doi.org/10.1038/s41531-021-00240-4 , Equation (5) onwards.