Back in perl 5.8 and older, the regex engine worked by recusing at the level of C functions, and so you could trigger stack overflows by giving it large strings.

Is this similar?

I wonder why gitlab even has its own regex compiler. Or is this a vulnerability in an underlying library or programming language?

Looks like Gitlab ships its own forked copy of the Ruby Onigmo regex engine.