The Yakuza seems to have a similar story as the American Mafia. Both have long histories, are favourite subjects of films and media, and both had a decline that sharpened in the 90s. A large part of that has been increasingly tight anti-Yakuza laws and ordinances. The whole "Law enforcement and indirect enforcement" section on Wikipedia is an interesting read, I linked part of it:

https://en.wikipedia.org/wiki/Yakuza#Current_situation

I do agree with the justice/prison system being incredibly scary, though!