Are eigenvalues or singular values used much in the popular recent stuff, like LLMs?

LoRa uses singular value decomposition to get the low rank matrices. In different optimizers, you'll also see eigendecomposition or some approximation used (I think Shampoo does something like this, but it's been a while).