Find the equilibrium probabilities from first eigenvector, use definition of entropy like you did. If it's not ergodic then you get subsystems with their own entropies. Second eigenvalue iirc basically characterizes dominant relaxation time.

This is all if you mean the equilibrium entropy of the underlying system, not the entropy rate.