I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other.

https://youtu.be/wjZofJX0v4M?si=QEaPWcp3jHAgZSEe&t=802

This even opens up a more data-based approach to linguistics, where it is also heavily used.