Linear algebra can indeed not do it. You need non-linearity to get the expressivity that we see in LLMs.