In terms of understanding why something is like that, Linear Algebra belongs to Geometry more than Algebra. Every formula in Linear Algebra which ultimately is justified by "it's just that way" can be better justified by geometry. The algebraic formulas are like animals who lost their natural habitat and were put in a zoo, making people think that these animals evolved in the zoo itself.

To give an example: A simple multiplication of two numbers is better seen as rotating one of the numbers to be perpendicular to the other and then quantifying the area/volume spanned by them. This gives vector dot product.

While geometry might better address "why", algebra gets into the work of "how to do it". Mathematics in old times, like other branches of science, did not encourage "why". Instead, most stuff would say "This is how to do it, Now just do it". Algebra probably evolved to answer "how to do it" - the need to equip the field workers with techniques of calculating numbers, instead of answering their "why" questions. In this sense, Geometry is more fundamental providing the roots of concepts and connecting all equations to the real world of spatial dimensions. Physics adds time to this, addressing the change, involving human memory of the past, perceiving the change.

I remember how betrayed I felt when I got to calculus and realized all of those equations I learned in high school for physics were just introductory calculus. Like why did we have to learn all that the hard way?

Not just Linear Algebra but every branch of Science/Mathematics should be taught as much as possible using Geometry (and other visualizations) before being mapped to abstract algebraic symbols. Basic Geometry (along with simple Arithmetic) is the oldest subfield of Mathematics for the reason that most of its concepts are intuitive and feels "natural" and mappable to the "Real World" by us Humans. While abstraction via symbol manipulation is necessary to generalize and extend mathematics it should come at a later stage after we have developed some intuition of the concepts being represented by the symbols, however limited/restricted they might be. All abstraction requires some mathematical maturity which can only happen over time.