This is like a type theory question - do you like it untyped or typed?

In physics, values have units too. Analogously, you could say - why incorporate units into the algebra in physics (as is often done)? Why not just add scalars etc. and not bother carrying around the units everywhere?

Well, because doing anything else is mostly nonsensical - it does not make sense to add meters and seconds together. Using unit algebra is the most basic sanity check as to whether your formula makes any sense.

Sometimes it makes sense to convert/cast between representations, but that should be explicit - distinguishing eg. objects and operations is more readable and more safe, and only comes with a bit of notational overhead. Nothing is free, but I think the benefits far outweigh the downsides.