Well empiricists think knowledge exists in the environment and is absorbed directly through the eyes and ears without interpretation, if we're being uncharitable.

Sure. The idea of raw, uninterpreted "sense data" that the empiricists worked with (well into the 20th century) is pretty clearly bunk. Much of philosophy took a turn towards anti-foundationalism, and rationalism and empiricism are, at least classically, notions of the "foundations" of knowledge. I mean, this is philosophy, it's all pretty ridiculous.