While it's a common joke, not quite - if you're hit by baseball, your velocity does not change significantly as energy is dissipated in other ways, and the change in velocity of the baseball is a consequence of the lost energy, not a cause of damage.

Dissipation of energy hurts you. :)