The pointer represents your (pointer) finger. Single-finger motions affect the pointer. Scrolling motions affect a representation of the document. When I move my (real) finger up, my (real) finger moves up. When I put my (real) finger on a (real) document and move it up, the paper of the document moves up, causing new text from the bottom to appear in my field of view.

Or scrolling represents a viewpoint, such as a frame, or a finger pointing at the document. Then it's the other way round again. You can't just declare what the metaphor is, it's arbitrary.