Many times I've looked at the output of a regression model, seen this effect, and then thought my model must be very bad. But then remember the points made elsewhere in thread.

One way to visually check that the fit line has the right slope is to (1) pick some x value, and then (2) ensure that the noise on top of the fit is roughly balanced on either side. I.e., that the result does look like y = prediction(x) + epsilon, with epsilon some symmetric noise.

One other point is that if you try to simulate some data as, say

y = 1.5 * x + random noise

then do a least squares fit, you will recover the 1.5 slope, and still it may look visually off to you.

Maybe comparing plots of residuals makes it clearest.