Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.

And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.

You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).

>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.