Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?

▲

danielmarkbruce 3 days ago | parent [-]

Brush up :)

The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.

▲

YeGoblynQueenne 3 days ago | parent [-]

Alright, I'll have to check up on that. Thanks for being nice about it.

▲

YeGoblynQueenne 3 days ago | parent [-]

Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.

To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).

So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.

Sutton & Barto 2ed is free here btw:

https://web.stanford.edu/class/psych209/Readings/SuttonBarto...

The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.

▲

danielmarkbruce 2 days ago | parent [-]

Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.

And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.

▲

YeGoblynQueenne 2 days ago | parent | next [-]

You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).

>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.

I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.

	▲	danielmarkbruce 2 days ago \| parent [-]
		But the next move is self evident if you have a prediction of the value of being in each of the states possible.

▲

2 days ago | parent | prev [-]

[deleted]