He's saying that the plum is not infinite because the plum is not real, only the perception of the plum is real and implying that the same is true of time.

It's an argument that is akin to, but not the same as, "the map is not the territory."