You cannot treat light as particles in that scenario. The primary wave gets absolutely and completely delayed, with no part getting ahead. It's not some photons doing something with a certain probability and then causing a macroscopic effect once the probability goes towards 1 once you passed sufficient matter.

What Feynman does (where this confusion comes from) is that you can look at discrete wave packets (i.e. photons) and the math comes out the right way for the primary wave if you assume that only some of these wave packets get phase-shifted, and add all elementary waves together afterwards.

But still, it's photons as "wave packets" that influence the whole system, not photons as independent particles that either bounce on something or don't.