When you have a conventional gravity assist, the speed (magnitude) is unchanged in the coordinate frame of the planet that's providing the assist. All that happens is the velocity vector is rotated in that frame. Thus the usefulness hinges on arriving with a large relative velocity, to start with—a large vector to rotate, allows for a large velocity change.

There's no heliocentric velocity in a slow-moving outer planet.