Skin damage from UV is cumulative. How careful you have to be depends on your skin type but the general advice is that sun protection should be worn when the UV index is over 3, or if you're spending any extended time in the sun.