The annual consumption of ice cream in Japan was 6.7 litres per person in 2021 (compared to 10 litres/person in Canada and 20 litres/person in the U.S.). For all dairy, Japanese people each ate 94 kg in 2022.

They eat less dairy, but hardly none. I have heard people say that a scoop of ice cream or a glass of milk each day is not a problem, but more can be. Intolerance also seems to increase with age, so younger people can consume more dairy.

A 1975 study in Japan puts intolerance (unable to drink 200ml of milk comfortably) at 19% of the population. I would suspect that massive exposure over the past 50 years has lowered that percentage significantly.