I think it was a combination of the two. The Apollo program was never popular. It took up an enormous portion of the federal budget, which the Republicans argued was fiscally unwise and the Democrats argued that the money should have been used to fund domestic social programs. In 1962, the New York Times noted that the projected Apollo program budget could have instead been used to create over 100 universities of a similar size to Harvard, build millions of homes, replace hundreds of worn-out schools, build hundreds of hospitals, and fund disease research. The Apollo program's popularity peaked at 53% just after the moon landing, and by April 1970 it was back down to 40%. It wasn't until the mid-80s that the majority of Americans thought that the Apollo program was worth it. Because of all this, I think it's inevitable that the Apollo program would wind down once it had achieved its goal of national prestige.

but think about that... If in the 70's they would have used the budget to build millions of homes.

The moral there is tech progress does not always mean social progress.