The advent of agriculture did not provide better food, it was just the only solution to avoid extinction due to the lack of food.

The archaeological evidence shows that for many generations the first neolithic farmers had serious health problems in comparison with their ancestors. Therefore it is quite certain that they did not transition to agriculture willingly, but to avoid starvation.

Later, when the agriculturalists have displaced everywhere the hunter-gatherers, they did not succeed to do this because they were individually better fed or stronger or smarter, but only because there were much more of them.

The hunter-gatherers required very large territories from which to obtain enough food. For a given territory size, practicing agriculture could sustain a many times greater population, and this was its advantage.

The maximum human brain size had been reached hundreds of thousands years before the development of agriculture, and it regressed a little after that.

There is a theory, which I consider plausible, that the great increase in size of the human brain has been enabled by the fact that humans were able to extract bone marrow from bones, which provided both the high amount of calories and the long-chain fatty acids that are required for a big brain.

I've seen the bone marrow hypothesis also, which is very interesting. Afaik. evidence shows at least that there was enough specialization during neolithic era to have bone marrow cooks where the hunters delivered their bones. Something you wouldn't expect based on just school knowledge (at least back in 90s/2000s).

I see your point about agriculture at first degrading quality of food. Are you aware of evidence of brain size degrading even? Is it visible in the temple bones?