This is something not talked about enough. I think it's one of those things where people really struggling with weight should be told CICO (calories in calories out) just to get their head in the right place but it's not strictly true for all the reasons you mentioned and others

FWIW, this sort of thing is constantly being talked about on the Zoe blog: https://zoe.com/blog