This article goes through great lengths to basically say your agent should probably not obey headers saying random things. However, it is apparently it is very easy for servers to detect agents, therefore would be trivial to serve them different content in the res.text() than to a human user. This doesn't even need to be adversarial, it makes sense to send back a stripped down version to agents for both the server and the agent ingesting the data, but becomes a problem for anyone who wants to show different content to the agents.

Curious: What are the trivial agent vs. human detection mechanisms?