I've never seen an argument like this that, if true, wouldn't also apply to the cognitive offloading we do by relying on culture, by working with others, or working with the artifacts built by others.

Cognitive offloading via culture has many forms and many of them are not sustainable at all.