It's difficult to design a language which has good usability both with and without a GC. Can users create a reference which points to the interior of an object? Does the standard library allocate? Can the language implement useful features like move semantics and destructors, when GCed objects have an indefinite lifetime?

You'd almost end up with two languages in one. It would be interesting to see a language fully embrace that, with fast/slow language dialects which have very good interoperability. The complexity cost would be high, but if the alternative is learning two languages rather than one...

I'm not saying you design a language with an optional GC, I'm saying the user can implement their own allocators i.e. large object pools nested in the GC-managed memory system. And then they get to avoid most of the allocation and deallocation overhead during runtime.