"Attention" is a feature that makes this whole thing work in the first place, it's not a flaw, although all current models are non-ideal at it in practice. Could be better for sure :)

Right but when the context window fills up, whatever miniscule guardrails are there magically disappear - even if this is by design, this is bad. Especially when it was marketed as magic