> what am I missing here?

There are gotchas, for instance Chrom,{e,ium} insists on XDG_DESKTOP_DIR != XDG_DOWNLOAD_DIR.

See this bug report from a confused user: https://issues.chromium.org/issues/41076564

> also of note, we (mostly) don't allow ~/Desktop as the download dir for security reasons

This isn't an XDG issue. It's a chromium engineers being silly pricks that think they know better than the power users who obviously went out of their way to create such a configuration. Also I bet it would work if you set your XDG_DESKTOP_DIR to ~/Download/