You can bypass the sec-fetch headers via service workers i think.

A better approach here would be to just serve svg with Content-security-policy: script-src 'none'; sandbox

But you can't make a link to https://your.domain/my_phishing_page.svg work as a phishing page using service workers unless you've pretty thoroughly pwned the site already. (And you can constrain what gets to run as a service worker using Sec-Fetch-Dest!)

I suppose an actual exception is Content-Disposition. If you want the user to save a file, you need to serve it with dest == document as far as I know.