I ran into this exact same thing recently with CSVs downloaded from my own app. I tried a few different filetypes and was baffled how seemingly any filetype I downloaded triggered Gatekeeper regardless of the app I set to open it (including stock apps).

I eventually found on Reddit that setting the default via the Get Info dialog was the only path that worked, so now I can click a CSV and open it in VS Code without needing to send Apple my passport and fingerprints. I keep seeing mixed opinions whether it's a bug that Get Info associations work differently vs the right click context menu, or if it's a deliberately obtuse garden path like the Settings/Open Anyway routine and "working" as intended.

Either way I hate it but it would be slightly more forgivable as a bug (assuming it was then fixed).