Is it even possible to actually learn the language without living in the environment, where it gets in your face all the time?

Yes. Linguists do that all the time. People do learn Latin, etc.

Disclaimer: I'm not an expert in anything, let alone linguistics or language acquisition. Just reporting what I've found.

Effective fluency (C2+) seems to be really, really difficult without immersion.

Anything past A1 seems to be really difficult without dedication and active effort. Pimsleur is my foundation and I think it's debatable whether that will get you out of A1 by itself.

I do Pimsleur (practical listening and speaking and reading), InnerFrench (listening comprehension, roughly A2 where I'm at right now but I think it progresses through B1 and B2 as he speaks faster and rephrases things less for ease of understanding), and reading (_A Game of Thrones_, currently, which is handy because I've read the books in English before and can use that as leverage).

I also watch some French films, but that's less helpful at present. I can't always find good French subtitles for my (pirated) films and because I watch a decent number of foreign films (especially Polish, shoutout to Wojciech Has enjoyers), the trying-to-learn-French part of my brain mostly shuts off. This is just a place where I need to put in more effort.

I have a lot of other French language podcasts and some YouTube channels cued up, but I'm not ready for most of them.

And this doesn't really address two of the major components, which are writing and speaking French. I don't have much going on there. I plan to add 2-3x/week iTalki next year so that I get feedback from fluent French speakers.

So TL;DR: your skepticism is absolutely justified and it's an uphill battle, a significant challenge and a sink of time, money, and willpower. I've been doing it for a year, on top of three years in high school and another year in college, and I think it's getting harder, not easier.

So we'll see.