I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.

Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.