| ▲ | Durable queues, streams, pub/sub, and a cron scheduler – inside your SQLite file(honker.dev) |
| 123 points by ferriswil 6 hours ago | 27 comments |
| |
|
| ▲ | tptacek 3 hours ago | parent | next [-] |
| "Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher". |
| |
| ▲ | felooboolooomba 3 hours ago | parent | next [-] | | "one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant. | | |
| ▲ | giraffe_lady 2 hours ago | parent [-] | | Talking out your ass, a select to this table in sqlite is like a couple hundred microseconds. Fossil uses hundreds of individual queries to build pages and still completes in under 30 ms. It's not exactly charming but it is not a performance problem. https://sqlite.org/np1queryprob.html | | |
| ▲ | rv64imafdc 2 hours ago | parent | next [-] | | Hold on -- if it really is "one lightweight SELECT per millisecond", and you're saying a select is "a couple hundred microseconds", say generously 200us?, then you're spending 200us out of every 1000us just selecting. That's a lot of polling! | | |
| ▲ | giraffe_lady an hour ago | parent [-] | | I mean only in the same sense that you spend 1 second per second doing something. Time is probably not the best way to evaluate the resources this consumes and I doubt it takes much of anything else either. It does seem weird though even for sqlite. I wonder how oban does it. I also wonder if OP knows oban can run on sqlite. |
| |
| ▲ | tptacek 2 hours ago | parent | prev [-] | | Yeah, again, to be clear: I get how SQLite works and I'm not dunking on the design, I'm just saying the comparison set up on this page snags. It's a classic LLM negated triptych, but "one of these things is not like the other": cache pressure: bad, writer contention: bad, kernel file watcher: ... good, actually? Intuitively seems better than this design? |
|
| |
| ▲ | ncruces 3 hours ago | parent | prev | next [-] | | If you're not making any changes to the database, does the SELECT "kill" you? And if you are making changes, don't you have to poll regardless after the file watcher wakes you? For WAL mode, SQLite can probably satisfy this query just by inspecting some shared memory. But it is busy waiting, sure. | |
| ▲ | d1l 3 hours ago | parent | prev [-] | | Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis. | | |
| ▲ | koito17 2 hours ago | parent | next [-] | | For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling. k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't seem to be excessive CPU usage despite dozens of watches existing in the simulated etcd. Of course, this doesn't say much about Honker, but it's nonetheless worth pointing out that sometimes the choice of database forces one towards a certain design. [1] https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructur... | | |
| ▲ | jallmann an hour ago | parent [-] | | With SQLite, you're basically funneled towards a single-writer / single-process design anyway ... in which case why not use a more traditional condvar + mutex rather than polling? |
| |
| ▲ | tptacek 2 hours ago | parent | prev | next [-] | | I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design. | | |
| ▲ | giraffe_lady 2 hours ago | parent [-] | | Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit. | | |
| ▲ | tptacek 2 hours ago | parent [-] | | Maybe, I'm really writing about the language on this page, not about the design (I responded about this upthread). | | |
|
| |
| ▲ | andai 2 hours ago | parent | prev [-] | | What's the CPU usage? Like 2%? I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU. Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.) |
|
|
|
| ▲ | wmanley 14 minutes ago | parent | prev | next [-] |
| I've implemented something similar in the past, but using inotify. You need to watch the -wal file for IN_MODIFY. To make it work reliably I found I had to run: BEGIN IMMEDIATE TRANSACTION; ROLLBACK;
Otherwise the new changes weren't guaranteed to be visible to the process. I'm sure there's a more targetted approach that would work instead - maybe flock on a particular byte in the `-shm` file. |
|
| ▲ | itopaloglu83 4 hours ago | parent | prev | next [-] |
| It’s an interesting approach and can be quite fun to use for new projects. > How it works: honker polls SQLite’s PRAGMA data_version every millisecond. That’s a monotonic counter SQLite increments on every commit from any connection, journal mode, or process — a ~3 µs read for a precise wake signal. |
|
| ▲ | EvanAnderson 4 hours ago | parent | prev | next [-] |
| Prior discussion a few days ago: https://news.ycombinator.com/item?id=47874647 |
|
| ▲ | vmsp 3 hours ago | parent | prev | next [-] |
| Reminds me of Litestack for Rails. Eventually, it was abandoned because Rails itself started going all out on SQLite. https://github.com/oldmoe/litestack |
| |
|
| ▲ | arlobish 3 hours ago | parent | prev | next [-] |
| At the end it says: "pg-boss and Oban are the Postgres-side gold standards" -- but Oban supports SQLite now too https://github.com/oban-bg/oban |
| |
|
| ▲ | maxdo an hour ago | parent | prev | next [-] |
| Almost feels like someone is trying to joke about similar postgres application . To make it look even more absurd . SQLite is not concurrent and you’ll have tons of problems using it practically . |
|
| ▲ | deferredgrant an hour ago | parent | prev | next [-] |
| This seems especially appealing in the awkward middle: too serious for in-memory queues, not big enough to justify Kafka-shaped machinery. |
|
| ▲ | andrewstuart an hour ago | parent | prev | next [-] |
| Suggestion for the author wind back the polling to once a second when nothing is happening. |
|
| ▲ | andrewstuart an hour ago | parent | prev | next [-] |
| I can’t see any benchmarks or performance stats. I’d like to see messages per second. |
|
| ▲ | canadiantim 2 hours ago | parent | prev [-] |
| Could this work with Turso, the SQLite rust rewrite? |