Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

▲

koito17 4 hours ago | parent | next [-]

For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling.

k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't seem to be excessive CPU usage despite dozens of watches existing in the simulated etcd. Of course, this doesn't say much about Honker, but it's nonetheless worth pointing out that sometimes the choice of database forces one towards a certain design.

[1] https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructur...

	▲	jallmann 3 hours ago \| parent \| next [-]
		With SQLite, you're basically funneled towards a single-writer / single-process design anyway ... in which case why not use a more traditional condvar + mutex rather than polling?
	▲	sroussey 42 minutes ago \| parent \| prev [-]
		Are you trying to avoid sleep?

▲

tptacek 4 hours ago | parent | prev | next [-]

I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.

▲

giraffe_lady 4 hours ago | parent [-]

Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.

▲

tptacek 4 hours ago | parent [-]

Maybe, I'm really writing about the language on this page, not about the design (I responded about this upthread).

	▲	giraffe_lady 4 hours ago \| parent [-]
		Oh, yes, I see what you mean now.

▲

andai 4 hours ago | parent | prev [-]

What's the CPU usage? Like 2%?

I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU.

Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.)

▲

booi an hour ago | parent [-]

i mean, technically this is once per millisecond, so this would happen 1000x more. In your case due to the kernel overhead you would likely not even be able to do it (300% CPU?).

Either way this does seem like a very large overhead due to the fact that there's just no other way to do it without a deeper kernel integration which might be outside the scope of what sqlite is trying to do.

	▲	nine_k a few seconds ago \| parent [-]
		If the fs tree scanned once per second had 1000 files, it would be once per millisecond for a file.