Sqlite over stdin, to a subprocess, and it's fast!

Twirrim 3 days ago | parent | next [-]

It's wild to me that stdin/stdout is apparently significantly faster than using the API in so many cases.

That's the kind of result that makes me wonder if there is something odd with the benchmarking.

▲

kreelman 3 days ago | parent | next [-]

That's an interesting thought. I wonder.

I wonder if the following things make the C driven version slower...

- prepare the send buffers (sqlite side)

- prepare the receive buffers (go side)

- do the call

- get the received data into go buffers of some kind

- free up the send buffers (happens automatically)

- free up the receive buffers (semi automatically in Go).

When using stdin/stdout, the system looks after send/receive buffers. It's simply reading/writing them. No allocation is needed. The stream can be as big or as little as wanted/needed. The OS will look after the integrity of the streams and these are probably fairly well tested subsystems on most operating systems.

stdin/stdout becomes a "library" for "fast data transfer".

Pretty neat.

	▲	raggi 3 days ago \| parent [-]
		fwiw, the tailscale fork of Crawshaw’s library has a good number of allocation removals and other optimizations, but cgo is still expensive.

▲

kitd 3 days ago | parent | prev [-]

And presumably that implies there's OS context switching going on underneath.

Still, I can see a few downsides. Though sqinn-go is pure Go, the forked process is pure C, so you'll need to either download a prebuilt one (Linux and Windows only atm), or build it yourself. This rather defeats the benefits of Go's killer feature of "single-binary distribution".

Still, I agree it's wild it is so fast.

▲

karel-3d 3 days ago | parent | prev | next [-]

note that the author of the benchmark is also author of this library.

▲

wener 3 days ago | parent | prev | next [-]

I used to use sqlite3 with stdio to read VoIP SQLite data. It's difficult or impossible to get a compatible SQLite version, and it's also hard to use cgo. I want to read the SQLite data on the server, and stdio is the only choice.

▲

sureglymop 3 days ago | parent | prev [-]

That's an interesting approach but doesn't it mean that if you want multiple connections at the same time, you'd need multiple subprocesses?

Perhaps I misunderstand though.

	▲	cvilsmeier 2 days ago \| parent [-]
		You are correct: Multiple connections means multiple sqinn subprocesses.