Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to.

As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?

▲

greaber 3 hours ago | parent | next [-]

I entirely agree that multiple connections shouldn't actually be necessary for speed, but they help in multiple situations (long-distance transfers, same DC between servers, NFS), and it's not just about amortizing per-file setup overhead.

There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful. I will work on one. On the other hand, I also tried hard to make it just go fast without needing the user to understand why it is fast or tune anything. For instance, the number of connections is auto-tuned by default.

	▲	AceJohnny2 2 hours ago \| parent [-]
		> There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful First, thank you for taking the feedback :) To be clear, I'm focusing on the first thing you should be telling users. It seems that, as a CLI program, the target audience of this tool is fairly technical users who are likely already familiar with SSH, rsync, and everything. As demonstrated in this thread, your first communication task is to convince them why it's worth their time to use your tool, considering they're probably satisfied with the pre-existing ones (or a replacement would already exist!). My point is, explaining why it's better/faster than rsync shouldn't be buried in the docs. It should be a paragraph on the front page. Hope this helps, and keep it up. HN is a rough audience to go public on ^^;

▲

ranger_danger 2 hours ago | parent | prev [-]

> I would've expected a single TCP connection to be able to saturate a line

There's several factors at play that make this (usually) not the case.

Besides physical latency (which includes those added by any VPNs/tunnels/etc., some of which may be internal to an ISP along the route and outside of your control), there's other things like the TCP window sizes / window scaling option[1] that can affect single stream performance, and those type of parameters can differ by OS/interface type on both ends.

Also for SSH specifically, it has its own fixed buffer size that also limits throughput unless you're using the HPN-SSH fork[2].

[1] https://en.wikipedia.org/wiki/TCP_window_scale_option

[2] https://www.psc.edu/hpn-ssh-home/introduction/

▲

greaber 2 hours ago | parent [-]

Also, in syq, a connection is the unit of independent transfer work. Probably it would be possible to get higher bandwidth per TCP connection by some kind of multiplexing, but that architecture would be more complex and probably not have many benefits.

	▲	ranger_danger 2 hours ago \| parent [-]
		There's also MPTCP (multipath TCP) which is even supported on mobile devices these days. So for example if your device has both a wifi and a cellular connection, and both ends of the stream support MPTCP, you can increase throughput/failover by combining multiple interfaces.