> Note there is no intrinsic reason running multiple streams should be faster than one.

The issue is the serialization of operations. There is overhead for each operation which translates into dead time between transfers.

However there are issues that can cause singular streams to underperform multiple streams in the real world once you reach a certain scale or face problems like packet loss.

In general TCP just isn't great for high performance. In the film industry we used to use a commercial product Aspera (now owned by IBM) which emulated ftp or scp but used UDP with forward error correction (instead of TCP retransmission). You could configure it to use a specific amount of bandwidth and it would just push everything else off the network to achieve it.

> has a hardcoded maximum receive buffer of 2MiB

For completeness, I want to add:

The 2MiB are per SSH "channel" -- the SSH protocol multiplexes multiple independent transmission channels over TCP [1], and each one has its own window size.

rsync and `cat | ssh | cat` only use a single channel, so if their counterparty is an OpenSSH sshd server, their throughput is limited by the 2MiB window limit.

rclone seems to be able to use multiple ssh channels over a single connection; I believe this is what the `--sftp-concurrency` setting controls.

Some more discussion about the 2MiB limit and links to work for upstreaming a removal of these limits can be found in my post [3].

Looking into it just now, I found that the SSH protocol itself already supports dynamically growing per-channel window sizes with `CHANNEL_WINDOW_ADJUST`, and OpenSSH seems to generally implement that. I don't fully grasp why it doesn't just use that to extend as needed.

I also found that there's an official `no-flow-control` extension with the description

> channel behaves as if all window sizes are infinite. > > This extension is intended for, but not limited to, use by file transfer applications that are only going to use one channel and for which the flow control provided by SSH is an impediment, rather than a feature.

So this looks exactly as designed for rsync. But no software implements this extension!

I wrote those things down in [4].

It is frustrating to me that we're only a ~200 line patch away from "unlimited" instead of shitty SSH transfer speeds -- for >20 years!

[1]: https://datatracker.ietf.org/doc/html/rfc4254#section-5

[2]: https://rclone.org/sftp/#sftp-concurrency

[3]: https://news.ycombinator.com/item?id=40856136

[4]: https://github.com/djmdjm/openssh-portable-wip/pull/4#issuec...

> It almost always indicates some bottleneck in the application or TCP tuning.

Yeah, this has been my experience with low-overhead streams as well.

Interestingly, I see a ubiquity of this "open more streams to send more data" pattern all over the place for file transfer tooling.

Recent ones that come to mind have been BackBlaze's CLI (B2) and taking a peek at Amazon's SDK for S3 uploads with Wireshark. (What do they know that we don't seem to think we know?)

It seems like they're all doing this? Which is maybe odd, because when I analyse what Plex or Netflix is doing, it's not the same? They do what you're suggesting, tune the application + TCP/UDP stack. Though that could be due to their 1-to-1 streaming use case.

There is overhead somewhere and they're trying to get past it via semi-brute-force methods (in my opinion).

I wonder if there is a serialization or loss handling problem that we could be glossing over here?

Uhh.. I work with this stuff daily and there are a LOT of intrinsic reasons a single stream would be slower than running multiple: MPLS ECMP hashing you over a single path, a single loss event with a high BDP causing congestion control to kick in for a single flow, CPU IRQ affinity, probably many more I’m not thinking like the inner workings of NIC offloading queues.

Source: Been in big tech for roughly ten years now trying to get servers to move packets faster

Note there is no intrinsic reason running multiple streams should be faster than one

If the server side scales (as cloud services do) it might end up using different end points for the parallel connections and saturate the bandwidth better. One server instance might be serving other clients as well and can't fill one particular client's pipe entirely.

Wouldn't lots of streams speed up transfers of thousands of small files?

The author tried running rsyncd demon so it's not _just_ the ssh protocol.

Single file overheads (opening millions of tiny files whose metadata is not in the OS cache and reading them) appears to be an intrinsic reason (intrinsic to the OS, at least).

I mean isn't a single TCP connections throughput limited by the latency? Which is why in high(er) latency WAN links you generally want to open multiple connections for large file transfers.

https://wintelguy.com/wanperf.pl

What kind of data analysis do you run with Go and do you use an open source library for it? My experience with stats libraries in Go has been lukewarm so far.

"Rclone is a fantastic tool, but my favorite part of it is actually the underyling FS library."

Related to this is the very useful:

  rclone serve restic ...

This howto is not rsync.net-specific - you can follow this recipe at any standard SSH endpoint:

https://www.rsync.net/resources/notes/2025-q4-rsync.net_tech...

I hope that you sponsor the rclone project given that it’s the core of your business! I couldn’t find any indication online that you do give back to the project. I hope I’m wrong.

How do you deal with how poorly rclone handles rate limits? It doesn't honor dropbox's retry-after header and just adds an exponential back off that, in my migrations, has resulted in a pause of days.

I've adjusted threads and the various other controls rclone offers but I still feel like I'm not see it's true potential because the second it hits a rate limit I can all but guarantee that job will have to be restarted with new settings.

The delta-transfer algorithm [0] is about detecting which chunks of a file differ on source and target [1], and limiting the transfer to those chunks. The savings depend on how and where they differ, and ofcourse there's tradeoffs...

You seem to be referring to the selection of candidates of files to transfer (along several possible criteria like modification time, file size or file contents using checksumming) [2]

Rsync is great. However for huge filesystems (many files and directories) with relatively less change, you'll need to think about "assisting" it somewhat (by feeding it its candidates obtained in a more efficient way, using --files-from=). For example: in a renderfarm system you would have additions of files, not really updates. Keep a list of frames that have finished rendering (in a cinematic film production this could be eg. 10h/frame), and use it to feed rsync. Otherwise you'll be spending hours for rsync to build its index (both sides) over huge filesystems, instead of transferring relatively few big and new files.

In workloads where you have many sync candidates (files) that have a majority of differing chunks, it might be worth rather disabling the delta-transfer algorithm (--whole-file) and saving on the tradeoffs.

[0] https://www.andrew.cmu.edu/course/15-749/READINGS/required/c...

[1] https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_...

[2] https://en.wikipedia.org/wiki/Rsync#Determining_which_files_...

Rclone can "sync" with a range of different ways to check if the existing files are the same. If no hashes are available (e.g. WebDAV) I think you can set it to check by timestamp (with a tolerance) and size.

Edit: oh I see, delta transfer only sends the changed parts of files?

Does rclone not do that? I thought they were specifically naming themselves similarly because they also did that.

> efficiently divide the set of files.

It turns out, fpart does just that! Fpart is a Filesystem partitioner. It helps you sort file trees and pack them into bags (called "partitions"). It is developed in C and available under the BSD license.

It comes with an rsync wrapper, fpsync. Now I'd like to see a benchmark of that vs rclone! via https://unix.stackexchange.com/q/189878/#688469 via https://stackoverflow.com/q/24058544/#comment93435424_255320...

https://www.fpart.org/

Sometimes find (with desired maxdepth) piped to gnu-parallel rsync is fine.

robocopy! Wow, blast from the past. Used to use it all the time when I worked in a Windows shop.

My go-to for fast and easy parallelization is xargs -P.

  find a-bunch-of-files | xargs -P 10 do-something-with-a-file

       -P max-procs
       --max-procs=max-procs
              Run up to max-procs processes at a time; the default is 1.
              If max-procs is 0, xargs will run as many processes as
              possible at a time.

Not just bad to use, but doesn't fully work. I've been trying to get my photos off Google Photos to backup elsewhere, but takeout misses something like 20%-30% of them.

Compression is off by default in OpenSSH, at least `man 5 ssh_config` says:

> Specifies whether to use compression. The argument must be yes or no (the default).

So I'm surprised you see speedups with your invocation.

We wrote https://github.com/chorusone/fastsync for fast ad-hoc transfers over multiple TCP streams.

zsync is better for that. zsync precalculates all the hashes and puts them in a file alongside the main one. The client downloads the hashes, compares them to what it has then downloads the parts it is missing.

With rsync, you upload hashes of what you have, then the source has to do all the hashing work to figure out what to send you. It's slightly more efficient, but If you are supporting even 10s of downloads it's a lot of work for the source.

The other option is to send just a diff, which I believe e.g. Google Chrome does. Google invented Courgette and Zucchini which partially decompile binaries then recompile them on the other end to reduce the size of diffs. These only work for exact known previous versions, though.

I wonder if the ideas of Courgette and Zucchini can be incorporated into zsync's hashes so that you get the minimal diff, but the flexibility of not having a perfect previous version to work from.

This is not a feature of rsync, but of your shell.

So the question "does rclone have that" doesn't make much sense, because it usually wouldn't be rclone implementing it.

For example, zsh does it here for rsync, which actually invokes `ssh` itself:

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

That said, some CLI tools come with tools for shells to help them implement such things. E.g. `mytool completion-helper ...`

But I don't get rclone SSH completions in zsh, as it doesn't call `_remote_files` for rclone:

https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...

rsync does what was designed to do and the lack of scope creep is not a bad thing. There is "fpsync" - another tool on top of rsync (which was mentioned in one of the comments at article's page) that covers the parallel processing use-case: https://manpages.debian.org/bullseye/fpart/fpsync.1.en.html

rclone is super cool, but unfortunately many of the providers it supports has such low ratelimits, that it's fairly difficult to use it to transfer much data at all.