Show HN: Syq – copy files between machines fast (better than rsync)
I got frustrated with the slowness of rsync and made an alternative that works faster by using multiple parallel connections, direct encrypted TCP when available, and other optimizations. I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs, the ability to download to your laptop while working in an ssh shell on a server, and the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). It is also designed to be more robustly/flexibly scriptable than alternatives.
Discussion Highlights (8 comments)
mika6996
Why is this better than rsync?
cynicalsecurity
I'm eager to replace battle-tested rsync with a new half-baked tool that no one uses for a tiny speed improvement. I'm also going to enable automatic updates of that tool too.
hikarudo
Very cool! How does the direct TCP mode work?
hxseven
I thought I'd share a more positive comment. I work on a tool in a similar area (moving stuff around) but on a higher level. I currently use rsync or rclone for the file moving parts. So to me your tool looks interesting and promising, thanks for sharing it :) Some suggestions for improvement for your website/Github: - I wanted to understand how it differs from rsync. A comparison page would be helpful. - A comparison to rclone would also be interesting, as it also uses multiple parallel connections. - As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.
unsnap_biceps
> I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's. > the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.
AceJohnny2
Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to. As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?
jttnr
Open page, read headline, read first sentence, emdash, close page.
Zenul_Abidin
Wow, interesting! How is the performance compared to rclone though?