inside Alan's cluttered head
tar + netcat = very fast copy
I reformatted a hard disk this weekend. In the process, I needed to copy a bunch of files from one machine to the other. Since both of these machines were smaller embedded devices, neither one of them had very capable CPUs. So I wanted to copy all of the files without compression or encryption.
Normally, I would use “rsync -avz --delete --progress user@other:/remote/path/ /local/path/“, but this does both compression (-z) and encryption (via rsync-over-ssh).
Here’s what I ended up with. It did not disappoint.
Step 1 – On the machine being restored:
box1$ netcat -l -p 2020 | tar --numeric-owner -xvf -
Step 2 – On the machine with the backup:
box2$ tar --numeric-owner -cvf - | netcat -w3 box1 2020
2 comments
Leave a Reply
You must be logged in to post a comment.
hi, [edited] reminds me of a similar script I came across online. It is more automatic, but when I run it, it always prompts that tar can’t recognize the file format. Could you take a look and hopefully fix it?
http://intermediatesql.com/linux/scrap-the-scp-how-to-copy-data-fast-using-pigz-and-nc/
Thank you.
Hi Sam, sorry it took so long to get to this, but the comment landed in an email folder that I seldom check. It sounds to me like whatever process is feeding tar is running out of input, so all tar sees is an empty stream. It’s hard to tell where exactly you went off the rails in your experiment. But thanks for sharing this interesting article about faster copy. I love the name “pigz”!