Fastest way to transfer files between two Linux boxes
I have two Linux machines and a gigantic file on one that I want on the other. The two machines are connected by Ethernet. scp is incredibly slow, about 1.6MB/second. I don't feel like setting up samba or nfs on either. I don't care about security, authenticity, or integrity; I can md5sum if necessary, and both machines are isolated from the Internet.
I assumed the scp slowness was a CPU thing with ssh encryption, but someone on the web said the problem was in fact at a lower level (64K buffer sizes in ssh). The best solution I came up with was python -c "import SimpleHTTPServer;SimpleHTTPServer.test()" on the source machine and then wget source:8000 myfile on the other machine. That got me about 8MB/sec, which was barely fast enough to get the job done, so I stopped investigating.
But for next time, there must be a clever command-line solution that a Linux guru can recommend. Maybe piping through nc?
Maybe ftp or tftp? I'b be curious how nc worked out as well, if you were ever motivated to give it a try.
I've used the following to send whole compressed filesystems with permissions intact.
At the source:
tar cjvf - . /dev/null | nc -l -p 9999
At the destination:
nc 10.9.8.7 9999 | tar xjvf -
When you see /dev/null arrive at the destination, hit Ctrl-C and you're done. There's probably a neater way to make everything shut down automatically.