Fast(er than rsync) network file transfers

Fast directory transfer across a LAN

When transferring a large set of files across a LAN and performance is needed more than security then using netcat and tar can provide the highest throughput.

Optionally you can also utilize pbzip2 for parallel compression, but in reality the CPU cost of compression usually results in slower throughput.  The pv output will display transfer speeds in terms of uncompressed size-on-disks–so you can compare apples to apples in terms of non-compressed versus compressed transfer.

In my setup with two servers, both Xeon quad-core 3GHz boxes with a 1Gbps link I’m bandwidth-bound (transfers of about 125MB/s) without compression, and am CPU-bound with compression at about 20MB/s.

For non-compressed transfer

Setup the listening server:

nc -l 8888 | pv | tar –strip-components 3 -xf – -C /mnt/raid10/elasticsearch-data/data

Begin the transfer on the sending server:

tar -cf – /var/lib/elasticsearch/ | pv | nc 2950-iii 8888

For pbzip2 compression use:

Setup the listening server:

nc -l 8888 | pbzip2 -dc | pv | tar -pxf – -C /mnt/raid10/elasticsearch-data/data

Begin the transfer on the sending server:

tar -cf – /var/lib/elasticsearch | pbzip2 -c | pv | nc 2950-iii 8888