Even faster is a tar pipeline:
cd /drive1
tar cf - . | ( cd /drive2; tar xf - )
because both cp and rsync do one file at a time. There will inherently be small
pauses at each file boundary. Actually, rsync might stream a little.
Using piped tars and many files, particularly many small files, the first tar
can get ahead of the second tar for better throughput - the data queued in the
pipe (which has a buffer, and a generous one on Linux) allows the first tar to
proceed until the pipe is full if the second tar is blocked. (The second tar
will of course be blocked writing to drive2, but it won't be blocked reading
from drive1 because the first tar can read followon files from drive1 which the
second tar reads from the pipe).
Hi,
I was doing this and it is definitely faster than rsync:
cd /drive1
tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
But, after about 16 hours, I am only 229G in (out of 3.7T). This is much slower than the
other thread with USB drives which did 400GB in 8 hours.
Is this a function of the health of the first disk?
Also, I wanted to ask: if this job died and I restarted it, would it be possible to not
have to start it from all over again? (Sort of like rsync can do.)
Best wishes,
Ranjan