betabug... Sascha Welter

home english | home deutsch | Site Map | Sascha | Kontakt | Pro | Weblog | Wiki

11 June 2009

Linedrops While Copying Files over scp? Use rsync!

At least you can restart...

What to do when you want to copy a big file over scp and you get dropped connections, hangs, stalls, stuck connections? The problem is that scp does not pick up where it left off, it doesn't reconnect or recover dropped transfers... the magic trick is to switch tools: Use rsync.

Yes, rsync is usually used to "sync" two folders of files, but it can very well be used to copy only a single file too. Given the right "switches", rsync will pick up where it was when your line dropped...

Here is a sample command:

rsync -e ssh \
--partial --append \
--progress \ .

Which would be used instead of:

scp .

So what are those options? --partial and --append are doing the important parts: They tell rsync to check what it has so far and pick up from there. It will actually checksum what it got (which takes some time, but in generally is a Good Thing™). I also use --progress, which informs me where we are. You can throw in an additional -v to get more verbose output.

The best thing when using rsync with --partial and --append is that after you discovered your scp transfer has dropped, rsync can pick up from what you've already got, you're wasting nothing for switching tools.

If you think your line could be congested, you could try --bwlimit=XX on the rsync command line (you could do that too maybe if you want to be nice and share your bandwidth with some other users).

Posted by betabug at 14:56 | Comments (2) | Trackbacks (0)
ch athens
Life in Athens (Greece) for a foreigner from the other side of the mountains. And with an interest in digital life and the feeling of change in a big city. Multilingual English - German - Greek.
Main blog page
Recent Entries
Best of
Some of the most sought after posts, judging from access logs and search engine queries.

Apple & Macintosh:
Security & Privacy:
Misc technical:
Athens for tourists and visitors:
Life in general:
<< Moved to New Garage | Main | Schaltkreis Wassermann @ Benaki >>
Re: Linedrops While Copying Files over scp? Use rsync!

Nice! until now, I was using a local web server and then wget to retrieve the file from the other end...


Posted by: Wu at June 11,2009 15:58
Re: Linedrops While Copying Files over scp? Use rsync!

Good tip

Posted by: r0sk at June 12,2009 10:25
You can trackback to:
There are no trackbacks.
Leave a comment