Sunday, March 13, 2011

Rsync and Backup Success

A blog post started long ago, back in Oct. 2010:

If you are a reader of my blog, you might have suffered along with me through a non-existent backup. Tomorrow, I'm setting off for UDS in Orlando, Florida, and I wanted to be sure to have an up-to-date backup. So I started up my 1.5 T backup, newly formatted, and looked up a reliable article about using rsync. No more relying on GUI front-ends to it! I wanted to use it direct from the command-line, so I could see the errors right away. is simple and understandable, even for me. "The basic syntax for rsync is simple enough -- just run rsync [options] source destination to copy the file or files provided as the source argument to the destination."

Written in the now:

Unfortunately, I didn't finish this blog post after returning from Florida, but today a major disaster became a minor disaster by means of 1. a small screwdriver, 2. an inexpensive disk drive enclosure ($14.99 at Fry's), and 3. this simple command:
rsync -av /media/home/valorie/Music /home/valorie/Music
Disaster was my laptop power supply dying on me, without a recent backup. I know, shame on me! Instead, I used the screwdriver to take out the hard drive and attempted to put it into the HD enclosure I already have. However, it didn't fit. So I wrapped up the HD securely, and made a trip to Fry's. I showed the salesman the drive, to ensure I got the right kind of enclosure (SATA).

Once home, I put the drive into the enclosure, hooked it up to my netbook (thanks, ZaReason!) and began transferring my music from my old laptop to my new one, while continuing to work on the Amarok Handbook, which is *almost done*! I took the opportunity of a freshly installed Amarok to make some screenshots with which to finish the book!

An update: One trouble I've had, besides some error messages that Sre-su helped me to figure out (thank you again, Sre-su!), is that the drive kept un-mounting. I don't know why -- bad USB connection, bad enclosure, bad HD -- but the solution is to just run the command again. Rsync checks the files it has already done, and then just charges ahead. I added -r (recursive) to be sure all music files were copied, and --progress so I can see at a glance what is going on. Also, --dry-run is cool -- it shows you what will happen, quick as a flash. Also, sre-su pointed out that I needed sudo before the command, which is now:
sudo rsync -avr --progress /media/home/valorie/Music /home/Music

I didn't catch my error in the above command until today, when I wondered why my music was STILL not in my Amarok collection! What must have happened is that while my original command was correct, the USB drive is touchy, and I needed to run it a few times. Not knowing this, I changed it, and put my Music not in /home/valorie/Music where it belongs, but in /home/Music/. Bleah.

Proper command, after all, is:
rsync -avr --progress /media/home/valorie/Music /home/valorie/Music


  1. This might look scary at first, but you might want to try git-annex.

    rsync is great and I use it extensively, but it's a swiss army bazooka which is aimed at your feet by default.

    While learning git-annex might be a bumpy ride at first, it will make sure you have at least one (or more) current copies elsewhere before it allows you to do much else.


    PS: Not allowing anonymous and/or trusting your commenters to enter their real name by hand without verification is a huge no-go, imo. I decided to log in to post this reply as I think you really need and want some helps and tips regarding data storage, but it was a close call. Normally, I simply wouldn't have commented and you wouldn't know git-annex existed.

  2. PPS: You moderate _and_ require a login? For shame.

  3. Richard, thanks for your hint; I'll take a look at git-annex.

    As for the comment settings, I didn't realize they were overly restrictive. One no longer has to login to comment, although I think that might make the comments less useful. We'll see.

    I refuse to use unmoderated comments, as I think that is irresponsible. This is my blog, and take responsibility for making sure that 1. bullying and 2. spam aren't here. I wish everyone else did, as well. The web would be a much more pleasant and useful place to hang out.

  4. Valorie: Re-reading my PPS, it can be misunderstood easily. To clarify: moderating is A Good Thing as it helps ensure a nice and helpful environment. Thanks for allowing comments from users who don't want to log in. I know it increases my likelihood of commenting on a blog _significantly_.

    In related news, the commnts by "Muhammad Azeem" are spam. They always follow the same pattern: Praise you for your helpful/interesting/whatever post and include a backlink to whatever they want to spam with, in this case, some sort of programming website.

    It's kind of ironic that this spam is in between this conversation, especially since it made it through both the registration (it's a blogger account) and moderation.

    Either way, good luck with your future back-ups; I suspect you have started to value them a _lot_ higher than in the past :)

  5. Thanks! I should have looked at the spammer links more carefully. He's now gone. :-)

  6. Hi,

    I've a 2.5" USB drive hooked up to our media PC, and it runs backintime via a cronjob.
    Actually, backintime is using rsync behind the scenes as well, The GUI is only for setup, then you'd run backintime -b.

    Main reason for the post is that the USB drives keeps disconnecting and reconnecting, and I don't have a clue why. This is on karmic.


  7. Steffen, I have the same problem with my USB drives. I think this is why I had problems with the GUI frontend to rsync I tried to use before, which left me with the secure feeling that I had a complete backup -- but no backup in fact.

    Now I won't trust anything but bare rsync. From the CLI, it's easy to use the up-arrow to enter the command again, and just keep running rsync until it has finished backing up.

  8. I just made a photo backup to external drive. Amount 338 gigabytes what toke 11 hours 45 minutes (15MB/s) and I used two konsole tabs for it.

    First tab executed rsync -rv -progress
    Second tab executed watch -n10 du -sh /backupdir

    I could nicely see where backup was going by files and the transferred data amount.

    Of course it could have been easy to use somekind GUI... but I wait until the official KDE backup software is finished...

  9. > watch -n10 du -sh /backupdir

    While a lot of metadata is being cached when you run du, doing it while your file system's tree is changing constantly means your caches get stale fast. I.e. you hurt your performance significantly with this

    I am the resident backup master at an ISP, so I deal with moving filesets a lot. I used to make the same mistake and it cost me hours upon hours ;)

    Also of interest: You want a modern file system if you have a lot of data. Barring religious wars, this means XFS or maybe EXT4. BTRFS might be great in a year or three.
    EXT3's performance with lots of small files is horrible.

    Finally, you want ncdu, not du, in most cases. Try it :)


    PS: Again, I would not have posted this if Name/URL hadn't been an option.

  10. My backup HD happens to be named FreeAgent Backup which has caused me all kinds of trouble. That space can be worked around: sudo rsync -avr --progress /etc/ /media/FreeAgent\ Backup/etc/ or sudo rsync -avr --progress /etc/ "/media/FreeAgent Backup/etc/" for backing up /etc.

    Another situation: I forgot to empty the trash! Next time, sudo rm -r /home/valorie/.local/share/Trash/* (-r gets the directories too).

    I think I need another post on rsync!