Purchase Linux CDs / DVDs / Flash Drives at OSDisc.com

Welcome to Our Community

While Linux.org has been around for a while, we recently changed management and had to purge most of the content (including users). If you signed up before April 23rd, 2017 please sign up again. Thanks!

if statement / rsync back-up query

Discussion in 'Getting Started' started by stjude1982, Aug 9, 2018.

  1. stjude1982

    stjude1982 New Member

    Joined:
    Aug 9, 2018
    Messages:
    3
    Likes Received:
    1
    Hi,

    I'm new to Linux and I'm not sure where to place this thread?
    I'm looking to start using raspberry pi for some projects around house.
    Want to be able to back up files automatically to usb using cron, but, first have to create bash script in order to do so.
    The folder I would like to back up is named ' family ' and I've managed to create an automatic mount point for usb as ' /media/usb ' (this seems to work if usb is mounted at boot but not if it's plugged in after boot).

    First thing I want to be able to do is verify the existence of the usb before back-up. Was going to use rysnc for back as this sounds like the best command based on the command only saving changes. I've a couple of questions:
    1-is rsync the best command to use? I've read that the command uses timestamp to check for differences and don't know how this will be affected with files saved to usb.
    2-is the following if statement going to check for the existence of a usb?
    3-what do I do with the else part of the statement if a usb is not present?

    if [-e /media/usb] ; then

    rsync -a ~/family /media/usb

    else ???????????

    fi



    Any help would be greatly appreciated.

    Thank you!
     
  2. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    1,699
    Likes Received:
    1,795
    Hi @stjude1982, and welcome!

    Not sure if you needed this... or what exactly that you did (create an entry in FSTAB?). The Pi will automount a USB at this location: /media/pi/device-id -- and I find that it is recognized there both before and after boot time. The device-id is like an embedded serial number that stays with it even if you insert it into different computers... however, the id will change if you reformat the device. This is handy so that you can rsync from the Pi to a flash drive, and then use a similar rsync to restore your family folder to another computer.

    rsync will fail if the destination is not present, so I'm not sure of the need to test for it first. But testing the rsync process is definitely worthwhile to make sure that your backup works as expected. But I hope that you're not using your actual family folder for tests! :eek::D

    So, a simple script file (remember to make it executable) that you can call from cron might look like this:
    Code:
    #!/usr/bin/env bash
    rsync --delete -avhs "/home/pi/family-test-folder" "/media/pi/device-id/"
    # may need to change "pi" in the source/destination if you have a personal username
    # just make sure these both source and destination paths are correct
    Caution: The --delete option means that rsync will delete old files previously backed up in the destination if they are deleted in the source. So when complete the source and destination will always be exactly the same. This is my preferred method, but you need to consider if this is right for you or not.

    Extra options I showed in example:
    -v = verbose
    -h = give number outputs in human understandable form
    -s = allows filenames and folder names to contain spaces

    Hope this helps. There is probably a dozen different ways to do it, most more elegant than my simple example. Personally, I have several rsync command strings saved in a text file (used for different backup purposes) and I just copy from the text file and paste into a terminal manually when I want to make a backup... some smaller to flash drive, some larger to external hard drive. My reasoning is partly because I don't want to leave devices connected all the time for fear of power surge or lightning strike. It would be bad enough to lose a computer, but really bad to lose both computer and backups at the same time. (Of course fire, flood, tornado, theft... there are plenty of risks that remain.:()

    Cheers
     
    Rob likes this.
  3. stjude1982

    stjude1982 New Member

    Joined:
    Aug 9, 2018
    Messages:
    3
    Likes Received:
    1
    Thank you for your reply @atanere

    Didn't think I needed amend the fstab tbh.
    The reason I did so - even though the usb had rwx permissions, I was struggling to get rsync to work or to create new folders.
    Think the reason was originally trying to copy to ~/media/usb
    This was failing as couldn't find /home/pi/media/usb

    Is there a way to create the destination as such that won't matter what usb is inserted? i.e. for your example where you don't want to leave usb constantly plugged in and you just use whatever device you happen to find.

    Thank you for pointing out that the if statement is not required also!

    Cheers!!
     
  4. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    1,699
    Likes Received:
    1,795
    I found a link on the web that mentioned a different modification to fstab, but I played with the Pi and found that this little trick works similarly without changing fstab at all. So let me explain...

    If you have multiple flash drives, all formatted as FAT32.... you can add a "disklabel" so that each of them is labeled as BACKUP (or anything else that you prefer). The label can be applied from Windows, or from Gparted in Linux (but I don't have Gparted in my Pi, so you might have to install that... I used a different computer to set these up). With these drives so labeled, each one that I insert into the Pi gets automounted at the location of /media/pi/BACKUP. And this allowed me to run the script on each flash drive to successfully backup my test folder.

    The catch: I tried the same trick with an external hard drive, formatted as NTFS, and it did not work. It shows up at the same location (/media/pi/BACKUP) but rsync is not transferring the folder and files. There may be a way around this, but I'm running out of time to experiment for now. It might work if the hard drive were also formatted as FAT32. On the other hand, one of my regular backup external hard drives is also NTFS and it always works.... not sure why unless it is that I use the device-id with my rsync command instead of a label.

    Another caution: I don't think I would insert 2 of these drives at the same time. I don't know how the Pi (or any Linux) would handle the conflict of having the same disklabel. At the least, it would not be a good idea to try a backup like that! :eek::D

    And another caution: I think that FAT32 has a filesize limit of 4 GB (this is from memory, so I might be wrong). Even though some flash drives are very large now and would hold very large files, this limitation that I'm thinking of could cause you problems if you have a need to backup anything of such a size (videos are the most likely). If you need to store/backup large files, you'll want to consider NTFS if you also need to transfer to Windows, or the Linux EXT4 filesystem if needed on Linux only.

    I'll be in and out for about the next 5 days as I return to another work cycle, but I'll keep thinking a bit about this and will run some more tests if I get a chance, and depending on if you can narrow the scope of your needs.... ie, FAT32 vs NTFS? Very large files? Flash drive only or also external hard drive? If hard drive also... HDD or SSD?

    Cheers
     
  5. stjude1982

    stjude1982 New Member

    Joined:
    Aug 9, 2018
    Messages:
    3
    Likes Received:
    1
    Thank you for your help.
    All interesting stuff!!
    At present I'm just backing up some office files. Going forward I would like to organise and back up photos, then our music and finally maybe organise some movies.
    I've got 1TB external hard drive I suspect is NTFS and HDD. Although don't know too much about HDD and SSD. Part of the reason for starting this is to become more familiar with/and make use of the equipment we have.
    Appreciate all your help.
    Cheers!!!
     
    atanere likes this.
  6. atanere

    atanere Moderator
    Gold Supporter

    Joined:
    Apr 6, 2017
    Messages:
    1,699
    Likes Received:
    1,795
    If you plug this external drive into your Pi, it will prompt you to open it with the File Manager, and from there you can see the device-id. With that, you can make a quick script from my example above to test and make sure it works that way. This is my normal method and it works quite well. (But don't forget to consider whether you want the --delete option or not.)

    Cheers
     

Share This Page