rsync problem: trying to backup changed files to 2 USB sticks

coderlen

Member
Joined
Jul 23, 2020
Messages
49
Reaction score
26
Credits
474
I am trying to use rsync to make a copy of all the changed files on the laptop's hard drive, and back up those changed files onto two separate USB drives. I use two identical scripts to do this. The 1st script works just fine on the 1st USB. The 2nd script, identical to the 1st except for the target USB name, is supposed to copy the changed files on the hard drive to the 2nd USB.

To allow the files on the USB drives to be read in Windows as well as Linux, the USB drives are formatted as FAT32. Both of the USB drives are identical SanDisk 32 GB USB drives, which have plenty of room for the files that I am backing up. I know that FAT32 is limited to 4 GB per file, but none of the source files that I am backing up is anywhere close to that size.

Here is the code for the 1st USB backup script:
Code:
rsync -vtrlptgoD --modify-window=2 --exclude={'*.iso','/home/len/Downloads','/home/len/Desktop'} /home/len/* "/media/len/DRIVE F"
And here is the code for the 2nd USB backup script:
Code:
rsync -vtrlptgoD --modify-window=2 --exclude={'*.iso','/home/len/Downloads','/home/len/Desktop'} /home/len/* "/media/len/DRIVE D"
I have tried a number of different options for rsync in these scripts:
rsync -avtu
AND
rsync -cu

These options work in the 1st script, but in the 2nd script the changed file is not copied to the 2nd USB.

This is how I test to see if changed files are being copied:
1) Add a new file to the laptop's hard drive
2) Enter some data into the new file
3) Save the new file
4) Execute the 1st script, which backs up the changed file on the hard drive to the 1st USB
5) Check to make sure the new file got copied to the 1st USB
6) Execute the 2nd script, which is supposed to back up the changed file on the hard drive to the 2nd USB
7) Check to make sure the new file got copied to the 2nd USB (but of course it does NOT get copied)

It appears to me that rsync marks any files it copies with the 1st script as "archived". It may be changing some kind of archive bit, which is what happens when you archive files under Windows. I have tried to see if there is some kind of archive bit in Linux, but I can't see any evidence of it, other than the fact that my second attempt to back up the changed file fails.

Please guide me and show me what I'm doing wrong. Perhaps there is a special option in rsync which will prevent it from changing the archive bit on a file which has been changed and is ready to be backed up. If I can find a way to do that, the 2nd script should work, copying the changed file to the 2nd USB. Thanks.
 


KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
4,017
Reaction score
3,693
Credits
32,276
Well, when it's something like this, I'd concentrate on what's different. In this case, the difference is the path.

Is "/media/len/DRIVE D" the actual path to the device? Drive enumeration doesn't work like it does on Windows. They'd normally be things like sdb1 and sdc1, or similar.

I don't think rsync sets any mysterious archive bit, 'cause I've used it with the && to run the same command but to sync it to a second device all in the same command.
 
OP
C

coderlen

Member
Joined
Jul 23, 2020
Messages
49
Reaction score
26
Credits
474
KGill, thanks for your reply. So, do you think I could execute the backup to BOTH USBs at the same time, with the same command? Perhaps you could give me an example.
 

Condobloke

Well-Known Member
Joined
Apr 30, 2017
Messages
3,740
Reaction score
3,345
Credits
18,329
Perhaps it would defeat some unstated purpose, but you could always use Timeshift
1. It uses rsync
2. it only copies file that have changed

Where you save that too is entirely up to you. The location it is saved to can be altered under Settings>>>Location
 
Last edited:

khedger

Active Member
Joined
Jun 7, 2019
Messages
152
Reaction score
85
Credits
1,294
as a test, after the first run and before the second, do 'touch' on your file and see if anything changes.....

keith
 

stan

Well-Known Member
Joined
Mar 19, 2018
Messages
965
Reaction score
1,099
Credits
8,982
rsync -avtu
It seems that you have read in man rsync that the "-a" option replaces ALL of "-rlptgoD," and it is always better to make your code/scripts easier to read when possible. I recommend "-a" for this reason.

It is also good NOT to use any options that are not needed. I would question whether "-t" and "--modify-window=2" are needed. They may be, but I'm not seeing it here. Try without them, as you can always restore them if you need them.

I know that GParted will let you put a filesystem label on each drive, as you have done (ie, DRIVE F), and rsync will work with those labels in the path. And you also said your first script works.

You didn't say, but I would test the ORDER of your script execution. Do the second script first instead. If that script has a syntax or other error, it will fail first and rule out sequence. Also, if the label is wrong, or there is some difference between the flash drives, it would still fail first. It's something to try, if you haven't already.

The most important thing to me may be the " * " that you use in your source, "/home/len/*" -- I have been burned by this syntax before, and I recommend removing the " * " character. When you remove that, you then need to consider whether to use a trailing " / " because there is a difference:

/home/len - will copy the "len" folder itself, with all of its files and subfolders except those you have excluded. This is what I usually prefer and will look less cluttered on the flash drive.

/home/len/ - will not copy the "len" folder. It will copy all of the loose files and subfolders onto your flash drive, except those you have excluded.

As @KGIII said above, you can join your two scripts together as one using &&. A simplified version would look something like this:

Code:
rsync -avhs --exclude={'*.iso','/home/len/Downloads','/home/len/Desktop'} /home/len "/media/len/DRIVE F" &&
rsync -avhs --exclude={'*.iso','/home/len/Downloads','/home/len/Desktop'} /home/len "/media/len/DRIVE D"


I added "-h" to make numbers human readable, and "-s" help to fix problems with spaces in folder names and other cases where the shell interprets things other than how I intended. You may like these options, or not. One more option you may want to check out is "-n" (so, "-navhs" if you try it). The "-n" does a "dry run" and tells you how rsync would perform if it were actually run. This is very handy. The appropriate flash drive will still need to be plugged in so that rsync sees it.

[EDIT] You should know too, if you don't already, that rsync is copying ALL of the hidden dot files and folders from your home folder to your flash drives. To me, that is a lot of overkill, but it depends on your needs. You could use an "include" or an "exclude" text file to expand on what you want or don't want. What I do instead is that I have a list of about 8 rsync commands, each ending with && (except the last one) and I copy and paste those into a terminal from a text file. It's basically the same as a script. Each rsync line copies just the folders that I want to backup. [/EDIT]
 
Last edited:

wizardfromoz

Administrator
Staff member
Gold Supporter
Joined
Apr 30, 2017
Messages
6,620
Reaction score
5,378
Credits
19,579
G'day all :)

I am moving this to Command Line because that is where scripting inquiries are best situated and responded to, albeit this Thread has got good responses.

All participants and Viewers are redirected.

Cheers

Wiz
 

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
4,017
Reaction score
3,693
Credits
32,276
I would like to add to last night's post, but editing it at this point would mean it'd not be seen.

I should point out that when I do this I use the && rather than the ; because I want it to stop and not execute the second command if the first one has failed.

Secondly, I didn't think to mention an application called 'luckyBackup'. It's basically a GUI front-end for rsync. You can configure multiple jobs, meaning you'll be able to first run for one USB device and then for a second.

Doesn't sound all that awesome? Well, lemme just show you what the command is to backup to an external drive.

Code:
rsync -h --progress --stats -r -tgo -p -l -D --super --update --exclude=**/*tmp*/ --exclude=**/*cache*/ --exclude=**/*Cache*/ --exclude=**~ --exclude=/mnt/*/** --exclude=/media/*/** --exclude=**/lost+found*/ --exclude=/var/run/** --exclude=/run/** --exclude=/proc/** --exclude=/dev/** --exclude=/sys/** --exclude=**/*Trash*/ --exclude=**/*trash*/ --exclude=**/.gvfs/ / /media/kgiii/MIRROR/NewMirror/

It generates that with just clicking buttons.

You can also cheat and use it to generate the commands you need and then save them or, better still, use them as an alias. Aliases are awesome. I f*ckin' love aliases. Aliases are the next best thing after sliced bread. Save 'em to a separate aliases file and you can even have multiple alias profiles.

You'd be able to chain two of 'em together and then type 'backup' to run both commands one after the other.

Now how awesome would that be?

This should have been in my first post, but I was pretty exhausted last night and I didn't think to add it.
 

Condobloke

Well-Known Member
Joined
Apr 30, 2017
Messages
3,740
Reaction score
3,345
Credits
18,329

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
4,017
Reaction score
3,693
Credits
32,276
.....but...are they better than lace up boots ?!!~

Hmm... It's a close call!

I use aliases for a whole lot of things. I even have a second alias file that I load as needed. I've exported and used those two files for a long time. They stay consistent across multiple computers, and multiple installs. They sometimes vary, as needed. For example, I may need to edit out the apt aliases because the system uses yum or zipper. But, it otherwise stays fairly consistent.

Some of them are pretty stupid - but I use 'em anyways.

Don't try this at home!
alias update="sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y && sudo apt clean -y"

Others are fairly benign.

If I'm running pcmanfm from the terminal, I obviously want to use it with elevated permissions:
alias pcmanfm="sudo pcmanfm"

If I want info from vnstat:
alias stat="vnstat -i wlxe4beed0e5f5c"

The list goes on, but I don't want to derail OP's thread too much!
 
OP
C

coderlen

Member
Joined
Jul 23, 2020
Messages
49
Reaction score
26
Credits
474
KGIII, that is a very interesting approach to backups, and I will check it out. Thanks for that, looks like it would solve my problem.

But back to the original problem. I think we need to wrap that up, just because.

I found the problem. Still short of a solution, however.

Here is the problem. I failed to mention that I cloned the 1st USB to the 2nd USB. I used dd for this. So what happened is that the 2nd USB is an EXACT copy of the first USB, with the SAME UUID! I'm sure when I try to run the script on the 2nd USB, rsync sees that it has already backed up the changed files, so it doesn't back them up to the 2nd USB.

Now my problem is getting the UUID of the 2nd USB changed. I'm trying to use tune2fs -U random /dev/sdc1 to generate a new UUID, but all I get is this error message:

Code:
tune2fs 1.44.1 (24-Mar-2018)
tune2fs: Bad magic number in super-block while trying to open /dev/sdc1
/dev/sdc1 contains a vfat file system labelled 'DRIVE D'
 

jglen490

Well-Known Member
Joined
Mar 10, 2020
Messages
736
Reaction score
500
Credits
4,106
I use sudo rsync -auv <source> <target> It works every time. I'm not sure what your other options do (I understand the --exclude).
-a = perform as an archive (previously noted)
-u = update the target /path/file if the source filename is newer
-v = verbose

And, yes, rsync does not set any changes on the source file. My <source> and <target> are fully qualified path names, and not device identifiers.
 
OP
C

coderlen

Member
Joined
Jul 23, 2020
Messages
49
Reaction score
26
Credits
474
KGIII, your suggestion about "luckyBackup" really helped. Instead of getting bogged down with the format of the rsync command, this tool just generates it for you. Cool! It really helped me, and I was able to copy the generated rsync options into my scripts. I had to change the labels on the USB sticks to remove the space. That is to say, instead of using "DRIVE F" as the label, I now use "DRIVE_F", and it works.

Thanks for your wonderful suggestion! You helped me break the logjam.
 

KGIII

Super Moderator
Staff member
Gold Supporter
Joined
Jul 23, 2020
Messages
4,017
Reaction score
3,693
Credits
32,276
It really helped me, and I was able to copy the generated rsync options into my scripts.

I'm glad you got it sorted. It's the easiest way I know to generate complex rsync commands that are pretty much guaranteed to be correct. I use luckyBackup as a regularly scheduled cron task and I use it to generate other commands as needed.

I'm not sure if that's what the author's intended, but it works really well to do it.

Thanks for letting us know what worked for you. (I've been without the 'net for a couple of days, thanks to Mother Nature.)

Also, to make things easier on yourself, you can do what I do. I pretty much NEVER have a space or odd character in a file/directory name. Never. They're alpha-numeric only and contain no spaces. That saves you the hassle of having to figure out where to put the quotes when you want to process something in the terminal by file name.

And, one last edit... Your 'thanks' is why I do this. It makes me happy to help folks out. I'm retired and this makes me feel productive and beneficial. I don't do it for the fake internet points that the various sites hand out, I do it because I enjoy the act of helping.
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!

Members online


Top