Hello ! Help me with rsync
there is a task to make daily copying of projects in a backup. Without archiving, just transferring important directories with contents and ignoring unimportant directories.
rsync seems to me to be the best for this. but the problem is that the amount of data in the source is several terabytes.
there are small files and there are several gigs.
besides that, I already made the same backup earlier just with the cp command ignoring files that match in date and size.
So the destination folder already has a copy of all files from the source
firstly I started rsync like this
rsync -av /path/to/src /path/to/backup --exclude-from "filters.txt"
It started creating a list of files. Then, after a few hours of work, I interrupted it and started it again ....
It continued from the file on which I interrupted him (let's say this is the file /path/to/src/data/tree_05.bmp)
and it was cool - that's how it should work. Maybe rsync somewhere "remembers" all the files it has processed
BUT !
After he worked for two days. I interrupted it again... then ran another command like ls, then again started rsync with the same parameters as before.
To my surprise, It did not continue from the file he reached in two days, It started from the one on which I interrupted it for the first time, that is, from /path/to/src/data/tree_05.bmp
that is, not from the very beginning and not from the place where they interrupted the last time. Most likely I'm doing something wrong.
The fact is that given the amount of data for this analysis, copying changes from rsync will take several days for each launch ????
and the task is to check the source and upload only files that have changed during the working day to backup.
Previously a simple cp command handled this perfectly (ran via cron at night) I wanted to use rsync because of the wide possibilities of the command, in particular, the flexible system of ignore filters but here's something I can't figure out how rsync works....
Help plz
How analysis of the files that need to be copied goes? as I understand it, it checks the checksums of the fragments, but if this procedure takes so much time.
How is it possible to organize a backup of many files, the total size of which is terabytes, but of course it is needed to copy not all, but only updated ones
there is a task to make daily copying of projects in a backup. Without archiving, just transferring important directories with contents and ignoring unimportant directories.
rsync seems to me to be the best for this. but the problem is that the amount of data in the source is several terabytes.
there are small files and there are several gigs.
besides that, I already made the same backup earlier just with the cp command ignoring files that match in date and size.
So the destination folder already has a copy of all files from the source
firstly I started rsync like this
rsync -av /path/to/src /path/to/backup --exclude-from "filters.txt"
It started creating a list of files. Then, after a few hours of work, I interrupted it and started it again ....
It continued from the file on which I interrupted him (let's say this is the file /path/to/src/data/tree_05.bmp)
and it was cool - that's how it should work. Maybe rsync somewhere "remembers" all the files it has processed
BUT !
After he worked for two days. I interrupted it again... then ran another command like ls, then again started rsync with the same parameters as before.
To my surprise, It did not continue from the file he reached in two days, It started from the one on which I interrupted it for the first time, that is, from /path/to/src/data/tree_05.bmp
that is, not from the very beginning and not from the place where they interrupted the last time. Most likely I'm doing something wrong.
The fact is that given the amount of data for this analysis, copying changes from rsync will take several days for each launch ????
and the task is to check the source and upload only files that have changed during the working day to backup.
Previously a simple cp command handled this perfectly (ran via cron at night) I wanted to use rsync because of the wide possibilities of the command, in particular, the flexible system of ignore filters but here's something I can't figure out how rsync works....
Help plz
How analysis of the files that need to be copied goes? as I understand it, it checks the checksums of the fragments, but if this procedure takes so much time.
How is it possible to organize a backup of many files, the total size of which is terabytes, but of course it is needed to copy not all, but only updated ones
Last edited:
