(Solved) Copying many files between removeable SSD and onboard SSD

[JohnJ]

Greetings,
I have an external SSD formatted as NTFS from my wife's windows computer and I need to copy across 500gb of data to my Linux mint 22.1 into an onboard SSD partition formatted as NTFS. The partition is on a different SSD from my Mint OS. I have used Freefilesync in the past but it really slows down trying to copy such large amounts of data. An added complication is that the data is structured into very many Folders, subFolders and subsubFolders. I need to maintain this structure.

So my question is:

1. Is there a better and/or different way to do this?

2. If I must only do this via the Terminal are there some simple (very simple) commands that will do the job?

Many thanks

 

[rado84]

Everything is slow because of the NTFS crap. That FS hasn't been updated since the dawn of time.
You can try copying files with the tool I use: rsync - it's a CLI only tool, no GUI. The use of rsync is pretty much the same as the traditional cp command with one major difference I should mention:
• if you use the command like this (no sudo is required, unless the receiving end is on root):

rsync -rv "/path/to/source dir" /path/to/target/dir

when it's completed, as a result you'll get "/path/to/target/dir/source dir".
For instance, if I decided to copy my "ETS 2" to another storage, I'd use:

rsync -rv "/B/GAMES/ETS 2" /D/GAMES_ARCHIVE

and when it's completed, I'd have "/D/GAMES_ARCHIVE/ETS 2".
BUT!!! And this is the important part: if you enter a backslash or whatever it is called after the source dir name (like this: "/B/GAMES/ETS 2/"), you'd get a bunch of files and dirs spread inside /D/GAMES_ARCHIVE. Basically, adding that backslash tells rsync "copy the contents of the directory to the target but not the directory itself".
This is what you should keep in mind when you use rsync. It's primary purpose is exactly what you wanna use it for: copying files from one storage to another. Works best when both source and target are the same filesystem, but will work with "NTFS to ext4" as well, or whatever combination, it's just gonna be slower.

 

[JohnJ]

Everything is slow because of the NTFS crap. That FS hasn't been updated since the dawn of time.
You can try copying files with the tool I use: rsync - it's a CLI only tool, no GUI. The use of rsync is pretty much the same as the traditional cp command with one major difference I should mention:
• if you use the command like this (no sudo is required, unless the receiving end is on root):

rsync -rv "/path/to/source dir" /path/to/target/dir

when it's completed, as a result you'll get "/path/to/target/dir/source dir".
For instance, if I decided to copy my "ETS 2" to another storage, I'd use:

rsync -rv "/B/GAMES/ETS 2" /D/GAMES_ARCHIVE

and when it's completed, I'd have "/D/GAMES_ARCHIVE/ETS 2".
BUT!!! And this is the important part: if you enter a backslash or whatever it is called after the source dir name (like this: "/B/GAMES/ETS 2/"), you'd get a bunch of files and dirs spread inside /D/GAMES_ARCHIVE. Basically, adding that backslash tells rsync "copy the contents of the directory to the target but not the directory itself".
This is what you should keep in mind when you use rsync. It's primary purpose is exactly what you wanna use it for: copying files from one storage to another. Works best when both source and target are the same filesystem, but will work with "NTFS to ext4" as well, or whatever combination, it's just gonna be slower.

Thanks rado. Whew. I was hoping for something a bit simpler to get my beginner head around. Someone mentioned grsync. Have you used this before? I reckon GUI might be easier for me.

 

[The Duck]

What Linux distro are you using.

 

[JohnJ]

What Linux distro are you using.

Mint 22.1 Cinnamon 6.4.8

 

[The Duck]

I've used this and it seemed to improve the transfer of data.

It's not hard to doe just follow the steps as it's all copy and paste.

Speed Up your Mint!

Easy tips, tweaks and tricks for Linux Mint and Ubuntu, both for beginners and for advanced users. Complete starter's guide with simple how-to's.

easylinuxtipsproject.blogspot.com

Look for this title and have a read.

 

[JohnJ]

I've used this and it seemed to improve the transfer of data.

It's not hard to doe just follow the steps as it's all copy and paste.

Speed Up your Mint!

Easy tips, tweaks and tricks for Linux Mint and Ubuntu, both for beginners and for advanced users. Complete starter's guide with simple how-to's.

easylinuxtipsproject.blogspot.com

Look for this title and have a read.

Thanks Duck. I can sort of understand the link but my laptop is 15 years old and I don't think it has much Ram to play around with. Cheers

 

[rado84]

Thanks rado. Whew. I was hoping for something a bit simpler to get my beginner head around. Someone mentioned grsync. Have you used this before? I reckon GUI might be easier for me.

It's not as hard as it looks, trust me. I was like you "once upon a time" - scared of the terminal window but once I got used to it, now I use it for 289 different things.

 

[JohnJ]

It's not as hard as it looks, trust me. I was like you "once upon a time" - scared of the terminal window but once I got used to it, now I use it for 289 different things.

Yes, it's something that I need to and would like to get my head around. I shall gird my loins.

 

[The Duck]

Yes, it's something that I need to and would like to get my head around. I shall gird my loins.

Take your time and take small steps and you will get there.

We were all new to Linux at one time.

 

[The Duck]

my laptop is 15 years old and I don't think it has much Ram to play around with. Cheers

Have a look at some of the tweaks here.

Home Page

Easy tips, tweaks and tricks for Linux Mint and Ubuntu, both for beginners and for advanced users. Complete starter's guide with simple how-to's.

easylinuxtipsproject.blogspot.com

Make sure you understand whatever tweak you are going to apply prior to applying it.

Some of the tweaks are helpful and useful.

 

[JohnJ]

Have a look at some of the tweaks here.

Home Page

Easy tips, tweaks and tricks for Linux Mint and Ubuntu, both for beginners and for advanced users. Complete starter's guide with simple how-to's.

easylinuxtipsproject.blogspot.com

Make sure you understand whatever tweak you are going to apply prior to applying it.

Some of the tweaks are helpful and useful.

OK. Will do. The easylinuxtipsproject is excellent. I basically are learning most of my stuff there. Cheers

 

[The Duck]

OK. Will do. The easylinuxtipsproject is excellent. I basically are learning most of my stuff there. Cheers

It's a right on solid resource I've used over the years without problems.

 

[MikeWalsh]

@JohnJ :-

TBH, the traditional "cp" (copy) command is as good as any. It's all we use in 'Puppy' to backup with.....no need for fancy backup programs/apps.

As @rado84 says, rsync is more or less based around it, as are many other solutions under the hood. Either that, or "dd". Both "cp" and "dd" are long-standing GNU tools that have been around at least as long as Linux itself, because both were inherited from Unix before it.

No, NTFS is not the best format you could have chosen. Be that as it may, when copying that sort of amount of data across, it IS going to take a while. It's not so much the sheer quantity of data that slows things down, so much as the multitude of small files that will invariably be amongst the rest of it.....these take as long as, if not longer to copy than large, contiguous files will.

On top of that, there's going to be a certain amount of "buffering" going on. Drives don't just copy stuff across continuously, non-stop; they'll fill up their caches, organise that onto the drive itself, then repeat this process, over & over, again & again, until the operation is completed.

~~~~~~~~~~~~~~~~~~~~​

SSDs, despite having higher read & write speeds, take longer to write data due to the way that an SSD works, compared to a traditional, spinning-rust HDD. Even if you just want to write to a single 'page' block, that block has to be copied first, then deleted before it can be written to again. That's a very simplified description of the process, but this is why the recommendation exists for SSDs.....of leaving at least 10%, minimum, of an SSD's capacity unformatted. Because it needs the space to perform all this juggling during the write process.

If you haven't left at least 10% of your SSD unformatted, the controller will be struggling to keep on top of what you've tasked it with doing...


Mike.

 

[JohnJ]

@JohnJ :-

TBH, the traditional "cp" (copy) command is as good as any. It's all we use in 'Puppy' to backup with.....no need for fancy backup programs/apps.

As @rado84 says, rsync is more or less based around it, as are many other solutions under the hood. Either that, or "dd". Both "cp" and "dd" are long-standing GNU tools that have been around at least as long as Linux itself, because both were inherited from Unix before it.

No, NTFS is not the best format you could have chosen. Be that as it may, when copying that sort of amount of data across, it IS going to take a while. It's not so much the sheer quantity of data that slows things down, so much as the multitude of small files that will invariably be amongst the rest of it.....these take as long as, if not longer to copy than large, contiguous files will.

On top of that, there's going to be a certain amount of "buffering" going on. Drives don't just copy stuff across continuously, non-stop; they'll fill up their caches, organise that onto the drive itself, then repeat this process, over & over, again & again, until the operation is completed.

~~~~~~~~~~~~~~~~~~~~​

SSDs, despite having higher read & write speeds, take longer to write data due to the way that an SSD works, compared to a traditional, spinning-rust HDD. Even if you just want to write to a single 'page' block, that block has to be copied first, then deleted before it can be written to again. That's a very simplified description of the process, but this is why the recommendation exists for SSDs.....of leaving at least 10%, minimum, of an SSD's capacity unformatted. Because it needs the space to perform all this juggling during the write process.

If you haven't left at least 10% of your SSD unformatted, the controller will be struggling to keep on top of what you've tasked it with doing...


Mike.

Thanks for your good points, Mike. I think that I will stick with Freefilesync. I can live with slowness as long as there is accuracy. Cheers John

 

[bob466]

Copy and paste or cut and paste in small amounts works for me...the Faster the CPU helps too.

 

[CaffeineAddict]

really slows down trying to copy

It could be wrestool doing something in background:

Solved Thread 'wrestool wasting all disk I/O'

Aug 31, 2024

I finally figured out why copying files to external HDD is painfully slow...

There is a directory on my external HDD which contains only Windows executables, 425 GB in total, cca. 200 *.exe files.
As soon as this directory is open in Dolphin a ton of wrestool instances start running examining and extracting all those executables. (noticed them by running htop)
Reason why it does so appears to be to calculate *.exe file sizes because I noticed sizes of sub directories in Dolphin are not immediately calculated but get slowly updated in UI.

Problem with those wrestool...

• CaffeineAddict
• wrestool
• Replies: 13
• Forum: General Linux Topics

 

[JohnJ]

It could be wrestool doing something in background:

Solved Thread 'wrestool wasting all disk I/O'

Aug 31, 2024

I finally figured out why copying files to external HDD is painfully slow...

There is a directory on my external HDD which contains only Windows executables, 425 GB in total, cca. 200 *.exe files.
As soon as this directory is open in Dolphin a ton of wrestool instances start running examining and extracting all those executables. (noticed them by running htop)
Reason why it does so appears to be to calculate *.exe file sizes because I noticed sizes of sub directories in Dolphin are not immediately calculated but get slowly updated in UI.

Problem with those wrestool...

• CaffeineAddict
• wrestool
• Replies: 13
• Forum: General Linux Topics

Understood. Thank you Cheers

 

[JohnJ]

Copy and paste or cut and paste in small amounts works for me...the Faster the CPU helps too.

View attachment 27626

G,day Bob,

I thought that I would let you know a bit more about what I found (and learned). Below is a what I said on a different post and forum as I was stumbling around looking for enlightenment. Maybe this is of some use to others trying to copy large amounts of data between computers. I hope it's useful to others on the forum. Cheers John

Below is my Blurb

My Linux Mint 22.1 laptop has one internal M2 SSD partitioned as Ext4 for the OS and one separate internal SSD with a couple of partitions some as NTFS (This is the one that I load up with my wife's windows data to and fro) and some as Ext4. The removable external drive that works between her Windows system and my Linux system is what I use to transfer data. The culprit partition (this is the one that failed) is on the internal SSD. I have discovered that the cause of the problem was me - good grief! I used Double Commander to try to copy 500gb or so of data from the removeable NTFS external drive to the onboard NTFS partition. Big mistake. I found out the hard way that Double Commander is not well suited to moving such large amounts of data. It worked reasonably fast but my laptop temp started to rise. It stopped and started a few times but I just kept going - another big mistake. Until eventually the NTFS partition stopped working. My initial thought was that the Mint 22.1 that I had clean installed the day before was the cause. I was so fixated on this that it took my old brain quite a while to think what else could have cause the partition to fail. Since my epiphany I re formatted the partition to NTFS, loaded up Freefilesync to do the job and 11 hours later the transfer completed perfectly and the NTFS partition was fine. Yes, it took a long time but the laptop temp did not rise and the file copying was steady with no errors. So, a valuable and big learning curve for me.