Slow and slower files transfers >200G

[George Blizzard]

I guess in the 20+ years of using linux I never needed to make any large file transfers. But with replacing some older drives, I must now transfer up to 1.3TB of data. Unable to do so, so reduced to about 150GB at a time, and it still drops and drops to <Mb/sec, after starting at 35-40MB/sec, and sometimes hangs and crashes. Searhing for cause/solution reveals linux has always had this problem and programmers have failed to address it. This is a game changer, and not for the better. Thank God I haven't had to do a full backup and restore. Currently Xubuntu 18.04 on Intel 7 with 16GB RAM. I thought Windows was bad............

 

[Rob]

Hey there George - welcome to the site!

What type of connections do you have to your network? I haven't experienced this.

 

[wizardfromoz]

G'day @George Blizzard and welcome to linux.org from DownUnder.

(Wizard appears in a puff of smoke dressed in an anorak and snowshoes because he thinks there is a blizzard about - his hearing is not what it was 1,000 years ago)

Mate I am in the same boat as Admin @Rob ... I am interested.

If you get the time, can you give us the output from Terminal of (# precedes my comments)

inxi -Fxs

# or

inxi -Fxz

Mine on this Linux Mint is as in the Spoiler

Spoiler: Wizard-s Specs

chris@Tara-Cinnamon-SSD:~$ inxi -Fxz
System: Host: Tara-Cinnamon-SSD Kernel: 4.15.0-38-generic x86_64
bits: 64 gcc: 7.3.0
Desktop: Cinnamon 3.8.9 (Gtk 3.22.30-1ubuntu1)
Distro: Linux Mint 19 Tara
Machine: Device: laptop System: Dell product: Inspiron 5770 serial: N/A
Mobo: Dell model: 0XH3XD v: A00 serial: N/A
UEFI: Dell v: 1.1.8 date: 08/15/2018
Battery BAT0: charge: 39.9 Wh 100.0% condition: 39.9/42.0 Wh (95%)
model: SMP DELL Y3F7Y6B status: Full
CPU: Quad core Intel Core i7-8550U (-MT-MCP-)
arch: Kaby Lake rev.10 cache: 8192 KB
flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 15936
clock speeds: max: 4000 MHz 1: 2000 MHz 2: 2000 MHz 3: 1909 MHz
4: 2000 MHz 5: 2000 MHz 6: 2000 MHz 7: 2000 MHz 8: 2000 MHz
Graphics: Card-1: Intel UHD Graphics 620 bus-ID: 00:02.0
Card-2: Advanced Micro Devices [AMD/ATI] Topaz XT [Radeon R7 M260/M265 / M340/M360 / M440/M445]
bus-ID: 01:00.0
Display Server: x11 (X.Org 1.19.6 )
drivers: modesetting,ati,amdgpu (unloaded: fbdev,vesa)
Resolution: [email protected]
OpenGL: renderer: Mesa DRI Intel UHD Graphics 620 (Kabylake GT2)
version: 4.5 Mesa 18.0.5 Direct Render: Yes
Audio: Card Intel Sunrise Point-LP HD Audio
driver: snd_hda_intel bus-ID: 00:1f.3
Sound: Advanced Linux Sound Architecture v: k4.15.0-38-generic
Network: Card-1: Realtek RTL8101/2/6E PCIE Fast/Gigabit Ethernet controller
driver: r8169 v: 2.3LK-NAPI port: d000 bus-ID: 02:00.0
IF: enp2s0 state: down mac: <filter>
Card-2: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter
driver: ath10k_pci bus-ID: 03:00.0
IF: wlp3s0 state: up mac: <filter>
Card-3: Atheros usb-ID: 001-005
IF: null-if-id state: N/A speed: N/A duplex: N/A mac: N/A
Drives: HDD Total Size: 6288.3GB (11.0% used)
ID-1: /dev/sda model: ST2000LM007 size: 2000.4GB
ID-2: /dev/sdb model: Micron_1100_SATA size: 256.1GB
ID-3: USB /dev/sdc model: My_Book_25EE size: 4000.8GB
ID-4: USB /dev/sdd model: TransMemory size: 31.0GB
Partition: ID-1: / size: 20G used: 12G (61%) fs: ext4 dev: /dev/sdb10
RAID: No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors: System Temperatures: cpu: 60.0C mobo: N/A gpu: 61.0
Fan Speeds (in rpm): cpu: N/A
Info: Processes: 304 Uptime: 1:30 Memory: 1879.1/15938.0MB
Init: systemd runlevel: 5 Gcc sys: 7.3.0
Client: Shell (bash 4.4.191) inxi: 2.3.56

If you have any privacy issues with eg Mac address, use the -Fxz option, as it filters out the address.

Then we can see what's what with your config.

Cheers

Chris Turner
wizardfromoz

 

[George Blizzard]

G'day @George Blizzard and welcome to linux.org from DownUnder.

(Wizard appears in a puff of smoke dressed in an anorak and snowshoes because he thinks there is a blizzard about - his hearing is not what it was 1,000 years ago)

Mate I am in the same boat as Admin @Rob ... I am interested.

If you get the time, can you give us the output from Terminal of (# precedes my comments)

inxi -Fxs

# or

inxi -Fxz

Mine on this Linux Mint is as in the Spoiler

Spoiler: Wizard-s Specs

chris@Tara-Cinnamon-SSD:~$ inxi -Fxz
System: Host: Tara-Cinnamon-SSD Kernel: 4.15.0-38-generic x86_64
bits: 64 gcc: 7.3.0
Desktop: Cinnamon 3.8.9 (Gtk 3.22.30-1ubuntu1)
Distro: Linux Mint 19 Tara
Machine: Device: laptop System: Dell product: Inspiron 5770 serial: N/A
Mobo: Dell model: 0XH3XD v: A00 serial: N/A
UEFI: Dell v: 1.1.8 date: 08/15/2018
Battery BAT0: charge: 39.9 Wh 100.0% condition: 39.9/42.0 Wh (95%)
model: SMP DELL Y3F7Y6B status: Full
CPU: Quad core Intel Core i7-8550U (-MT-MCP-)
arch: Kaby Lake rev.10 cache: 8192 KB
flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 15936
clock speeds: max: 4000 MHz 1: 2000 MHz 2: 2000 MHz 3: 1909 MHz
4: 2000 MHz 5: 2000 MHz 6: 2000 MHz 7: 2000 MHz 8: 2000 MHz
Graphics: Card-1: Intel UHD Graphics 620 bus-ID: 00:02.0
Card-2: Advanced Micro Devices [AMD/ATI] Topaz XT [Radeon R7 M260/M265 / M340/M360 / M440/M445]
bus-ID: 01:00.0
Display Server: x11 (X.Org 1.19.6 )
drivers: modesetting,ati,amdgpu (unloaded: fbdev,vesa)
Resolution: [email protected]
OpenGL: renderer: Mesa DRI Intel UHD Graphics 620 (Kabylake GT2)
version: 4.5 Mesa 18.0.5 Direct Render: Yes
Audio: Card Intel Sunrise Point-LP HD Audio
driver: snd_hda_intel bus-ID: 00:1f.3
Sound: Advanced Linux Sound Architecture v: k4.15.0-38-generic
Network: Card-1: Realtek RTL8101/2/6E PCIE Fast/Gigabit Ethernet controller
driver: r8169 v: 2.3LK-NAPI port: d000 bus-ID: 02:00.0
IF: enp2s0 state: down mac: <filter>
Card-2: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter
driver: ath10k_pci bus-ID: 03:00.0
IF: wlp3s0 state: up mac: <filter>
Card-3: Atheros usb-ID: 001-005
IF: null-if-id state: N/A speed: N/A duplex: N/A mac: N/A
Drives: HDD Total Size: 6288.3GB (11.0% used)
ID-1: /dev/sda model: ST2000LM007 size: 2000.4GB
ID-2: /dev/sdb model: Micron_1100_SATA size: 256.1GB
ID-3: USB /dev/sdc model: My_Book_25EE size: 4000.8GB
ID-4: USB /dev/sdd model: TransMemory size: 31.0GB
Partition: ID-1: / size: 20G used: 12G (61%) fs: ext4 dev: /dev/sdb10
RAID: No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors: System Temperatures: cpu: 60.0C mobo: N/A gpu: 61.0
Fan Speeds (in rpm): cpu: N/A
Info: Processes: 304 Uptime: 1:30 Memory: 1879.1/15938.0MB
Init: systemd runlevel: 5 Gcc sys: 7.3.0
Client: Shell (bash 4.4.191) inxi: 2.3.56

If you have any privacy issues with eg Mac address, use the -Fxz option, as it filters out the address.

Then we can see what's what with your config.

Cheers

Chris Turner
inxi -Fxz
wizardfromoz

Here's my output of "inxi -Fxz"
inxi -Fxz
System: Host: local33702 Kernel: 4.13.0-46-generic x86_64 bits: 64 gcc: 7.2.0
Desktop: Xfce 4.12.3 (Gtk 2.24.31) Distro: Ubuntu 17.10
Machine: Device: laptop System: TOSHIBA product: Satellite S70-B v: PSPPNU-03N00W serial: N/A
Mobo: TOSHIBA model: Portable PC v: MP serial: N/A UEFI: Insyde v: 1.10 date: 04/29/2014
CPU: Quad core Intel Core i7-4710HQ (-HT-MCP-) arch: Haswell rev.3 cache: 6144 KB
flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 19954
clock speeds: max: 3500 MHz 1: 1787 MHz 2: 2566 MHz 3: 2185 MHz 4: 1800 MHz 5: 2182 MHz 6: 2117 MHz
7: 2134 MHz 8: 1812 MHz
Graphics: Card-1: Intel 4th Gen Core Processor Integrated Graphics Controller bus-ID: 00:02.0
Card-2: Advanced Micro Devices [AMD/ATI] Venus PRO [Radeon HD 8850M / R9 M265X] bus-ID: 01:00.0
Display Server: x11 (X.Org 1.19.5 ) drivers: modesetting,ati,radeon (unloaded: fbdev,vesa)
Resolution: [email protected]
OpenGL: renderer: Mesa DRI Intel Haswell Mobile version: 4.5 Mesa 17.2.8 Direct Render: Yes
Audio: Card-1 Intel 8 Series/C220 Series High Definition Audio Controller
driver: snd_hda_intel bus-ID: 00:1b.0
Card-2 Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
driver: snd_hda_intel bus-ID: 00:03.0
Sound: Advanced Linux Sound Architecture v: k4.13.0-46-generic
Network: Card-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
driver: r8169 v: 2.3LK-NAPI port: 3000 bus-ID: 07:00.0
IF: enp7s0 state: up speed: 100 Mbps duplex: full mac: <filter>
Card-2: Intel Wireless 7260 driver: iwlwifi bus-ID: 08:00.0
IF: wlp8s0 state: down mac: <filter>
Drives: HDD Total Size: 16003.2GB (20.5% used)
ID-1: /dev/sda model: ST2000LM007 size: 2000.4GB
ID-2: /dev/sdb model: ST2000LM003_HN size: 2000.4GB
ID-3: USB /dev/sdd model: External_USB_3.0 size: 6001.2GB
ID-4: USB /dev/sde model: Ext_HDD_1021 size: 2000.4GB
ID-5: USB /dev/sdc model: Expansion_Desk size: 4000.8GB
Partition: ID-1: / size: 46G used: 37G (85%) fs: ext4 dev: /dev/sdb5
ID-2: /usr size: 42G used: 4.7G (12%) fs: ext4 dev: /dev/sdb6
ID-3: /boot size: 361M used: 150M (45%) fs: ext4 dev: /dev/sdb3
ID-4: swap-1 size: 17.00GB used: 0.00GB (0%) fs: swap dev: /dev/dm-0
RAID: No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors: System Temperatures: cpu: 70.0C mobo: N/A gpu: N/A
Fan Speeds (in rpm): cpu: N/A
Info: Processes: 280 Uptime: 14:57 Memory: 1954.5/15952.6MB Init: systemd runlevel: 5 Gcc sys: 7.2.0
Client: Shell (bash 4.4.121) inxi: 2.3.37

 

[wizardfromoz]

OK now there's something we can get our teeth into, Ta for the output (Ta is Aussie for thanks, pron. "tar").

First up - do I need to tell you support ended in July for 'Artful Aardvark" - Distro: Ubuntu 17.10 ? Actually, you said 18.04 so did you upgrade through the existing install? If so, inxi is reading the original version, that might be of interest (to me, not to you )

ID-1: /dev/sda model: ST2000LM007 size: 2000.4GB
ID-2: /dev/sdb model: ST2000LM003_HN size: 2000.4GB
ID-3: USB /dev/sdd model: External_USB_3.0 size: 6001.2GB
ID-4: USB /dev/sde model: Ext_HDD_1021 size: 2000.4GB
ID-5: USB /dev/sdc model: Expansion_Desk size: 4000.8GB

For all that to be showing, you must have a hub of some sort, or other connection-chaining process. Is it USB 2 or USB 3? Although I am on the new Dell Inspiron now, my No.2 unit is a Satellite S70-A, with a 1TB drive and Insyde setup utility, so handy too.

Looks like a combination of Seagates and perhaps Western Digital.

ID-1: /dev/sda model: ST2000LM007 size: 2000.4GB

Seagate have a data sheet here https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf that indicates a max transfer rate of 140MB/s

...after starting at 35-40MB/sec, and sometimes hangs and crashes.

... but that start speed is not unusual in my experience, slower and crashing is.

Are the old drives USB 2 or USB 3? And I am duty-bound (not being patronising) to ask if you are aware that, even if all drives are USB 3 capable and enabled, that if there is a USB 2 port or cable somewhere in the process, data can only be transferred at USB 2 speeds?

Finally, for now,

Searhing for cause/solution reveals linux has always had this problem and programmers have failed to address it.

... can you remember some of the sites or references? I would like to take a look.

Cheers

Wizard

 

[George Blizzard]

OK now there's something we can get our teeth into, Ta for the output (Ta is Aussie for thanks, pron. "tar").

First up - do I need to tell you support ended in July for 'Artful Aardvark" - Distro: Ubuntu 17.10 ? Actually, you said 18.04 so did you upgrade through the existing install? If so, inxi is reading the original version, that might be of interest (to me, not to you )



For all that to be showing, you must have a hub of some sort, or other connection-chaining process. Is it USB 2 or USB 3? Although I am on the new Dell Inspiron now, my No.2 unit is a Satellite S70-A, with a 1TB drive and Insyde setup utility, so handy too.

Looks like a combination of Seagates and perhaps Western Digital.



Seagate have a data sheet here https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf that indicates a max transfer rate of 140MB/s



... but that start speed is not unusual in my experience, slower and crashing is.

Are the old drives USB 2 or USB 3? And I am duty-bound (not being patronising) to ask if you are aware that, even if all drives are USB 3 capable and enabled, that if there is a USB 2 port or cable somewhere in the process, data can only be transferred at USB 2 speeds?

Finally, for now,



... can you remember some of the sites or references? I would like to take a look.

Cheers

Wizard

I upgraded 17.04 to 17.10 to 18.04, maybe it didn't work. I have two dedicated USB 3.0 ports and two USB 2.0 ports on laptop. Ports in use verifed with lusb, @ lsusb -v -s *:*
1-I'm copying from a USD 2.0 drive through USB 2.0 port and cable to USB 3.0 drive through a USB 3.0 port and cable
2-Also, at different time, USB 3.0 drive through USB 3.0 port and cable to an internal drive on a CD-to-HD caddy(but that's another problem)
3-If I limit transfer size to <50GB, everything moves at about 35MB/se(100GB+/hour)
4-Inrease file transfer sixe to >100GB, slow downs start. >150GB, expect it to bog down and take about 3-4 hours.
5-Try >300-500GB or even >1TB and it will stop, amongst some errors on target, "can't create dir XXXXX". If you selet "skip, autoskip or canel" the target drive will crash and contents deleted, inluding entry in fstab:must reformat and start over. This all with root priviledges.
sites:https://superuser.com/questions/424512/why-do-file-copy-operations-in-linux-get-slower-over-time
https://askubuntu.com/questions/995946/decrease-of-transfer-rate-when-copying-large-amount-of-data
https://unix.stackexchange.com/questions/117680/make-disk-disk-copy-slower

 

[wizardfromoz]

OK, Ta for those links, I've read a fair bit of them and can see where the people concerned, and yourself, appear to have a legitimate case for investigation.

Next question is what method/s were you using with the copying (I note your doing them as Root, so guess Terminal):

eg

• dd
• rsync
• good ol' cp
• other

If you cannot recall exact syntax, then you could interrogate your bash history (in /home/george or in /root) with for example

cat /root/.bash_history | grep rsync

Also can you tell us a little more, or even show us a Gparted screenshot, of the source drives and their partitions.

I am wondering if you have made the acquaintance over the years with Steven Shiau's Clonezilla, which might be suitable for cloning drive to drive or partition to partition, &c?

Cheers

Wizard

 

[tchakatak]

Is the target your copying your data using a SSD ?

 

[wizardfromoz]

Not unless he has a 2TB SSD in which case I am green with envy

Wiz

 

[Condobloke]

@ tchakatak ....is it possible the transfer speed of the usb 3 etc is too much for the write speed of a hdd......and actually needs a ssd to keep up the pace ?

(or am I talking out of my ear again ?)

 

[tchakatak]

On modern SSD technology most of them have a fast cache, and a slow memory.

Generally after a hundreds of Gb you have a ridiculous drop in performances due to the saturation of the cache, and the firmware start to send the data directly to the slow memory.

Maybe something similar is happening during your transfer ?

 

[George Blizzard]

OK, Ta for those links, I've read a fair bit of them and can see where the people concerned, and yourself, appear to have a legitimate case for investigation.

Next question is what method/s were you using with the copying (I note your doing them as Root, so guess Terminal):

eg

• dd
• rsync
• good ol' cp
• other

If you cannot recall exact syntax, then you could interrogate your bash history (in /home/george or in /root) with for example

cat /root/.bash_history | grep rsync

Also can you tell us a little more, or even show us a Gparted screenshot, of the source drives and their partitions.

I am wondering if you have made the acquaintance over the years with Steven Shiau's Clonezilla, which might be suitable for cloning drive to drive or partition to partition, &c?

Cheers

Wizard

I'm not copying a whole drive, just portions, certain dirs
I use mc as root "copy", or krusader as root to "copy" or "synchronize"(I know krusader and xfce4 don't mix, but it's worked for me for 20+years)

 

[wizardfromoz]

I'm not copying a whole drive, just portions, certain dirs

Thanks for the screenshots, we could understand them a little better if you edited your post and entered a couple of lines nad put a caption under each shot. eg Screenshot 1 - WD 2TB, Screenshot 2 - Seagate 1TB &c.

Are there discrete Partitions for which you want to transfer the whole contents?

I must now transfer up to 1.3TB of data. Unable to do so, so reduced to about 150GB at a time, and it still drops and drops to <Mb/sec, after starting at 35-40MB/sec, and sometimes hangs and crashes

I am speculating that mc (Midnight Commander, I take it) and Konqueror may have limits built in on transfer speeds and quantity of data transferred, something like a MAX_ARG built in, I am not sure of the term.

Rsync at the command line, might be your answer, both for folders, and for partitions.

I have a 4TB WD My Book hooked in to my Dell Inspiron lappie, which itself has a 256GB SSD and a 2TB HDD, 16GB RAM, same as you. As far as I am aware the WD My Book is a 5400rpm.

I have on it a 400GB Partition currently using 252.9GB of that allocated space. I set up to transfer it to my HDD by setting up a 400GB Partition using GParted.

I started the process with rsync, using

sudo rsync -av /run/media/root/Timeshift /run/media/chris/dba8d798-bf20-4cc8-83c8-b055cb864d21/

(I am in Manjaro, hence the /run/media/root). The part following /run/media/chris/ is the UUID for the new Partition, which I have not labelled or named yet.

If you wanted to do this in your environment with a folder at a time, you could use eg

rsync -av /media/sdc1/Pictures/ /media/sdb1/Pictures/

...using appropriate naming conventions.

With the options, in this case -av , v is verbose, and a covers a whole bunch of options, see

man rsync

I let the process run for 18 mins, and it copied 42GB or so, this is speeds of around 45MB/sec.

I did not ever intend to do the lot, so I can't say with certainty that there might not be a slowdown somewhere.

When I broke the process with a Ctrl-C, I then restarted it with a repeat of the same command, to confirm that where rsync breaks off (whether through error or user input) it picks up where it left off and so there is no wasted effort.

If this worked for your whole 1.3TB, you might be looking at 8 - 9 hours, perfect for an overnight job (turn off screensavers and power saving).

That's the best I can suggest for now, I am off to reclaim my 42GB

Wizard

 

[George Blizzard]

Today I used grsync to transfer 816GB of date from an internal drive to a USB 3.0 external drive. Took 5H 40M, @ 1444GB/hr,or 40MB/sec. Pretty Good. No hangup, crashes, or errors. Rsync or it's GUI Grsync is the way to go for large file transfers on linux.
Thanks to all for suggestions, and help, insight.

 

[wizardfromoz]

or 40MB/sec

That is most excellent news, George, goodonyer.

When you get your new environment set up the way you like it, and when you get a chance, if you have not already, make the acquaintance of Timeshift.

You can read my Tute here

https://www.linux.org/threads/timeshift-similar-solutions-safeguard-recover-your-linux.15241/

.. or there is a good summary in Linux Lite's Manual, here

https://www.linuxliteos.com/manual/tutorials.html#timeshift

It has as its underlying engine, rsync, but it does not yet extend across a network, but for drive to drive, easy peasy.

Cheers, enjoy your Linux and

Avagudweegend

Chris
wizardfromoz