dd failure

BHarts

New Member
Joined
Aug 11, 2020
Messages
13
Reaction score
2
Credits
101
Hello. I have used the Linux dd command many times. Recently, on a USB-connected 4TB drive, I went to use it to copy one small partition to another. After about 15GB of copying, it failed with a message "Input/Output error". Literally worded, that was all; there was no clue as to the reason for the failure. It is a rather new drive, and running the fsck command on its files turns up no errors. (Yes, I know that the dd command does not need to pay attention to file structures; I ran the fsck to check for obvious errors). I got the block size on the partition and used the 'bs' option to specify block sizes. Nothing so far has made the error message change. Eventually, I was able to use the Clonezilla utility to get the partition copied. That was 100% successful; no errors at any point. Still, I would like to know what could cause the dd command to fail. Perhaps it cannot handle drives larger than a certain size, e.g., 2TB, even for small partitions such as the 32GB size. I have searched for articles on the limitations of "dd", and so far, no luck. btw: the host system is running the Devuan variant of Debian. I even tried booting up with an up-to-date version of System Rescue; its dd operations had the same results.

Comment is invited.
 


forester

Active Member
Joined
Mar 5, 2022
Messages
289
Reaction score
167
Credits
2,090
Was Clonezilla Live used interactively so errors could be noted?
Clonezilla uses dd
 
OP
B

BHarts

New Member
Joined
Aug 11, 2020
Messages
13
Reaction score
2
Credits
101
Was Clonezilla Live used interactively so errors could be noted?
Clonezilla uses dd
I am new to Clonezilla; I basically followed the most direct instructions it had. I compared the results of the copy, and they appear to be 100% good.
 

shebang

New Member
Joined
Apr 30, 2022
Messages
2
Reaction score
1
Credits
13
dd if=/dev/sdd1 of=/dev/sdd2 status=progress
I used to use dd on my mac but that's been years ago. dd uses 512k bs by default. I believe for a partition copy they suggest 64 and use noerror

I would try,
dd if=dev/sdd1 of=dev/sdd2 bs=64M conv=noerror
 

forester

Active Member
Joined
Mar 5, 2022
Messages
289
Reaction score
167
Credits
2,090
Still, I would like to know what could cause the dd command to fail.
it's probably the external 4TB hard drive but cannot say for sure. Just use Clonezilla or try
Code:
dd if=/dev/sdd1 of=/dev/sdd2 bs=4096 conv=sync status=progress
 

dos2unix

Well-Known Member
Joined
May 3, 2019
Messages
1,392
Reaction score
1,020
Credits
8,557
Make sure that sdd2 is created in advance. Also make sure it's at least as big sdd1. (I usually go a couple of GB larger).
The bs usually isn't needed, dd will typically load as much as it can into memory between reads and writes anyway.
I don't usually recommend using dd on the same input/target device. In fact If you have another drive, I've had better
luck dd'ing to an external drive image files, and then back to the target disk. But it should work on the same device.
nvme m.2 drive are full duplex, so they can read and write same time. Sata SSD/HDD drives are not so you get slower
i/O, because they can't do both at same time.

If it's a external USB drive, it'll be even slower.
 
Last edited:
OP
B

BHarts

New Member
Joined
Aug 11, 2020
Messages
13
Reaction score
2
Credits
101
I used to use dd on my mac but that's been years ago. dd uses 512k bs by default. I believe for a partition copy they suggest 64 and use noerror

I would try,
dd if=dev/sdd1 of=dev/sdd2 bs=64M conv=noerror
Hello! Thanks for the "64" suggestion. I did a "man dd" on the Linux system. For the noerror option, it says to continue after a read error is encountered. I find this puzzling ...
 
OP
B

BHarts

New Member
Joined
Aug 11, 2020
Messages
13
Reaction score
2
Credits
101
Make sure that sdd2 is created in advance. Also make sure it's at least as big sdd1. (I usually go a couple of GB larger).
The bs usually isn't needed, dd will typically load as much as it can into memory between reads and writes anyway.
I don't usually recommend using dd on the same input/target device. In fact If you have another drive, I've had better
luck dd'ing to an external drive image files, and then back to the target disk. But it should work on the same device.
nvme m.2 drive are full duplex, so they can read and write same time. Sata SSD/HDD drives are not so you get slower
i/O, because they can't do both at same time.

If it's a external USB drive, it'll be even slower.
I hear you. I had used another drive as the destination partition and got the same results. The source drive in this case is an external USB connection; I do not know about the "full duplex" factor for USB; that looks interesting!
 

forester

Active Member
Joined
Mar 5, 2022
Messages
289
Reaction score
167
Credits
2,090
i believe input from 4TB ext drive may be too slow. Hence, sync. if not, config file editing somewhere. Try.
 

dos2unix

Well-Known Member
Joined
May 3, 2019
Messages
1,392
Reaction score
1,020
Credits
8,557
dd uses 512k bs by default.

For me personally I've found that ibs and obs make the copy faster than bs.
Maybe if you only have 2 or 4GB of RAM it won't make much difference but you have 32 or 64GB.
I've had it do a complete copy in one pass (50GB).
ibs=32G obs=32G if you can. But even 4G, 8G or 16G makes a difference.
 
OP
B

BHarts

New Member
Joined
Aug 11, 2020
Messages
13
Reaction score
2
Credits
101
For me personally I've found that ibs and obs make the copy faster than bs.
Maybe if you only have 2 or 4GB of RAM it won't make much difference but you have 32 or 64GB.
I've had it do a complete copy in one pass (50GB).
ibs=32G obs=32G if you can. But even 4G, 8G or 16G makes a difference.
Thanks!
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!

Members online


Top