System rebooted unexpectedly with a USB drive attached - now I cannot remount said drive.

[Priest_Apostate]

My Rocky desktop encountered an issue in which it just shut down in the middle of me working with it (I was working on a previous issue of cleaning up my /var directory. Unfortunately, I had a 5TB USB drive mounted to it, which now has the drive as not being able to be mounted - with an unknown error message stating that the drive can't be mounted. While I do have some backups, I did obtain a significant amount of content on said drive that I needed to backup onto another drive. I'm unsure of how to fix said drive, as I'm unsure if dd can retrieve the content before running some sort of disk repair application. As the drive can't be mounted, I'm not even sure if that can be run. If anyone can suggest anything to possibly recover said data, that would be helpful.

 

[Condobloke]

Try this: Boot to a usb stick, so thta are running a Live version....(not fully installed)

can you see the 5tb drive from there ?

I am not familiar with these types of dramas, but it has always been my understanding that you should do as little as possible with the 5tb, OS that if you need to run data recovery, that drive should not be used as this overwrites space on the disc.

If you can see the disk from the booted usb stick, other people here will have necessary knowledge regarding how to handle it from there.

(The members below may be able to help....they will drop by if they are able)

@GatorsFan
@Terminal Velocity
@Brickwizard
@f33dm3bits

 

[Brickwizard]

My Rocky desktop encountered an issue in which it just shut

Not over familiar with modern RHEL distros, so looking at it from a general standpoint,
1] did the machine/CPU overheat and shut down,
2] check ram for errors and poor joints
3] unplug drive switch on if everything else is working then re-connect drive and re-start is the drive now seen? is it working, can you at least run a drive health check on it?

 

[kc1di]

there are several things that could have happened, but as others have said disconnect the drive then reboot and reconnect the drive does the system recognize the drive then?

 

[osprey]

My Rocky desktop encountered an issue in which it just shut down in the middle of me working with it (I was working on a previous issue of cleaning up my /var directory. Unfortunately, I had a 5TB USB drive mounted to it, which now has the drive as not being able to be mounted - with an unknown error message stating that the drive can't be mounted. While I do have some backups, I did obtain a significant amount of content on said drive that I needed to backup onto another drive. I'm unsure of how to fix said drive, as I'm unsure if dd can retrieve the content before running some sort of disk repair application. As the drive can't be mounted, I'm not even sure if that can be run. If anyone can suggest anything to possibly recover said data, that would be helpful.

As @Condobloke mentioned in post #2, it's best if nothing is run that involves the 5TB drive if you want to rescue as much as possible from it. Any more activity on the disk will only alter the filesystem which will likely disturb it more than it already is.

One common approach is to use the testdisk program and the photorec program, both of which are outlined here: https://www.cgsecurity.org/wiki/TestDisk. There's a learning curve to using them but the documentation is quite extensive.

When recovering files, it's really necessary for best results to leave the problematical external drive unmounted, and use the recovery tools, testdisk and photorec, from either a live disk or the installed operating system if it's functioning properly. You will need a location to copy recovered files to which could be a usb, a connected computer, a directory on a properly functioning installed system, another external hard disk etc. If the installed system doesn't boot, use a live disk or rescue disk.

It's not clear why the system would spontaneously reboot if you were working in the
/var directory. Uninitiated rebooting is often associated with faulty memory, so one might consider testing the RAM with the memtest86+ program.

Check the logs to see if there's any information that might be helpful. One or more of the commands like the following can help:

journalctl -b
journalctl -b -x -p 3
journalctl -b | grep -i error
dmesg

Other than faulty RAM, there are other possibilities that can cause uninitiated reboots like an overheated CPU, power supply fluctuations or motherboard issues which can get a bit technical, but the RAM check would be the first check I'd make.

 

[Priest_Apostate]

Try this: Boot to a usb stick, so thta are running a Live version....(not fully installed)

can you see the 5tb drive from there ?

I didn't try the USB stick approach. I'll try to see if it can be detected with my Tails USB install.
I did, however, attempt to mount it upon my Debian laptop. Both systems are able to detect the device when I run a sudo blkid command. However, I am still unable to mount the drive from the labtop, or the Rocky Linux Desktop.

 

[Priest_Apostate]

Not over familiar with modern RHEL distros, so looking at it from a general standpoint,
1] did the machine/CPU overheat and shut down,
2] check ram for errors and poor joints
3] unplug drive switch on if everything else is working then re-connect drive and re-start is the drive now seen? is it working, can you at least run a drive health check on it?

This is the first time that the system has exhibited the behavior - there was no storm or browout.

I checked the logs - no indication showed regarding overheating. I'm currently installing memtester to check the RAM health.
I unplugged and replugged the drive, into another system (also switched out the data cable). Same issue when connected to my Debian Laptop. As I can't mount the drive, I can't run the health check.

It seems that I am able to run a creating of a BACKUP_1.img.gz file for the backup - but the weird part is that out of the 200GB currently on the drive, only 20GB was saved to the backup file.

 

[Alexzee]

How old is the 5TB drive and is by chance a Seagate brand?

The dmesg log like osprey showed in code for you could show what caused the shutdown.
The journals help with seeing what systemd, the session manager's state and the kernel are doing.

I hope for you bro that the RAM isn't bad.

 

[Priest_Apostate]

How old is the 5TB drive and is by chance a Seagate brand?

The dmesg log like osprey showed in code for you could show what caused the shutdown.
The journals help with seeing what systemd, the session manager's state and the kernel are doing.

I hope for you bro that the RAM isn't bad.

1. less than a year old - it is a Seagate. As you asked about it being a Seagate, I checked for Seagate specific tools, but couldn't find anything, @Alexzee - why did you ask about the brand?
2. I've been poring through the dmesg log file - but can't find anything that would indicate an issue.
3. I've been researching about journals and systemd references for information on how to attempt to investigate further.
4. I'm running memtester now, on 9GB out of 16GB (10 GB was free, so I figured on saving 1 GB to spare) - I'm on Loop 3, and no issues have come up yet.

 

[Priest_Apostate]

there are several things that could have happened, but as others have said disconnect the drive then reboot and reconnect the drive does the system recognize the drive then?

1. Doing the disconnect/reconnect didn't work.
2. Tried mounting onto my Debian laptop: no improvement.
3. Running Memtester on 9 of the 10GB free of the memory - currently, it is on loop 4, with no errors showing.
4. Also researching on how to perform data recovery of the drive.

 

[Priest_Apostate]

Try this: Boot to a usb stick, so thta are running a Live version....(not fully installed)

can you see the 5tb drive from there ?

I am not familiar with these types of dramas, but it has always been my understanding that you should do as little as possible with the 5tb, OS that if you need to run data recovery, that drive should not be used as this overwrites space on the disc.

If you can see the disk from the booted usb stick, other people here will have necessary knowledge regarding how to handle it from there.

(The members below may be able to help....they will drop by if they are able)

@GatorsFan
@Terminal Velocity
@Brickwizard
@f33dm3bits

The only thing I did to the drive was attempt to run a dd backup. As the backup file only created a 20GB file out of 200GB stored on the system, I am not optimistic that that data is recoverable through my efforts. I may need to pay for professional recovery services.

I think I mentioned this earlier today (I can't remember) but I will attempt to work with my Tails drive, to see if the drive can then be detected.

 

[Priest_Apostate]

Try this: Boot to a usb stick, so thta are running a Live version....(not fully installed)

can you see the 5tb drive from there ?

I am not familiar with these types of dramas, but it has always been my understanding that you should do as little as possible with the 5tb, OS that if you need to run data recovery, that drive should not be used as this overwrites space on the disc.

Out of curiosity, how would using a USB stick with a live OS differ from testing on another system entirely?

 

[osprey]

The only thing I did to the drive was attempt to run a dd backup. As the backup file only created a 20GB file out of 200GB stored on the system, I am not optimistic that that data is recoverable through my efforts. I may need to pay for professional recovery services.

I think I mentioned this earlier today (I can't remember) but I will attempt to work with my Tails drive, to see if the drive can then be detected.

Now that you've mentioned using dd as a backup, a few questions arise as to how this was done, which may reflect on the reasons for the problems with the external disk.

Can you supply the exact command you used for the back up? It might help greatly.

Typically dd is used to copy whole disks as images so that the image can be rewritten if necessary to the source disk to restore the filesystem and data to the whole disk. It restores the installation to that earlier state when the dd command wrote the image file.

It's also possible to use dd to copy a partition and for that partition to be restored.

One can also use dd to copy data and lodge it as a file on an external disk, but dd must be told where to place that file. If it isn't given that information by way of an offset, dd will just begin writing at the start of the external disk and proceed from there overwriting the metadata such as a partition table and headers that may be there.

If the partitioning information is thus overwritten, then the operating system is going to be unable mount partitions because it lacks the necessary information about the disk's partitioning. This is one hypothesis as to why the operating system cannot see the external disk, that is, the partitioning information has been overwritten.

Given these considerations, the exact dd command that you used could be revealing.

Backing up data can be done quite robustly and effectively without using tools like dd, but that's perhaps something to look at later.

 

[bob466]

My Tower shut down three times and all were Hardware issues...Power Supply...HDD and Motherboard.

These would be my first things to check...I've never had a USB Drive connected when the above happened...maybe the problem isn't with the External Drive.

 

[f33dm3bits]

First thing I would try is running a file-system check on whatever partition or volume the file-system is on.

 

[Priest_Apostate]

As @Condobloke mentioned in post #2, it's best if nothing is run that involves the 5TB drive if you want to rescue as much as possible from it. Any more activity on the disk will only alter the filesystem which will likely disturb it more than it already is.

One common approach is to use the testdisk program and the photorec program, both of which are outlined here: https://www.cgsecurity.org/wiki/TestDisk. There's a learning curve to using them but the documentation is quite extensive.

When recovering files, it's really necessary for best results to leave the problematical external drive unmounted, and use the recovery tools, testdisk and photorec, from either a live disk or the installed operating system if it's functioning properly. You will need a location to copy recovered files to which could be a usb, a connected computer, a directory on a properly functioning installed system, another external hard disk etc. If the installed system doesn't boot, use a live disk or rescue disk.

It's not clear why the system would spontaneously reboot if you were working in the
/var directory. Uninitiated rebooting is often associated with faulty memory, so one might consider testing the RAM with the memtest86+ program.

Check the logs to see if there's any information that might be helpful. One or more of the commands like the following can help:

journalctl -b
journalctl -b -x -p 3
journalctl -b | grep -i error
dmesg

Other than faulty RAM, there are other possibilities that can cause uninitiated reboots like an overheated CPU, power supply fluctuations or motherboard issues which can get a bit technical, but the RAM check would be the first check I'd make.

1. The only thing I tried was to perform a

sudo dd if=/dev/sdg2 status=progress | gzip -c > run/media/XXXXXXXX/FFE1-CBSE/BACKUP_PLUS_Backup_1.img.gz

- the drive was not mounted, as I am unable to do so at this time (but it is able to be detected via the

blkid

command. @osprey

2. Thanks for the link to programs - while I'm reading them (and wanting to try this out on another drive, I'm not comfortable performing the test on this drive just yet.

The dd command from above came from this article.

3. I am reviewing the dmesg file - but nothing stands out. I will attempt to research the other commands you provided.

4. I completed the

sudo memtester 9G

- which completed 9 loops with everything showing up as being okay.

 

[osprey]

1. The only thing I tried was to perform a

sudo dd if=/dev/sdg2 status=progress | gzip -c > run/media/XXXXXXXX/FFE1-CBSE/BACKUP_PLUS_Backup_1.img.gz

- the drive was not mounted, as I am unable to do so at this time (but it is able to be detected via the

blkid

command. @osprey

The dd command from above came from this article.

Thanks for that command. Unfortunately, that command does not seem to appear on the link which you provided. Just to be clear your command was:

sudo dd if=/dev/sdg2 status=progress | gzip -c > run/media/XXXXXXXX/FFE1-CBSE/BACKUP_PLUS_Backup_1.img.gz

but the command I see in the article is:

sudo dd if=/dev/disk/by-id/YOUR_FLASH_DRIVE status=progress | gzip -c > /home/USERNAME/backups/BACKUP_NAME.img.gz

The difference is that your command sends the data to a system directory under the directory
/run. I assume you were in the root directory when executing the command because then you could use "run" instead of "/run". However, the command in the article sends the data to a file in the home directory where it will stay until the user decides to do something with it. The
/home/$USER directory contrasts with the /run directory and its subdirectories since /run and everything under it is all on a temporary filesystem called "tmpfs" which is deleted when the user powers off or reboots. In tmpfs, the "backup" file would need to be moved out of there once created, to a more permanent location. If left in tmpfs, it's not a backup at all really because it will disappear.

The partition of /dev/sdg2 I presume is from the installed system and not the external disk, so it's unclear how this could affect the external disk. Perhaps if you could provide the output of the following commands as root, with the external drive connected, it may help further:

lsblk
blkid

It's good to know that memory is looking to be in good order with your testing. Now that factor can likely be discounted in the issue. It seems there's still a way to go to unravel this one.

Thanks for the link to programs - while I'm reading them (and wanting to try this out on another drive, I'm not comfortable performing the test on this drive just yet.

If you boot up a live disk which has testdisk and photorec on them, they can recover files on the external drive, if they see it, whilst the external drive is unmounted, so no more harm would be done to the external disk.

 

[Priest_Apostate]

Okay, I'm now confused: I did copy/paste that command from that page - I only modified the original command (sudo dd if=/dev/disk/by-id/YOUR_FLASH_DRIVE status=progress | gzip -c > /home/USERNAME/backups/BACKUP_NAME.img.gz) to the data-path I mentioned in #16:

sudo dd if=/dev/sdg2 status=progress | gzip -c > run/media/XXXXXXXX/FFE1-CBSE/BACKUP_PLUS_Backup_1.img.gz

 

[Priest_Apostate]

I am also a bit confused: there have been multiple recommendations to not do any work on the external disk, as that could further impact in my ability to recover the data - but there are then recommendations to perform work on the disk.

Is this to say that filesystem check (fsck) won't cause further complications - and if not, what work should I avoid doing on the drive? I can't mount the drive, so it isn't as if I can write data to it, so I am getting confused.

 

[bob466]

OP,

In your other Thread..you were saying you wanted to "clean" out var and I said you shouldn't touch System Files...leave them be...remember ?

In this Thread you say...you were "cleaning" var and the computer shut down...more like the Distro shut down because what you did in var...looks like the chickens have come home to roost.