Data recovery tool messed with my OS

[blynx]

Hello guys,

I am new here and am currently experiencing a bad problem with my laptop. Not sure if this belongs here, but I could not find any help here using the search engine, so I decided to create a new thread.

I am using elementary OS on a thinkpad X230 which has been running fine without any troubles. However recently I discovered I had accidentally deleted a file a while ago (.json) and hoped to be able to recover it using photorec. I created a new folder in my home directory, which I intended to use for the recovered files. I let the program run and since it said to take over an hour for it to finish, left the laptop running while I went out. When I came back home I discovered that the process had stalled at some point, because there was too little space left on the SSD.

And this is the problem now. I cannot do anything at all. Not even login to my account anymore. If I attempt to run a process or programm, even delete or copy a file from the command line it will say

-bash: cannot create temp file for here-document: No space left on device

df -h lets me know, that the main partition is 100% full. The recovered files must have been more than 400GB.

Is there anybody who can help me with this matter? I have no idea what to do at this point... And I cannot afford to reinstall the system and loose the files on the machine.

Any help will be greatly appreciated.

Thank you,
B.

 

[Steve]

I am using elementary OS on a thinkpad X230 which has been running fine without any troubles. However recently I discovered I had accidentally deleted a file a while ago (.json) and hoped to be able to recover it using photorec. I created a new folder in my home directory, which I intended to use for the recovered files. I let the program run and since it said to take over an hour for it to finish, left the laptop running while I went out. When I came back home I discovered that the process had stalled at some point, because there was too little space left on the SSD.

Try to use Ctrl+Alt+1..6 to bring up a terminal. If you are legitimately unable to even do that to bring up a terminal to cd/rm files, then your next thing would be to poweroff and boot off of a LiveCD (hopefully you have one lying around - anything that can mount the filesystem and do basics, like "cd", "find", and "rm"). Delete some large ISOs, BMPs, etc and free up space. Then, try to find your missing file (<code>find ~/dir/ -iname '*.json' -execdir mv {} ~/recovered_json_files/;</code> might do it, but I'm still learning find) That's how to bring it back to a working system.

The problem is that you've now overwritten 100% of your free space with recovered files, which contained your missing JSON file. Additionally, if this was on an SSD and TRIM was executed, photrec wouldn't have been any help regardless. photorec works especially well on platter-based HDDs, as it goes through the entire drive, and performs file-carving - looking for matching file patterns based off of a hex header/footer (JPEG files are FFD8 and FFD9). There might be some remnant of it in the swap partition, but I cannot advise there.

 

[blynx]

Thank you Steve! Mounting it on a live OS worked fine, I saved important files first, then deleted some bigger files and could login without any troubles after that!

Everything working like before, except that my harddrive is full. Would you recommend a clean install after that or just continue using it after deleting all the excess data? And can you recommend a recovery tool that is more suitable for the task? It is kinda important for me to get that file back...

 

[Steve]

Everything working like before, except that my harddrive is full. Would you recommend a clean install after that or just continue using it after deleting all the excess data? And can you recommend a recovery tool that is more suitable for the task? It is kinda important for me to get that file back...

I'd suggest looking through the list of files photorec was able to recover. IF it's there, rejoice. If not, ... then I'd suggest looking back at old e-mails, backups, off-sites, etc. It helps to explain some how SSDs read/write physically. Traditional HDDs record data (servo information, checksums, in addition to the actual data) via a signal onto a magnetic medium (the platter). "Deleted" data is stored in the "free space" of the file-system, and is physically overwritten when the file-system requires more free space. SSDs work via memory chips and have a dedicated controller chip that keeps track of where "sectors" are, scattered across the multiple memory chips. If a single memory cell on a chip were to be rewritten over and over, it will wear out and die. This is where wear-leveling comes into play, putting some of the writes over here, writes over there, but the controller keeps track of how many writes are on the cells, and where LBA (logical block address) or Sector # X is at. When an SSD actually does a write, it has to actually take all of the data in a page within the SSD itself, erase it, and write new data in it. The TRIM command actually will actually tell the SSD controller to look for pages that are ready to be erased ahead of time. So while you may not be actually reading/writing to/from the drive, the SSD is quite active all the time, clearing pages, performing garbage collection, shuffling data if needed, etc.

Now what happens when photorec goes through and looks for freespace on the SSD? The OS asks the SSD what's in LBA # 1,234,567,890, the controller goes back, finds where that LBA is pointing to on the chip, page, cell, etc., and then returns it through the SATA controller/interface. So depending on how the SSD controller actually is programmed, it could say "this is free-space - return 0x00 (nothing)." If you are able to return data from this free space and write it back, then it would need to be written back to the SSD, erasing more of the free space. (If you are more interested, check out this video:

)

All this to say, it is most likely gone. Sorry.

Going forward: photorec is fine for this kind of stuff. Data recovery on SSDs are extremely difficult due to the TRIM command destroying data in the background, and even during imaging, power is applied and therefore the controller is shuffling data on the back-end. Data recovery on an HDD is much more possible/reliable. But:
1) ALWAYS save recovered files to a different drive than from ones you are trying to recover from.
2) When using photorec, limit filetypes. If you tell it to, it will recover as much as possible and will be done when either it's recovered as much as it can or the destination drive is full.

Like I said, try a "find . -iname '*.json'" to find all .JSON files. If it's there, rejoice! If not, I'm sorry.

Once you are done, delete the recovered files directory, let TRIM be ran, and go on. Make backups/snapshots going forward. There have been a large number of writes on the SSD, so the overall life of the SSD has been reduced, but you should be OK for a while still.

 

[blynx]

Thanks for the information Steve! It is almost too technical for me hehe, I'm not very experienced in this field ;-)

I couldn't find the file, but it seems that alot of the recovered files don't go by their original name and thus cannot be found by a search engine. I might turn to a professional for this matter, but have not decided yet...

 

[atanere]

I might turn to a professional for this matter, but have not decided yet.

Thanks to @Steve for the excellent video about drives and forensic recovery! About $850 to get your lost file back would be a bit steep for most of us, but professionals are expensive in all fields. I certainly would NOT suggest Best Buy!

Good luck... I hope you get it!