Undelete Files on Linux Systems

Discussion in 'Filesystem' started by DevynCJohnson, Jul 22, 2013.

  1. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    Often times, a computer user will delete a needed file accidentally and not have an easy way to regain or recreate the file. Thankfully, files can be undeleted. When a user deletes a file, it is not gone, only hidden for some time. Here is how it all works. On a filesystem, the system has what is called a file allocation list. This list keeps track of what files are where on the storage unit (hard-drive, MicroSD card, flash-drive, etc.). When a file is deleted, the filesystem will perform one of two tasks on the allocation table. The file's entry on the file allocation table marked as "free space" or the file's entry on the list is erased and then the space is marked as free. Now, if a file needs to be placed on the storage unit, the operating system will put the file in the space marked as empty. After the new file is written to the "empty space", the deleted file is now gone forever. When a deleted file is to be recovered, the user must not manipulate any files because if the "empty space" is used, then the file can never be retrieved.


    How do undelete program work?

    The majority of filesystems only mark the space as empty. With these filesystems, the undelete program looks at the file allocation list and copies the deleted file to another storage unit. If the files were copied to the same storage unit, then the user could lose other deleted files that are needed.
    Rarely do filesystems erase the allocation table entry. If a filesystem does, this is how an undelete program undeletes the file. The program searches the storage unit for file headers. All files have a specific string of code that is at the very beginning of the file. This is called a magic number. For example, the magic number of a compiled JAVA class is the hex number "CAFEBABE". So, an undelete program would find "CAFEBABE" and copy that file to another storage unit. Some undelete programs can look for a specific file type. The user may want a PDF, so the program searches for the hex magic number "25504446" which is the ASCII code for "%PDF". Other undelete programs search for all magic numbers. Then, the user can select which deleted files to recover.
    If a part of the file has been written over, then the whole file will be corrupted. The file can usually be recovered, but the contents will be useless. For instance, recovering a corrupted JPEG file will be pointless because the image viewer will not be able to generate an image from the file. So, the user has the file, but the file is useless.


    Device Locations:

    Before we continue, here is some information that will aid in directing the undelete utilities to the correct storage unit. All devices are in the /dev/ folder. The name of each device (not the name that the admin gave each partition or device) that is given by the system follows a predictable scheme. The second partition on the first SATA hard-drive would be sda2. The first letter indicates the storage type, in this case SATA, but an "s" could also mean SCSI, FireWire, or USB. The second letter "d" means disk. The third letter indicates the device number, so an "a" would be the first SATA and a "b" would be the second. The number displays the partition. To name the whole device with all partitions type the letters without the number. For this example that would be sda. Other possible letters "h" as the first letter. This means PATA hard-drive (IDE). As some examples of this scheme, a user has a computer with one SATA hard-drive (sda). The drive has four partitions - sda1, sda2, sda3, and sda4. The user deletes the third one, but sda4 remains sda4 until sda4 is reformatted. The user then plugs in a usb memory card (sdb) with one partition - sdb1. The user then adds a IDE hard-drive with one partition - hda1. Next, the user adds a SCSI hard-drive - sdc1. Then, the user removes the USB memory card (sdb). Now, the SCSI remains sdc, but if the SCSI is removed and added back, it will be sdb. Even though other storage device existed, the IDE drive will have the "a" because it is the first IDE drive. IDE devices are numbered separately from SCSI, SATA, FireWire, and USB devices.


    Recovery:

    Each undelete program has different abilities, features, and support for various filesystems. Below are some instructions for using TestDisk to recover files on a set of filesystems.

    FAT16, FAT32, exFAT (FAT64), NTFS, and ext2/3/4:

    TestDisk is an open-source, free program that works on Linux, *BSD, SunOS, Mac OS X, DOS, and Windows. TestDisk can be found here: http://www.cgsecurity.org/wiki/TestDisk. TestDisk can also be installed by typing "sudo apt-get install testdisk". TestDisk has many abilities, but this article is concerned with undeleting files.

    Open TestDisk in a terminal using root privileges by typing “sudo testdisk”.

    Now, the TestDisk command-line application will execute. The terminal appearance will change. TestDisk asks the user if it can keep logs. This is entirely up to the user. If the user is recovering files from the system storage, then do not keep a log. The choices are "Create", "Append", and "No Log". If the user wants a log, it is kept in that user's home folder.

    screen1.jpg

    In the following screen, the storage devices are listed using the /dev/* method. For my system, the system's storage unit is /dev/sda. This means that my storage unit is a SATA hard-drive (sd) and it is the first hard-drive (a). The size of each storage unit is displayed in Gigabytes. Use the up and down arrows to select a storage device and hit enter.

    screen2.jpg

    The next screen displays a list of partition table (also called partition map) types. Just as there is the file allocation table for files, there is a table for the partitions. Partitions are dividers on a storage device. For instance, on almost all Linux systems there is at least two partitions - EXT3/4 and Swap. Each partition table will be briefly described. TestDisk does not support all partition tables, so this is not a complete list.

    screen3.jpg

    Intel - This partition table is very common on Windows systems and many Linux systems. This table is also know as MBR.
    EFI GPT - This is usually used with Linux systems. This partition map is most recommended for Linux because the concept of logical/extended partitions does not apply to GPT (GUID Partition Table) tables. This means that a Linux user can multiboot many forms of Linux with one Linux OS on each partition. There are other advantages to using GPT, but that is beyond this article.
    Humax - Humax maps are used with device made by the South Korean company Humax.
    Mac - The Apple Partition Map (APM) is used by Apple devices.
    None - Some devices do not have a partition table. For instance, many Subor game consoles do not use a partition map. If a user tried to undelete a file on these devices thinking that the partition map was one of the other choices, the user will be confused by the fact that TestDisk does not find any filesystem or files.
    Sun - The Sun partition table is used by Sun systems.
    Xbox - The Xbox uses the Xbox partition map for its storage devices.

    If a user selects "Xbox" even though their system uses GPT, TestDisk will not be able to find a partition or filesystem. If it does, then it will guess incorrectly. (The image below displays the output when the incorrect partition type)

    xbox.jpg

    Once the user picks the correct choice for their device, on the next screen, select "Advanced".

    screen4.jpg

    Now, the user should see a list of all of their filesystems/partitions on the storage unit. If the user had chosen the wrong partition map, then here is where they will know if they made the incorrect selection. If there are no errors, highlight the partition that contains the deleted file by placing the text-based cursor on it. Use the left and right arrows to highlight "List" on the bottom of the terminal. Now, hit enter.

    screen5.jpg

    A new screen is displayed with a list of files and folders. The whitish files are current files that are not deleted. The red files have been deleted. On the far right column is the files' names. The next column over to the left is the creation date of the file. One column over to the left again is the files' sizes in bytes. To the far left is a column with dashes, "d"s, "r"s, "w"s, and "x"s. These are the file permissions. A "d" indicates that the item is a directory. The rest of the permission syntax is irrelevant to this article. The item on the top of the file list titled "." means the current directory. The second object titled ".." means go up one directory, so a user can move up a directory by selecting this line. For an example, I will go into the directory "Xaiml_Dataset". The folder is nearly full of deleted files. I will undelete "computers.xaiml" by pressing "c" on the keyboard. I am now asked to select a destination directory. Of course, I will put it on another partition. I am in my home folder, and I press "c". It does not matter what folder is highlighted. The current folder is the destination directory. Now, I am back to the list of files. At the top of the screen is a message that says "Copy Done!". In my home folder is a folder called "Xaiml_Dataset", and inside is the Xaiml file. If I press "c" on more deleted files, they will be placed in the new folder without asking me for a destination.

    screen6.jpg

    screen7.jpg

    screen8.jpg

    When finished press "q" repeatedly until the normal terminal is seen. The folder "Xaiml_Dataset" can only be accessed by the root. To fix this, use root privileges to change the folder permissions and the contained files. After that, the files have been recovered and accessible to the user.

    ReiserFS:

    To undelete a file from a ReiserFS filesystem, make a backup of all of the files on the partition because this method can cause the file to be lost if something goes wrong. Next, execute the following command where DEVICE is the device in the form sda2. Some files will be put in the lost+found directory and other will remain where they were before deletion.

    Code:
    reiserfsck --rebuild-tree --scan-whole-partition /dev/DEVICE
    Recover Deleted File that is Still Open in Program:

    Assume a user accidentally deletes a file that a program has open. The file of the hard-drive was deleted, but the program is using a copy of the file that is on the RAM. Thankfully, there are two easy solutions.
    If the program has save capabilities like a text editor, the user can resave the file. Thus, the file editor will write the file to the hard-drive.
    Assume that this is an MP3 file in a music player. The music player cannot save the MP3 file. This task requires a little more time than the previous situation. Unfortunately, this method does not work on all systems and applications. To begin, type the following command.

    Code:
    lsof -c smplayer | grep mp3
    This command LiSts all of the Open Files used by Smplayer. This list is piped (given) to grep which searches for mp3. The output looks like the following.

    Code:
    smplayer  10037 collier  mp3    169r      8,1  676376  1704294 /usr/bin/smplayer
    (deleted)
    Now, type the following command to recover the file directly from the RAM (on Linux systems, /proc/ is the RAM) and copy it to a folder of choice. The "cp" is the copy command. The 10037 number comes from the process number given in the output. The 169 is the file descriptor shown in the output. The "~/Music/" is the destination directory. Lastly, "music.mp3" is the file name that the user wants for the file.

    Code:
    cp /proc/10037/fd/169 ~/Music/music.mp3

    Real Deletion:


    To make for sure that a file can never be recovered, use a command that "wipes" the hard-drive. Wiping the hard-drive means writing meaningless data to the disk. For example, many wiping programs write zeros, random letters, or random data to the hard-drive. No space is taken up or lost. The wiping program just overwrites the "empty space". If the storage unit is ever full of files with no free space remaining, then all of the previously deleted files will be gone.

    The purpose of wiping hard-drives is to make sure that private data is never seen. For illustration, a company may order new computers. The manager decides to sell the old computers. However, there is concern that the new owners may view company secrets or customer information like credit card numbers and addresses. Thankfully, a computer technician in the company can wipe the hard-drives before selling the old computers.

    To install secure-delete, a wiping program, type "sudo apt-get install secure-delete". This installs a set of four commands that make sure that deleted files are never recovered.

    srm - permanently delete a file. Usage: srm -f ./secret_file.txt
    sfill - wipe the free space. Usage: sfill -f /mount/point/of/partition
    sswap - wipe swap space. Usage: sswap -f /dev/SWAP_DEVICE



    If computers were to truly delete a file selected for deletion, then more time would be required to perform the task. It is quick and easy to mark some space as free, but to make the file gone forever requires time. Wiping a storage unit, for instance, takes a few hours to complete (depending on storage size). Overall, the current system works well because even when a user empties the recycle bin, they still have another chance to change their mind.

    Attached Files:

    • slide.jpg
      slide.jpg
      File size:
      66.3 KB
      Views:
      118,391
    Haider92, Kovax, darlof and 6 others like this.
  2. Rob

    Rob Administrator Staff Member

    Messages:
    565
    Likes Received:
    171
    Trophy Points:
    43
  3. cm1967

    cm1967 New Member

    Messages:
    5
    Likes Received:
    3
    Trophy Points:
    3
    Very useful info! Thanks for the great article.
    DevynCJohnson likes this.
  4. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    Thank you for reading the article. For articles that you like, click the "Like" link near the bottom on the article.
    Haider92 likes this.
  5. cm1967

    cm1967 New Member

    Messages:
    5
    Likes Received:
    3
    Trophy Points:
    3
    Thanks, Devyn. I meant to do that and forgot. :)
    DevynCJohnson likes this.
  6. cm1967

    cm1967 New Member

    Messages:
    5
    Likes Received:
    3
    Trophy Points:
    3
    I was pleasantly surprised at how detailed the article was written.
    DevynCJohnson likes this.
  7. situ

    situ New Member

    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    1
    very useful info Devyn, thank u for sharing
    DevynCJohnson likes this.
  8. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
  9. kpenrose

    kpenrose New Member

    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    1
    Great info, but I have a quick question: If I have deleted a file on a windows system, could I boot with a linux recovery disk (something like knoppix) and recover deleted files on the windows partition using your methods?
    DevynCJohnson likes this.
  10. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    Excellent question. Yes, if you delete a file on Windows, you can use a live Linux disc to retrieve your file. Just make sure testdisk is installed or can be installed on the live Linux system.

    Just in case someone asks in the future, a live Linux disc can be used to retrieve a deleted file on an Apple system.
    Haider92 likes this.
  11. Amos

    Amos New Member

    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    1
    For the cases of deleted open files, especially if you can't tell the program to re-write the file (e.g. a daemon which keeps running and updating an sqlite database), there is also the fdlink (https://github.com/amosshapira/fdlink) kernel module I wrote, which is the basis for a potentially more current frelink (https://github.com/pkt/frelink) project. This allows you to actually link back a deleted file to any name under the same file system it was deleted from, so absolutely no data loss involved and it's designed to be installable without rebooting the system, so the daemon holding the file open doesn't have to be stopped.
    DevynCJohnson likes this.
  12. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    Cool! Thanks for sharing! Welcome to the site and enjoy!
    Haider92 likes this.
  13. fault-tolerant

    fault-tolerant New Member

    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    1
    I was less lucky with TestDisk on Ubuntu 13.10. I've booted from external flash drive first. It has restored plenty of files - with zero length, only a couple of files were actually restored - out of dozens that were deleted in three directories. Nevertheless, ExtUndelete was way more helpful, even though it is less convenient for use - only a command line interface. It has restored about 60% of the files, luckily.

    Normally you are reading this in crisis, so good luck! :)
    DevynCJohnson likes this.
  14. LaneLester

    LaneLester New Member

    Messages:
    2
    Likes Received:
    1
    Trophy Points:
    3
    Those are great instructions, and I plan to save them for possible future use. Unfortunately, they didn't work for my present disaster. In deleting some photos with Ctrl-click, I neglected to notice that a folder that I wanted to keep was already selected. I believe I did the deletion with GwenView (or maybe Krusader) in Kubuntu 12.04.

    I used TestDisk to navigate to the deleted folder, but was appalled to see it in red. This was on a separate partition to which I had done no operations since the deletion. I'm puzzled why the folder was lost.
    DevynCJohnson likes this.
  15. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    The red means it was deleted (which you have). It can still be recovered, or have you tried? White means files you can see in a file manager. Red means it was deleted and is still there or at least a portion of the file. Your files are still recoverable.
    Haider92 likes this.
  16. LaneLester

    LaneLester New Member

    Messages:
    2
    Likes Received:
    1
    Trophy Points:
    3
    Thank you for the clarification, Devyn. I checked again, and the folder I deleted was indeed red. Unfortunately, when I navigated into the folder I found:
    Code:
    Directory /People/Family
    No file found, filesystem may be damaged.
    So, it seems the files are really gone. I don't think I've done any other manipulation of the partition, other than to delete some other files. So it's still a bit puzzling how those went away.

    Here's hoping I have a CD of those photos! Or maybe on my external drive. Or maybe... :(

    Lane
  17. DevynCJohnson

    DevynCJohnson Super Moderator Staff Member Staff Writer

    Messages:
    973
    Likes Received:
    857
    Trophy Points:
    93
    That sucks. I wonder why you were unable to recover the file. o_O
    Haider92 likes this.

Share This Page