XFS File System

Discussion in 'Filesystem' started by Jarret W. Buse, Jul 29, 2013.

  1. Jarret W. Buse

    Jarret W. Buse Well-Known Member Staff Writer

    Messages:
    183
    Likes Received:
    270
    Trophy Points:
    63
    XFS File System

    The XFS file system is an extension of the Extent File System (EFS). XFS was originally referred to as the ‘X’ File System (XFS) and the name was used ever since. The file system was created by Silicon Graphics in 1993.


    XFS was first used on IRIX 5.3 and ported to Linux in 2001. It was first added to the Linux kernel version 2.4 in 2002.

    The file name size limit is 255 characters. To support large files and a larger partition (more addressing values), the file system is 64-bit. The file and space limitations are as follows:

    32-bit system 64-bit system
    File size: 16 Terabytes 16 Exabytes
    File system: 16 Terabytes 18 Exabytes

    For added file growth, XFS allows a large number of inodes and directories.

    The file system consistency is guaranteed by the use of Journaling. The Journal size is calculated by the partition size used to make the XFS file system. If a crash occurs, the Journal can be used to ‘redo’ the transactions. When the file system is mounted and a previous crash is detected, the recovery of the items in the Journal is automatic.

    For support of faster throughput, Allocation Groups are used. Allocation Groups provide for simultaneous I/O by multiple application threads at once. The ability allows systems with multiple processors or multi-core processors to provide better throughput with the file system. These benefits are better when the XFS file system spans multiple devices.

    For multiple devices to be used within the same XFS file system, RAID 0 can be implemented. The more devices used in the striped file system, the higher the throughput can be achieved.

    To also provide higher throughput, XFS uses Direct I/O to allow data retrieved from the XFS file system to go directly to the application memory space. Since the data is bypassing cache and the processor, the retrieval and writing of data is faster.

    Another feature to provide faster throughput is Guaranteed Rate I/O. Guaranteed Rate I/O provides a method to reserve bandwidth to and from the file system. The method provides faster access for real-time applications.

    Another ability of XFS to increase performance and reduce fragmentation is by using Delayed Allocation. This method works well for files of unknown size being written while they are being modified.

    Another method that XFS uses to prevent fragmentation is to use sparse files. Here the real file contains large sections of zeroes. The zeroes are eliminated, but metadata is used to represent the zeroes. Instead of writing all the zeroes it writes the metadata, saving space. When the file is accessed, it is expanded again to its normal state in memory.

    Even with the file system having methods to reduce fragmentation, it can still occur when it becomes low on free space. To help alleviate this issue, Online Defragmentation is a process which can move files into contiguous blocks to reduce fragmentation. The process can occur while the XFS volume is mounted and being used.

    Allocating space on the file system is accomplished by using Extents. To manage the free space on the file system, B+ Trees are used to track these spaces. Other file systems use a bitmap to track free and used space. Two B+ Trees are used to track free space on an XFS file system. One tree is used to store the starting block of the free extents, while the second B+ Tree indexes the number of free extents for each starting block. To write a file, the file system can check for a free space with enough contiguous extents, and then find the starting block to begin writing.

    NOTE: Bitmaps are used to track used and unused space. These bitmaps are not images, but a file where each bit represents an addressable block. Each bit is either on (1) or off (0) to represent if it is used or free.

    For space preservation, block sizes are variable and can be set as 512 bytes to 64kb. If a system will have many small files, then the smaller block size should be used. If larger files are to be used, then use larger block sizes. The block size is set at the time of file system creation. Wasted space by block size is discussed in the article on Extents.

    To provide more storage space than is physically available, Data Management API (DMAPI) can be utilized to support offline storage for unused files. Files that are rarely accessed can be moved to another storage device allowing the space to be used by other files. When the file is requested, DMAPI moves the offline file back to the hard drive for access by the application.

    When disk space is running low on an XFS volume, it is possible to perform an Online Resizing to increase the free space. Extra space is created by using unused partitions on other hard disks. The space is added to the existing file system increasing its size.

    If multiple user accounts are set up, as well as groups, it is possible to monitor the disk usage by each user and/or group. Atomic Disk Quotas allow for disk usage management to not only monitor the usage but to place limits on them.

    To provide better backups, Snapshots can be used to create a read-only ‘image’ of the file system so it can be backed up even though the ‘real’ file system is still being used and modified.

    XFS also provides for Native Backup/Restore capabilities. Utilities such as xfsdump and xfsrestore allow a user to backup and restore files while the file system is in use. The process can be done without creating a snapshot and can even be performed in multiple streams to various devices. The restore and backup process can be interrupted and resumed without causing issues.

    Attached Files:

    • slide.jpg
      slide.jpg
      File size:
      12.7 KB
      Views:
      125,003
    DevynCJohnson likes this.
  2. Cristal Skull

    Cristal Skull Member

    Messages:
    44
    Likes Received:
    23
    Trophy Points:
    8
    my favorite filesystem!
  3. Don Sturdy

    Don Sturdy New Member

    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    Correction, the Original XFS implementation under Irix at SGI was
    NOT derived nor heavily influenced by EFS of System V fame.
    In fact SGI explicitly set out to build a new FS from scratch for the high
    end Origin servers with a completely new design for a journaled
    FS. The late Jeff Beck was the first manager and cheerleader
    of the effort and along with numerous engineers who did the work
    on the first implementation. The XFS port to Linux was started
    in 1999 and took considerable work to "prove" that there was
    no contaminated code and every single header file and code
    file was examined line-by-line. The effort was further complicated
    by the use of Vnodes and the System V and BSD 4.X code and
    FS framework under Irix. Hence the Linux FS interfaces required
    some rethinking and rework to adapt the stock XFS code.

    The work Dave Chinner, aka RedHat, has done on XFS over the years
    has greatly improved the performance in numerous areas.

    It is simply too bad that SGI has forgotten XFS and all of the originals
    are now longer at the company to continue the legacy.
  4. dhiraj_bho

    dhiraj_bho New Member

    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    1
    Hey i just read this post and wanted to have XFS file system on my Red Hat Enterprise Linux box. When i tried to install RHEL 6.5 with XFS file system on /boot it prompted me with error as shown in screenshot attached.
    I had allocated /boot with xfs file system
    / with ext4 and allocated swap space also. Does it matter?
    I wanted to know why it is happening?

    Attached Files:

    DevynCJohnson likes this.
  5. Cristal Skull

    Cristal Skull Member

    Messages:
    44
    Likes Received:
    23
    Trophy Points:
    8
    it's because RHEL/CentOS not support native instalation on XFS filesystem
  6. dhiraj_bho

    dhiraj_bho New Member

    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    1
    I came to know that after starting installation but i want to know "why" RHEL not supporting this amazing file system on bootable partition?
    Help me because i did not got any links explaining that by Red Hat Customer portal.
  7. Cristal Skull

    Cristal Skull Member

    Messages:
    44
    Likes Received:
    23
    Trophy Points:
    8
    I think you did not understand well what is written in the documentation RedHat / CentOS.
    Both distributions are not supported for installing native kernel XFS. POINT!
    XFS is a filesystem developed especially for large files, not small files that the boot partition
  8. Jarret W. Buse

    Jarret W. Buse Well-Known Member Staff Writer

    Messages:
    183
    Likes Received:
    270
    Trophy Points:
    63
    When performing a fresh install of any OS, it is sometimes best to perform the formatting from the installer. If the installer does not support it, then it should not be attempted. Even then, some OSs may still allow you to do something that will bomb later. You would think that the programmers would set up the installers to not allow such problematic issues to be possible. Oh well, we are not perfect, but we try.
    DevynCJohnson likes this.
  9. dhiraj_bho

    dhiraj_bho New Member

    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    1
    Thanks that information is helpful. Why they just shown the option to select xfs file system on bootable partition?
  10. Jarret W. Buse

    Jarret W. Buse Well-Known Member Staff Writer

    Messages:
    183
    Likes Received:
    270
    Trophy Points:
    63
    It is a bad error on their part. It definitely seems they would disallow options that they do not support later. It wasn't thought through on their part.
  11. dhiraj_bho

    dhiraj_bho New Member

    Messages:
    4
    Likes Received:
    1
    Trophy Points:
    1
    I am really thankful that you answered each question i asked.
    Really appreciable.
    Best Regards...
  12. DevynCJohnson

    DevynCJohnson Well-Known Member Staff Member Staff Writer

    Messages:
    1,354
    Likes Received:
    1,083
    Trophy Points:
    113
    In general, if the /boot/ directory is placed on its own partition, the partition should be formatted with ext3/4 since ext3/4 are the default/standard Linux filesystems.
  13. Swapan

    Swapan New Member

    Messages:
    3
    Likes Received:
    3
    Trophy Points:
    3
    Hi Jarret,

    I went through your two blogs pertaining to DMAPI (the above one and XFS one).

    I was looking for exploring DMAPI on Linux for storing data on secondary storage for my free-time project. i realise that DMAPI is something that
    filesystem need to support.

    I see you mentioning about XFS as the filesystem supporting DMAPI.

    I did the following before reaching you.

    (1).Checked other filesystems like ZFS which do talk about HSM but thats all for the online tiered storage
    (2). No success exploring other FS et3, 4, Rieser for DMAPI support.
    (3). I do remember similar kind of functionality while working on openVMS OS, where we used to support HSM by using Remote Device facility (RDF)
    and Media Robotic utility(MRU) to control Tape Libraries.

    My questions :

    (1). Do the latest Linux distributions which have XFS, support DMAPI (from XFS) out of the box ?
    (2). Any particular Linux distribution and version you can recommend which will DMAPI built in ?
    (3). If not, what's the way out using DMAPI on Linux ? May be a different FS ? or Kernel and library modules for XFS to support DMAPI ?

    Thx.

    Regards,
    Swapan
    DevynCJohnson likes this.
  14. DevynCJohnson

    DevynCJohnson Well-Known Member Staff Member Staff Writer

    Messages:
    1,354
    Likes Received:
    1,083
    Trophy Points:
    113
    @Swapan , XFS has native DMAPI (to the best of my knowledge). JFS may have support, or you can apply a patch to your kernel or to the JFS module specifically. Other "obscure" filesystems support DMAPI, but no other main-stream filesystems support DMAPI. EXT2/3/4 does not have any DMAPI support. However, you may find that other filesystems could have alternatives to DMAPI. Many distros support XFS natively; if not, then install the driver. Try checking if Debian or Fedora support XFS out-of-the-box. Irix natively supports XFS. However, Irix is not Linux. Operating systems like *BSD, Solaris, and similar systems may not support XFS as well as Linux or Irix. You could compile your own kernel with built-in, low-level, native support for XFS. Doing so is easy. I can provide simple and quick instructions. Then, you can continue to use your preferred distro.
    Swapan likes this.
  15. Swapan

    Swapan New Member

    Messages:
    3
    Likes Received:
    3
    Trophy Points:
    3
    @DevynCJohnson

    Thank you for the reply.

    I tried my below stuff with Fedora 20, which has XFS. But DMAPI dosnt seem to be there by default and nor I was able to YUM update.

    I did the following:

    - Started working with USB drive.
    - Created XFS FS on it :

    $sudo mkfs.xfs -f /dev/sdb1 ---- WORKED

    - Tried mounting XFS with DMAPI ON :

    $sudo mount -t xfs -o dmapi,mtpt=/run/media/usr1 /dev/sdb1 /run/media/usr1
    mount : wrong fs type, bad option, bad superblock on /dev/sdb1


    - However, when I try mounting without DMAPI option, it works:

    $sudo mount -t xfs /dev/sdb1 /run/media/usr1

    - I tried updating DMAPI, but dint work :

    $yum install kmod-xfs xfsdump xfsprogs dmapi
    .....
    no package dmapi available
    .....


    It would be great if you can let me know the steps to get the DMAPI built which an be used with XFS.

    Thanks.
    DevynCJohnson likes this.
  16. DevynCJohnson

    DevynCJohnson Well-Known Member Staff Member Staff Writer

    Messages:
    1,354
    Likes Received:
    1,083
    Trophy Points:
    113
    DMAPI has been moved to the "xfs-cmds" package. To use DMAPI, your system must use devfs or udev. First, try compiling the code as a kernel module before making a kernel with native support.

    After looking here (ftp://ftp.kernel.org/pub/linux/utils/fs/xfs/), it appears that the Kernel-Developers have removed the DMAPI src. After some research, I see that support for the mainline kernel has been removed. You can still get DMAPI by going here (http://oss.sgi.com/cgi-bin/gitweb.cgi?a=project_list;pf=xfs), obtain the source code, and then make your own kernel module. SGI offers their own Linux kernel that already has the XFS tools and DMAPI added in (http://oss.sgi.com/projects/sgi_propack/). However, the kernel is version 2.4.*, so this is old. Their kernel comes packaged as an RPM (this helps save you time since you are already using Fedora).

    So, you have two options, use SGI's kernel or make your own module. Which would you like to try first?

    Helpful Links
    http://oss.sgi.com/projects/xfs/
    http://www.tack.ch/unix/dmapi/
    https://github.com/bonzini/xfstests/tree/master/dmapi/src/suite2/dist
    https://github.com/bonzini/xfstests
    Swapan likes this.
  17. Swapan

    Swapan New Member

    Messages:
    3
    Likes Received:
    3
    Trophy Points:
    3
    I would go ahead using SGI 2.4 kernel as of now. This will allow me get into testing DMAPI.

    Later, I would move to current kernel by building modules.

    Thanks.

    - Swapan
    DevynCJohnson likes this.
  18. Sparxz

    Sparxz New Member

    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    1
    Had a bad experince few years back with an Iomega 1 GB external Hard drive that used XFS. The drive was mainly for storing raw video footage among three people. I knew nothing about XFS at the time. I onced used the drive to do a backup of my own data for a reinstall of my system. Everthing seemed fine. Could read my data etc. After I went to retreive my data after formatting my desktop hard drive, my data on the XFS 'vanished', and the drive was reporting free space instead. Tried alot of things. Never got my data back, and caused a fall out with a friend of mine who forgot to tell me it happned to him even once(could have saved me alot of trouble!).
    Called Iomga before on behalf of a friend on a seperate issue, and they were not helpful. I have become very wary of XFS, that can perfrom vanishing tricks just as easily as FAT16 & FAT32. I have lost track of XFS devlopment, and any related maintenance tools. I have never lost data in Ext3 or EXT4, but occasionaly in NTFS, but later recovered the data. I jumped into BTRFS and have not had any problems even before it was offically declacred stable and reliable.
    I was surprised to read this article, not that I doubt the spec and capability of XFS, but the reliability issue will hang in my mind. It is possible that the problems I experinced was a firmware problem and not XFS, so Iwould like to read up on XFS devlopment, and reliabilty testing a bit more, before intrusting a singular copy(during large back-up process) of data, again in the future.
    DevynCJohnson likes this.

Share This Page