DMAPI (Data Management API)

Discussion in 'Filesystem' started by Jarret W. Buse, Jul 22, 2013.

  1. Jarret W. Buse

    Jarret W. Buse Well-Known Member Staff Writer

    Messages:
    175
    Likes Received:
    259
    Trophy Points:
    63
    DMAPI (Data Management Application Programming Interface)

    The DMAPI is an interface used by the XDSM specification which allows for special interfaces to the files. The interface is file system independent, meaning that someone who creates an interface can use it on any DMAPI file system.

    Originally, DMAPI allowed for a file system to transfer file data from a primary (online) storage unit (harddisk) to a secondary (offline) storage unit (backup device). This is not special in itself, but the transfer is transparent. The files are not time-stamped by the Linux Kernel, nor are user applications aware of the data transfer.

    DMAPI was also originally implemented to allow unused files to be stored offline, usually before on tape, until the file was requested. Once requested, the file was retrieved for use by the requesting user. The method worked to allow the file system to be larger than the physical storage device (harddisk). In some cases, it may be termed a “bottomless” disk. This concept can still be used on server systems in a network environment.

    NOTE: On newer systems, the offline storage can be on external drives such as SATA.

    In server environments, DMAPI could be:

    • set to watch for read, writes or other events (such as mounting, low disk space, etc.)
    • used to punch hole a file (copy it to offline storage) and free the space on disk keeping only the first block
    • invisible read and writes can occur and not update the file's timestamp
    • used to read a file from offline storage and not require to back it up since it is already on tape

    NOTE: DMAPI does not store directories or inodes on tape.

    You need to remember that the first block is not migrated to disk to allow for the OS to check a file's type (stored in the first few bytes of a file). The first few bytes can be checked for execution to determine if the file is binary or a shell script.

    If a file is requested that is currently on tape, the file is transferred back to the harddisk. If there is not enough free space, more files can be placed on tape. Once enough free space has been cleared for the requested file, the write is resumed for the user to have access to the requested file.

    More currently, DMAPI is the interface used by XDSM (Data Storage Management) API to use events to notify actions the Data Management (DM) application should perform.

    NOTE: For DMAPI use on a file system such as JFS, a library needs to be installed and a kernel patch performed.

    For a DM application to perform the actions set to occur when an event occurs, the DM application requires one of the following:

    • DM_RIGHT_NULL – no rights
    • DM_RIGHT_SHARED – right to read or query an object
    • DM_RIGHT_EXCL – right to read, query or modify an object

    NOTE: Multiple applications can have DM_RIGHT_SHARED for one object, but only one application can have DM_RIGHT-EXCL on a single object. That is, the one can be shared among applications, but another can only be exclusive to one application. Once an application has exclusive rights to a file, no other application can access it until it is released.

    Access rights can be determined by the DM application submitting one of the following:

    • dm_query_right() – determines the access rights to an object
    • dm_request_right() – requests rights to an object
    • dm_release_right() – releases access rights to an object

    When a dm_query_right is sent to a file, the DM application will be notified if it requires shared or exclusive rights. If the application has proper rights, it can then send a dm_request_right to gain access to the file. If approved, such as no application has exclusive rights, then the application can access the file. Once the application is done, it will submit a dm_release_right to the file and release its rights to the file so other applications can use the file.

    Sometimes, a portion of a file may need to be managed. For example, a specific area of a database that is accessed less than other portions. When managing a portion of a file, three items must be set:

    • offset start in file
    • length of monitored portion
    • events to monitor (read, write, etc.)

    DMAPI allows for rapid data transfer without requiring knowledge of the file system’s structure or workings.

    DMAPI applications can set file attributes which are not normal attributes in a file system without DMAPI. When a DM application is installed, new file attributes can be created that are used by the DM application. These can include:


    • Online/Offline status – On local harddisk or on offline storage
    • Purge status – Does the file still exist on harddisk after it has been requested
    • Stub size – Amount of file on disk (first few bytes)

    NOTE: The extra file attributes can sometimes be termed DMAPI File Metadata. It may be best to have a backup/restore system that can backup the DMAPI metadata.

    Attached Files:

    • slide.jpg
      slide.jpg
      File size:
      41.1 KB
      Views:
      105,233
    ryanvade and DevynCJohnson like this.
  2. DevynCJohnson

    DevynCJohnson Well-Known Member Staff Member Staff Writer

    Messages:
    1,336
    Likes Received:
    1,073
    Trophy Points:
    113
    Just in case someone asks, I will answer now.

    Which filesystems support DMAPI?

    XFS is the main supporter of DMAPI. Other supporting filesystems include JFS, VxFS, AdvFS, StorNext and GPFS.


    Why is my JFS filesystem not using DMAPI?

    Not all Linux kernels come with a JFS driver with added DMAPI support.

Share This Page