DMAPI (Data Management API)

J

Jarret W. Buse

Guest
DMAPI (Data Management Application Programming Interface)

The DMAPI is an interface used by the XDSM specification which allows for special interfaces to the files. The interface is file system independent, meaning that someone who creates an interface can use it on any DMAPI file system.

Originally, DMAPI allowed for a file system to transfer file data from a primary (online) storage unit (harddisk) to a secondary (offline) storage unit (backup device). This is not special in itself, but the transfer is transparent. The files are not time-stamped by the Linux Kernel, nor are user applications aware of the data transfer.

DMAPI was also originally implemented to allow unused files to be stored offline, usually before on tape, until the file was requested. Once requested, the file was retrieved for use by the requesting user. The method worked to allow the file system to be larger than the physical storage device (harddisk). In some cases, it may be termed a “bottomless” disk. This concept can still be used on server systems in a network environment.

NOTE: On newer systems, the offline storage can be on external drives such as SATA.

In server environments, DMAPI could be:

  • set to watch for read, writes or other events (such as mounting, low disk space, etc.)
  • used to punch hole a file (copy it to offline storage) and free the space on disk keeping only the first block
  • invisible read and writes can occur and not update the file's timestamp
  • used to read a file from offline storage and not require to back it up since it is already on tape

NOTE: DMAPI does not store directories or inodes on tape.

You need to remember that the first block is not migrated to disk to allow for the OS to check a file's type (stored in the first few bytes of a file). The first few bytes can be checked for execution to determine if the file is binary or a shell script.

If a file is requested that is currently on tape, the file is transferred back to the harddisk. If there is not enough free space, more files can be placed on tape. Once enough free space has been cleared for the requested file, the write is resumed for the user to have access to the requested file.

More currently, DMAPI is the interface used by XDSM (Data Storage Management) API to use events to notify actions the Data Management (DM) application should perform.

NOTE: For DMAPI use on a file system such as JFS, a library needs to be installed and a kernel patch performed.

For a DM application to perform the actions set to occur when an event occurs, the DM application requires one of the following:

  • DM_RIGHT_NULL – no rights
  • DM_RIGHT_SHARED – right to read or query an object
  • DM_RIGHT_EXCL – right to read, query or modify an object

NOTE: Multiple applications can have DM_RIGHT_SHARED for one object, but only one application can have DM_RIGHT-EXCL on a single object. That is, the one can be shared among applications, but another can only be exclusive to one application. Once an application has exclusive rights to a file, no other application can access it until it is released.

Access rights can be determined by the DM application submitting one of the following:

  • dm_query_right() – determines the access rights to an object
  • dm_request_right() – requests rights to an object
  • dm_release_right() – releases access rights to an object

When a dm_query_right is sent to a file, the DM application will be notified if it requires shared or exclusive rights. If the application has proper rights, it can then send a dm_request_right to gain access to the file. If approved, such as no application has exclusive rights, then the application can access the file. Once the application is done, it will submit a dm_release_right to the file and release its rights to the file so other applications can use the file.

Sometimes, a portion of a file may need to be managed. For example, a specific area of a database that is accessed less than other portions. When managing a portion of a file, three items must be set:

  • offset start in file
  • length of monitored portion
  • events to monitor (read, write, etc.)

DMAPI allows for rapid data transfer without requiring knowledge of the file system’s structure or workings.

DMAPI applications can set file attributes which are not normal attributes in a file system without DMAPI. When a DM application is installed, new file attributes can be created that are used by the DM application. These can include:

  • Online/Offline status – On local harddisk or on offline storage
  • Purge status – Does the file still exist on harddisk after it has been requested
  • Stub size – Amount of file on disk (first few bytes)

NOTE: The extra file attributes can sometimes be termed DMAPI File Metadata. It may be best to have a backup/restore system that can backup the DMAPI metadata.
 

Attachments

  • slide.jpg
    slide.jpg
    41.1 KB · Views: 107,231


Just in case someone asks, I will answer now.

Which filesystems support DMAPI?

XFS is the main supporter of DMAPI. Other supporting filesystems include JFS, VxFS, AdvFS, StorNext and GPFS.


Why is my JFS filesystem not using DMAPI?

Not all Linux kernels come with a JFS driver with added DMAPI support.
 
Just in case someone asks, I will answer now.

Which filesystems support DMAPI?

XFS is the main supporter of DMAPI. Other supporting filesystems include JFS, VxFS, AdvFS, StorNext and GPFS.


Why is my JFS filesystem not using DMAPI?

Not all Linux kernels come with a JFS driver with added DMAPI support.
SGI XFS and Cray DMF are also examples of DMAPI implementations. The model for DMAPI was developed originally at NASA Ames in 1989. The DMIG committee was formed in 1993/1994. DMAPI lives on in spirit in a number of other files systems, although the actual API does not conform to the "standard". Peter Lawthers wrote and excellent paper on DMAPI which was presented in the 14th IEEE symposium on mass storage. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=528242
 
This thread is about a decade old.

Might I ask your motivation for posting to it? You don't appear to be a spammer or anything like that. So, that's not a problem. Are you on some sort of mission and hunting down places to answer this question? I don't get it.

(I don't mind. I'm a moderator, so it's my job to be a little curious.)
 
I am the author and inventor of the DMIG spec. The spec is based on work I did at NASA in 1989/1990. Adopted almost wholesale by Cray... as Cray DMF, later SGI DMF, now HPE DMF. I worked with SGI to design the requirements for XFS now cxfs which also included the HSM DMAPI interface. Jim Orosz did the actual implementation at SGI. Peter Lawthers wrote up the standard for the IEEE (took my design for NASA and "sanitized" it for the DMIG group). I have written kernel modules for various Unix / Linux OSes over the last 3 decades, including SCSI stack ( with Matt Jacobs) and did the first Infiniband driver for Mellanox (now Nvidia). While at Seagate, I worked directly for Peter Braam, the inventor of Lustre. I also was Chief Architect for Seagate and architected the (now HPE) ClusterStor Lustre file system. I have a number of patents for things such as HPE GridRAID, and while at NASA I funded Garth Gibson and the Berkeley RAID team and then implemented the very first RAID-5 disk driver for NASA in 1989. I have given ACM lectures on RAID. I am about to retire in the next year or so... so I am looking around to "correct the record" on the history of these technologies... even if nobody cares about them anymore or they think "it has always been this way". Lloyd Alan Poston (aka Alan Poston)
 
Well now we know who you are, though it still doesn't explain the motivation. It's all good. Knock yourself out. Hopefully someone will see your answer via search engine results and have a solution.
 
Well now we know who you are, though it still doesn't explain the motivation.
The creator of DMAPI is just passionate about their creation :)
 
Meh... They're not breaking anything or causing any harm. If anything, they're adding information that someone may want someday. So, there's that.

As a mod, it's kinda my role to be a bit curious.
 

Members online


Latest posts

Top