J
Jarret W. Buse
Guest
DMAPI (Data Management Application Programming Interface)
The DMAPI is an interface used by the XDSM specification which allows for special interfaces to the files. The interface is file system independent, meaning that someone who creates an interface can use it on any DMAPI file system.
Originally, DMAPI allowed for a file system to transfer file data from a primary (online) storage unit (harddisk) to a secondary (offline) storage unit (backup device). This is not special in itself, but the transfer is transparent. The files are not time-stamped by the Linux Kernel, nor are user applications aware of the data transfer.
DMAPI was also originally implemented to allow unused files to be stored offline, usually before on tape, until the file was requested. Once requested, the file was retrieved for use by the requesting user. The method worked to allow the file system to be larger than the physical storage device (harddisk). In some cases, it may be termed a “bottomless” disk. This concept can still be used on server systems in a network environment.
NOTE: On newer systems, the offline storage can be on external drives such as SATA.
In server environments, DMAPI could be:
NOTE: DMAPI does not store directories or inodes on tape.
You need to remember that the first block is not migrated to disk to allow for the OS to check a file's type (stored in the first few bytes of a file). The first few bytes can be checked for execution to determine if the file is binary or a shell script.
If a file is requested that is currently on tape, the file is transferred back to the harddisk. If there is not enough free space, more files can be placed on tape. Once enough free space has been cleared for the requested file, the write is resumed for the user to have access to the requested file.
More currently, DMAPI is the interface used by XDSM (Data Storage Management) API to use events to notify actions the Data Management (DM) application should perform.
NOTE: For DMAPI use on a file system such as JFS, a library needs to be installed and a kernel patch performed.
For a DM application to perform the actions set to occur when an event occurs, the DM application requires one of the following:
NOTE: Multiple applications can have DM_RIGHT_SHARED for one object, but only one application can have DM_RIGHT-EXCL on a single object. That is, the one can be shared among applications, but another can only be exclusive to one application. Once an application has exclusive rights to a file, no other application can access it until it is released.
Access rights can be determined by the DM application submitting one of the following:
When a dm_query_right is sent to a file, the DM application will be notified if it requires shared or exclusive rights. If the application has proper rights, it can then send a dm_request_right to gain access to the file. If approved, such as no application has exclusive rights, then the application can access the file. Once the application is done, it will submit a dm_release_right to the file and release its rights to the file so other applications can use the file.
Sometimes, a portion of a file may need to be managed. For example, a specific area of a database that is accessed less than other portions. When managing a portion of a file, three items must be set:
DMAPI allows for rapid data transfer without requiring knowledge of the file system’s structure or workings.
DMAPI applications can set file attributes which are not normal attributes in a file system without DMAPI. When a DM application is installed, new file attributes can be created that are used by the DM application. These can include:
NOTE: The extra file attributes can sometimes be termed DMAPI File Metadata. It may be best to have a backup/restore system that can backup the DMAPI metadata.
The DMAPI is an interface used by the XDSM specification which allows for special interfaces to the files. The interface is file system independent, meaning that someone who creates an interface can use it on any DMAPI file system.
Originally, DMAPI allowed for a file system to transfer file data from a primary (online) storage unit (harddisk) to a secondary (offline) storage unit (backup device). This is not special in itself, but the transfer is transparent. The files are not time-stamped by the Linux Kernel, nor are user applications aware of the data transfer.
DMAPI was also originally implemented to allow unused files to be stored offline, usually before on tape, until the file was requested. Once requested, the file was retrieved for use by the requesting user. The method worked to allow the file system to be larger than the physical storage device (harddisk). In some cases, it may be termed a “bottomless” disk. This concept can still be used on server systems in a network environment.
NOTE: On newer systems, the offline storage can be on external drives such as SATA.
In server environments, DMAPI could be:
- set to watch for read, writes or other events (such as mounting, low disk space, etc.)
- used to punch hole a file (copy it to offline storage) and free the space on disk keeping only the first block
- invisible read and writes can occur and not update the file's timestamp
- used to read a file from offline storage and not require to back it up since it is already on tape
NOTE: DMAPI does not store directories or inodes on tape.
You need to remember that the first block is not migrated to disk to allow for the OS to check a file's type (stored in the first few bytes of a file). The first few bytes can be checked for execution to determine if the file is binary or a shell script.
If a file is requested that is currently on tape, the file is transferred back to the harddisk. If there is not enough free space, more files can be placed on tape. Once enough free space has been cleared for the requested file, the write is resumed for the user to have access to the requested file.
More currently, DMAPI is the interface used by XDSM (Data Storage Management) API to use events to notify actions the Data Management (DM) application should perform.
NOTE: For DMAPI use on a file system such as JFS, a library needs to be installed and a kernel patch performed.
For a DM application to perform the actions set to occur when an event occurs, the DM application requires one of the following:
- DM_RIGHT_NULL – no rights
- DM_RIGHT_SHARED – right to read or query an object
- DM_RIGHT_EXCL – right to read, query or modify an object
NOTE: Multiple applications can have DM_RIGHT_SHARED for one object, but only one application can have DM_RIGHT-EXCL on a single object. That is, the one can be shared among applications, but another can only be exclusive to one application. Once an application has exclusive rights to a file, no other application can access it until it is released.
Access rights can be determined by the DM application submitting one of the following:
- dm_query_right() – determines the access rights to an object
- dm_request_right() – requests rights to an object
- dm_release_right() – releases access rights to an object
When a dm_query_right is sent to a file, the DM application will be notified if it requires shared or exclusive rights. If the application has proper rights, it can then send a dm_request_right to gain access to the file. If approved, such as no application has exclusive rights, then the application can access the file. Once the application is done, it will submit a dm_release_right to the file and release its rights to the file so other applications can use the file.
Sometimes, a portion of a file may need to be managed. For example, a specific area of a database that is accessed less than other portions. When managing a portion of a file, three items must be set:
- offset start in file
- length of monitored portion
- events to monitor (read, write, etc.)
DMAPI allows for rapid data transfer without requiring knowledge of the file system’s structure or workings.
DMAPI applications can set file attributes which are not normal attributes in a file system without DMAPI. When a DM application is installed, new file attributes can be created that are used by the DM application. These can include:
- Online/Offline status – On local harddisk or on offline storage
- Purge status – Does the file still exist on harddisk after it has been requested
- Stub size – Amount of file on disk (first few bytes)
NOTE: The extra file attributes can sometimes be termed DMAPI File Metadata. It may be best to have a backup/restore system that can backup the DMAPI metadata.