Chapter 4. The DMF Daemon

The DMF daemon, dmdaemon(8), is the core component of DMF. The daemon passes messages between commands, the MSPs, and the kernel. It also assigns bit file identifiers (bfids) to migrated files and maintains the DMF database entries for offline copies.

When DMF is started, the daemon database is automatically initialized. To start the daemon manually, use the dmdaemon command, as follows (assuming your PATH environment variable includes /etc/dmf/dmbase/etc):

dmdaemon

Typically, dmdaemon should be called as part of the normal system startup procedure by using a direct call in a system startup script in the /etc/rc2.d directory.

After dmdaemon is activated, the dmget(1) and dmput(1) user commands can be used to manage file system space manually.

You can restart the daemon by using the dmdaemon command.

The following sections provide additional information about the daemon database and daemon processing.

Daemon Processing

After initialization, dmdaemon performs the following steps:

  1. Isolates itself as a daemon process.

  2. Checks for the existence of other dmdaemon processes. If another dmdaemon exists, the newer one terminates immediately.

  3. Initializes the dmdaemon log.

  4. Opens the daemon database.

  5. Initializes the daemon request socket.

  6. Initiates the MSPs.

  7. Enters its main request processing.

The daemon uses log files and journal files as described in “Daemon Logs and Journals”.

The main request processing section of the DMF daemon consists of the following sequence:

  • The select(2) system call, which is used to wait for requests or for a default time-out interval

  • A request dispatch switch to read and process requests detected by the select call

  • A time processor, which checks activities (such as displaying statistics and running the administrator tasks) done on a time-interval basis

This processing sequence is repeated until a stop request is received from the dmdstop(8) command. When a normal termination is received, the MSPs are terminated, the database is closed, and the logs are completed.

A typical request to the daemon starts with communication from the requester. The requester is either the kernel (over the DMF device interface) or a user-level request (from the command pipe). A user-level command can originate from the automated space-management commands or from an individual user.

After receipt, the command is dispatched to the appropriate command processor within the daemon. Usually, this processor must communicate with an MSP before completing the specified request. The commands are queued within the daemon and are also queued to a specific group of database entries. All entries referring to the same file share the same bfid. The command is dormant until the reply from the MSP is received or the MSP terminates. When command processing is completed, a final reply is sent to the issuing process, if it still exists.

A final reply usually indicates that the command has completed or an error has occurred. Often, error responses require that you analyze the dmdaemon log to obtain a full explanation of the error. An error response issued immediately usually results from an invalid or incorrect request (for example, a request to migrate a file that has no data blocks). A delayed error response usually indicates a database, daemon, or MSP problem.

DMF Daemon Database and dmdadm(blank)

The DMF daemon maintains a database that resides in the directory HOME_DIR/daemon_name (HOME_DIR is specified by the HOME_DIR configuration parameter). This database contains information about the offline copies of a given file, as well as some information about the original file. The database also contains the bit file identifier (bfid), which is assigned when the file is first migrated.

Other information maintained on a per-entry basis includes the following:

  • File size (in bytes)

  • MSP name and MSP recall path

  • Date and time information, including the following:

    • Time at which the database record was created

    • Time at which the database record was last updated

    • A check time for use by the administrator

    • A soft-delete time, indicating when the entry was soft-deleted

  • Original device and inode number

  • Base portion of the original file name, if known

The dmdadm(8) command provides maintenance services for the daemon database.

dmdadm executes directives from stdin or from the command line when you use the -c option. All directives start with a directive name followed by one or more parameters. Parameters may be positional or keyword-value pairs, depending on the command. White space separates the directive name, keywords, and values.

When you are inside the dmdadm interface (that is, when you see the adm command_number > prompt), the command has a 30-minute timeout associated with it. If you do not enter a response within 30 minutes of the prompt having been displayed, the dmdadm session terminates with a descriptive message. This behavior on all the database administrative commands limits the amount of time that an administrator can lock the daemon and MSP databases from updates.

dmdadm Directives

The dmdadm directives are as follows:

Directive 

Description

count 

Displays the number of records that match the expression provided.

create 

Deletes an existing database record.

dump 

Prints the specified database records to standard out in ASCII; each database field is separated by the pipe character (|).

help 

Displays help.

list 

Shows the fields of selected database records. You may specify which fields are shown.

load 

Applies records to the database obtained from running the dump directive.

quit 

Stops program execution after flushing any changed database records to disk. The abbreviation q and the string exit produce the same effect.

set 

Specifies the fields to be shown in subsequent list directives.

update 

Modifies an existing database record.

The syntax for the dmdadm directives is summarized as follows:

count selection [limit]
delete selection [limit]
dump selection [limit]
help
list selection [format]
load filename
quit (or q or exit)
set [format]
update selection [limit] to fields...  

The value for selection can be one of the following:

  • A bfid or range of bfids

  • The keyword all

  • A period (.), which recalls the previous selection

  • An expression involving any of the above, field value comparisons, and, or, or parentheses.

A field value comparison may use < (less than), > (greater than), = (equal to), <= (less than or equal to), or >= (greater than or equal to) to compare a field keyword to an appropriate value.

The syntax for selection is as follows:

selection       ::=    or-expr
or-expr        ::=    and-expr [ or or-expr  ]
and-expr      ::=    nested-expr [ and and-expr  ]
nested-expr    ::=    comparison | ( expression )
comparison    ::=    bfid-range | field-keyword op  field-value
op           ::=    < | > | = | >= | <=
bfid-range      ::=    bfid [ - bfid] | [bfid - [bfid]] | key-macro
key-macro     ::=    all
field-keyword   ::=    name or abbreviation of the record field
field-value     ::=    appropriate value for the field
bfid          ::=    character representation of the bit file identifier

Thus valid selections could be any of the following:

305c74b200000010-305c74b200000029
7fffffff000f4411-
-305c74b2000004c8
all
origsize>1m
. and origage<7d
mspkey 456 to origuid 2570

dmdadm Field and Format Keywords

The field keywords listed below specify new values for fields. Some of the keywords are valid only if you also specify the -u option.

Keyword 

Description

checkage (ca) 

The time at which the database record was last checked; the same as checktime, except that it is specified as age. Valid only in unsafe (-u) mode.

checktime (ct) 

The time at which the database record was last checked; an integer that reflects raw UNIX time. Valid only in unsafe (-u) mode.

deleteage (da) 

The time at which the database record was soft-deleted; the same as deletetime, except that it is specified as age. Valid only in unsafe (-u) mode.

deletetime (dt) 

The time at which the database record was soft-deleted; an integer that reflects raw UNIX time. Valid only in unsafe (-u) mode.

mspname (mn) 

The name of the MSP with which the file is associated; a string of up to 8 characters. Valid only in unsafe (-u) mode.

mspkey (mk) 

The string that the MSP can use to recall a database record; a string of up to 50 characters. Valid only in unsafe (-u) mode.

origage (oa) 

Time at which the database record was created; the same as origtime, except that it is specified as age.

origdevice (od) 

Original device number of the file; an integer.

originode (oi) 

Original inode number of the file; an integer.

origname (on) 

Base portion of the original file name; a string of up to 14 characters.

origsize (os) 

Original size of the file; an integer.

origtime (ot) 

Time at which the database record was created; an integer that reflects raw UNIX time.

origuid (ou) 

Original user ID of the database record; an integer.

updateage (ua) 

Time at which the database record was last updated; the same as updatetime, except that it is specified as age.

updatetime (ut) 

Time at which the database record was last updated; an integer that reflects raw UNIX time.

The time field keywords (checktime, deletetime, origtime, and updatetime) have a value of either now or raw UNIX time (seconds since January 1, 1970). These keywords display their value as raw UNIX time. The value comparison > used with the date keywords means newer than the value given. For example, >36000 is newer than 10AM on January 1, 1970, and >852081200 is newer than 10AM on January 1, 1997.

The age field keywords (checkage, deleteage, origage, and updateage) let you express time as age. age is a string in a form such as 8w12d7h16m20s, meaning 8 weeks, 12 days, 7 hours, 16 minutes, and 20 seconds old. The age keywords display their value as an integer followed by w, d, h, m, or s (weeks, days, hours, minutes, and seconds, respectively). The comparison > used with the age keywords means older than the value given (that is, >5d is older than 5 days).

The limit keywords restrict the records acted upon:

Keyword 

Description

recordlimit (rl) 

Limits the number of records acted upon to the value that you specify; an integer.

recordorder (ro) 

Specifies the order that records are scanned; may be either bfid or data. bfid specifies that the records are scanned in bfid order. data specifies that the records are scanned in the order in which they are found in the database data file. data is more efficient for large databases, although it is essentially unordered.

The format keyword selects a format to use for the display. If, for example, you want to display fields in a different order than the default or want to include fields that are not included in the default display, you specify them with the format keyword. Values for format can be default, keyword, or a list of field keywords enclosed in quotation marks.

For any field that takes a byte count, you may append the letter k, m, or g (in either uppercase or lowercase) to the integer to indicate that the value is to be multiplied by one thousand, one million, or one billion, respectively.

The following is sample output from the dmdadm list directive; recordlimit 20 specifies that you want to see only the first 20 records.

adm 3>list all recordlimit 20
            BFID     ORIG        ORIG   ORIG MSP      MSP
                      UID        SIZE    AGE NAME     KEY
------------------------------------------------------------------------
305c74b200000010    20934    69140480   537d silo1    88b49f
305c74b200000013    26444      279290   537d silo1    88b4a2
305c74b200000014    10634       67000   537d silo1    88b4a3
305c74b200000016    10634   284356608   537d silo1    88b4a5
305c74b200000018    10634     1986560   537d silo1    88b4a7
305c74b20000001b    26444      232681   537d silo1    88b4aa
305c74b20000001c    10015     7533688   537d silo1    88b4ab
305c74b200000022    8964     23194990   537d silo1    88b4b1
305c74b200000023    1294    133562368   537d silo1    88b4b2
305c74b200000024    10634       67000   537d silo1    88b4b3
305c74b200000025    10634   284356608   537d silo1    88b4b4
305c74b200000026    10634     1986560   537d silo1    88b4b5
305c74b200000027    1294      1114112   537d silo1    88b4b6
305c74b200000028    10634       25270   537d silo1    88b4b7
305c74b200000029    1294     65077248   537d silo1    88b4b8
305c74b20000002b    9244      2740120   537d silo1    88b4ba
305c74b200000064    9335         9272   537d silo1    88b4f3
305c74b200000065    9335        10154   537d silo1    88b4f4
305c74b200000066    9335         4624   537d silo1    88b4f5
305c74b200000067    9335        10155   537d silo1    88b4f6
adm 4>

The following example displays the number of records in the database that are associated with user ID 11789 and that were updated during the last five days:

adm 3>count origuid=11789 and updateage<5d
72 records found.

dmdadm(blank) Text Field Order

The text field order for daemon records generated by the dmdump(8), dmdumpj(8), and the dump directive in dmdadm is listed below. This is the format expected by the load directives in dmdadm:

  1. bfid

  2. origdevice

  3. originode

  4. origsize

  5. origtime

  6. updatetime

  7. checktime

  8. deletetime

  9. origuid

  10. origname

  11. mspname

  12. mspkey

To isolate the mspname and mspkey from the daemon records soft-deleted fewer than three days ago, use the following command:

dmdadm -c "dump deleteage<3d and deletetime>0" | awk "-F|" '(print $11,$12}'

Daemon Logs and Journals

The DMF daemon uses log files to track various types of activity. Journal files are used to track DMF database transactions.

The ASCII log of daemon actions has the following format (SPOOL_DIR refers to the directory specified by the SPOOL_DIR configuration parameter):

SPOOL_DIR/daemon_name/dmdlog.yyyymmdd

The file naming convention is that yyyy, mm, and dd correspond to the date on which the log was created (representing year, month, and day, respectively). Logs are created automatically by the DMF daemon.


Note: Because the DMF daemon will continue to create log files and journal files without limit, you must remove obsolete files periodically by configuring the run_remove_logs and run_remove_journals tasks in the configuration file, as described in “Configuring Daemon Maintenance Tasks” in Chapter 2.


The DMF daemon automatically creates journal files that track database transactions. They have the following path name format (JOURNAL_DIR refers to the directory defined by the JOURNAL_DIR configuration parameter):
JOURNAL_DIR/daemon_name/dmd_db.yyyymmdd[.hhmmss]

Existing journal files are closed and new ones created in two circumstances:

  • When the first transaction after midnight occurs

  • When the journal file reaches size defined by the JOURNAL_SIZE configuration parameter

When the first transaction after midnight occurs, the existing open journal file is closed, and the suffix .235959 is appended to the current file name no matter what the time (or date) of closing. The closed file represents the last (or only) transaction log of the date yyyymmdd. A new journal file with the current date is then created.

When the journal file reaches JOURNAL_SIZE, the file is closed and the suffix .hhmmss is added to the name; hh, mm, and ss represent the hour, minute, and second of file closing. A new journal file with the same date but no time is then created.

For example, the following shows the contents of a JOURNAL_DIR/daemon_name directory on 15 June 1998:

dmd_db.19980604.235959  dmd_db.19980612.235959
dmd_db.19980605.235959  dmd_db.19980613.145514
dmd_db.19980608.235959  dmd_db.19980613.214233
dmd_db.19980609.235959  dmd_db.19980613.235959
dmd_db.19980610.235959  dmd_db.19980614.235959
dmd_db.19980611.094745  dmd_db.19980615
dmd_db.19980611.101937
dmd_db.19980611.110429
dmd_db.19980611.235959

For every date on which database transactions occurred, there will exist a file with that date and the suffix .235959, with the exception of an existing open journal file. Some dates have additional files because the transaction log reached JOURNAL_SIZE at a specified time and the file was closed.

You can configure daemon_tasks parameters to remove old journal files (using the run_remove_journals.sh task and the JOURNAL_RETENTION parameter. For more information, see “Configuring Daemon Maintenance Tasks” in Chapter 2.


Warning: If a daemon database becomes corrupt, recovery consists of applying journals to a backup copy of the database. Database recovery procedures are described in “Database Recovery” in Chapter 7.