Eloquence B.07.00 - Database

Forward-logging

Contents:

• Introduction
• Recovering from abnormal server termination
• Forward-log configuration
• Forward-log management
• The dbrecover utility

Introduction

Forward logging adds an incremental backup mechanism to the Eloquence database which allows recovery of database changes since the last backup in case of a disk or operating system failure. If necessary, a backup is restored and the changes since the last backup are applied with the dbrecover utility.

Recovering from abnormal server termination

The situation of a database environment not shutdown cleanly can be caused by various reasons:

1. There is no more space left in either the volume(s) or the file system(s) where the volumes are located. In such a case, the server has no chance to shutdown the database environment cleanly and therefore just terminates immediately.

   Once the underlying problem is solved (eg. by creating additional data volumes or by providing additional disk space), a subsequent startup-recovery will always be successful.

2. The system was inadvertently switched off or went down due to a power interruption (eg. if no UPS device is attached) or an operating system failure (eg. a kernel panic due to insufficient or corrupted swap space).

   In such a case, the integrity of the log volume's contents determines whether a startup-recovery will be successful or not. If the server was operated synchronously (with the SyncMode configuration activated) damage in the log volume is less likely than with asynchronous operation.

3. The database environment is corrupted by either a disk failure or an operating system problem (eg. overwriting parts of the data volume with invalid data). A failure like this usually causes the server to abort immediately (internal server panic).

   Typically, a subsequent startup-recovery will fail because the data and/or log volumes are either not accessible anymore or their contents have been corrupted.

A startup-recovery will be successful if all data and log volumes are present and accessible and if their contents have not been corrupted. In particular, if the operating system had a chance to write all pending modifications to the volumes after the server terminated abnormally, the subsequent startup-recovery will succeed.

If any of these conditions is not met, it is possible that a startup-recovery will fail. This normally requires restoration of all data volumes from the last (on-line or off-line) backup. The database server will probably start successfully afterward.

However, any database modifications since the last backup would be lost unless they were contained in a forward-log. Combined with the last backup, the recovery from a forward-log would restore the contents of the entire database environment up to the very moment when the problem occurred.

In this way, a forward-log provides an additional safety mechanism in case the startup-recovery cannot be performed successfully.

Configuring Forward-logging

The [ForwardLog] section is contained in the new template configuration file which is part of Eloquence B.07.00 (eloqdb6.cfg in the /opt/eloquence6/newconfig/config directory on HP-UX and Linux, eloqdb6.cfg.sam in the etc installation subdirectory on Windows).

To update an existing configuration file, add the following lines just before the [Volumes] section at the end of the file:

    ### Forward log

    [ForwardLog]

    # FwLog         Configures the file, device or pipe to be used for forward-
    #               logging. Using the %N token in the file name activates
    #               automatic file management (not possible for devices or pipes).
    #               By default, forward-logging is inactive.
    #
    #               The examples below configure an automatically managed file
    #               and a pipe which compresses the data on-the-fly:

    #FwLog = /mnt/disk2/data/db-forward-%N.log
    #FwLog = |gzip -c >/mnt/disk2/data/db-forward.log.gz

    # FwRecovery    Configures the file, device or pipe to be used during forward
    #               recovery. If not set, the Log setting is used by default.
    #
    #               The example below configures a pipe which uncompresses the
    #               data on-the-fly:

    #FwRecovery = |gzip -dc /mnt/disk2/data/db-forward.log.gz

    # FwOnFailure   Configures the action to be taken in case the forward-log
    #               cannot be written, e.g. due to insufficient disk space.
    #               Possible values are disable or panic. If set to disable,
    #               forward-logging will be disabled on failure. As soon as the
    #               problem is solved it can be manually enabled using dbctl.
    #               If set to panic, the eloqdb6 server will issue a panic and
    #               abort itself. The default value is disable.

    #FwOnFailure = disable

    # FwMaxSize     Limits the maximum size of automatically managed forward-log
    #               files (in megabytes). If not set or set to zero, the file
    #               size limit is 2 gigabytes. The default value is 0 (not set).

    #FwMaxSize = 0

    ### Data base environment
    ...

Once the FwLog configuration entry has been defined (and the server restarted), forward-logging will take place at the specified location. Additionally, it can be temporarily disabled and re-enabled during server runtime. This is discussed in detail in the Forward-Log Management section below.

The FwLog configuration entry specifies where the forward-log is be written. The following options are available:

• A regular file: The database server will create this file and append its transaction records to it.

  Example:
  FwLog = /mnt/disk2/data/db-forward.log

  Note: On current HP-UX and Linux platforms, the maximum size of a single regular file is limited to 2GB. Since a forward-log is supposed to hold all modifications of a particular database environment which occur between two (on-line or off-line) backups, a busy database server with many users attached may cause the forward-log to grow beyond this limit.

• Regular files with automatic management: The database server will automatically create a new file whenever necessary, for example on every server start, during on-line backup or whenever the current file's size would otherwise grow beyond the configured maximum size (which is 2GB by default).

  To enable this mode, the file name must contain the %N token. This token will be replaced by the volume generation count followed by a sequence number.

  Example:
  FwLog = /mnt/disk2/data/db-forward-%N.log

  Note: The %N token must be specified in the last path component (the file name itself). It is not evaluated if used in a directory name.

• A (tape) device file can be specified as well. Depending on the particular device's capacity, this could overcome any disk space limitations even on a busy database server with many users attached.

  Example:
  FwLog = /dev/rmt/c1t0d0BEST

  Note: Using a slow device will probably degrade the overall performance of the database server. Instead, it is preferable to create the forward-log file(s) on disk and later copy them to tape.

• If the file name starts with a pipe character (|) it is considered a command which is passed to the shell. This command is connected to the database server by means of a pipe where the forward-log is written to or read from.

  This could be useful if, for example, a single forward-log file is needed. To overcome the 2GB file size limit in such a scenario, the forward-log could be piped into the gzip command which would compress its contents on the fly.

  Example:
  FwLog = |gzip -c >/mnt/disk2/data/db-forward.log.gz

  This would cause the forward-log to be compressed on the fly into /mnt/disk2/data/db-forward.log.gz with the gzip -c command.

  Note: Since it is likely that a different command would be needed during recovery (where the forward-log's contents must be read from a pipe), an additional FwRecovery configuration entry can be specified.

  Example:
  FwRecovery = |gzip -dc /mnt/disk2/data/db-forward.log.gz

  This would cause the compressed /mnt/disk2/data/db-forward.log.gz to be decompressed with the gzip -dc command.

Choosing an appropriate forward-log location is vital. It should provide enough capacity so that all modifications of a particular database environment occurring between two (on-line or off-line) backups can safely be stored.

In addition, it should be separated physically from the database environment (the volumes). We recommend using a separate disk or a fast tape device. This ensures that the forward-log would still be usable even if one of the disks where the volumes are located suffers a hardware failure.

Forward-log management

• A new forward-log is created
  • whenever the server starts
  • during each on-line backup
  • when forward-logging is re-enabled after it was temporarily disabled
  • if a regular forward-log file with automatic management is about to grow beyond the configured maximum size (which is 2GB by default)

• If a regular forward-log file with automatic management has been configured, each new file will be created with the %N token replaced by the volume generation count followed by a sequence number.

• If an unmanaged regular file, a device file or a pipe has been configured, the new forward-log will be appended to the existing one.

For administrative purposes, forward-logging can be temporarily disabled and re-enabled during server runtime. You could, for example, disable it during a dbimport or dbrestore which normally would cause the forward-log to grow very quickly.

This is controlled by means of the dbctl utility:

• dbctl -u dba forwardlog enable
  This enables forward-logging if it has been previously disabled.

  Please note: After re-enabling, the forward-log is not immediately created. A forward-log requires an on-line or off-line backup as a starting point. Therefore, creation of the forward-log is delayed until the next on-line backup or server restart.

• dbctl -u dba forwardlog disable
  This disables forward-logging if it has been previously enabled.

• dbctl -u dba forwardlog restart
  This forces a new forward-log to be created if forward-logging has been previously enabled (similar to disable followed by enable but performed in a single operation without requiring a new on-line or off-line backup).

• dbctl -u dba forwardlog status
  This displays the current forward-logging status and the name of the current forward-log file/device/pipe if any.

The dbrecover utility

1. All data volumes should be restored from the last (on-line or off-line) backup. Should the associated server configuration file (usually eloqdb6.cfg) be damaged, it should be restored from this backup as well.

2. Next, the dbvolextend utility should be used to re-create the missing log volume(s). They are listed in the [Volumes] section of the associated server configuration file (usually eloqdb6.cfg).

   For example:

   dbvolextend -t log /db/log/log01.vol
   dbvolextend -t log /db/log/log02.vol

3. Finally, the dbrecover utility should be used to apply all modifications which are reported in the forward-log to the database environment. See below for details.

4. Afterward, the server can be restarted and should operate normally. The state of the database environment should be current up to the very moment when the problem occurred.

The dbrecover utility's usage is:

 dbrecover [options]
 options:
  -t tmpdir  - directory to be used for temporary files
  -v         - verbose
  -d flags   - debug flags
  -c cfg     - configuration file name

During recovery, incomplete user transactions are stored in temporary files until they have been completed. Because of this, an appropriate amount of temporary disk space is needed. By default, these temporary files are created in the current directory from where the dbrecover utility has been invoked. This can be overridden with the -t command line option.

Whenever the recovery process fails (eg. due to insufficient temporary disk space) the entire process must be repeated, starting with step 1 (restoring the data volumes from the last backup). This is necessary because a partial recovery from a forward-log would have modified the data volumes and the required starting point would no longer be present.