Forward-logging adds an incremental backup mechanism to the Eloquence database which allows recovery of database changes since the last backup in case of a disk or operating system failure. If necessary, a backup is restored and the changes since the last backup are applied with the dbrecover utility.

Contents

• Recovering from abnormal server termination
• Forward-log configuration
• Forward-log management
• The dbrecover utility

Recovering from abnormal server termination

The situation of a database environment not shutdown cleanly can be caused by various reasons:

1. There is no more space left in either the volume(s) or the file system(s) where the volumes are located. In such a case, the server has no chance to shutdown the database environment cleanly and therefore just terminates immediately.

   Once the underlying problem is solved (eg. by creating additional data volumes or by providing additional disk space), a subsequent startup-recovery will always be successful.

2. The system was inadvertently switched off or went down due to a power interruption (eg. if no UPS device is attached) or an operating system failure (eg. a kernel panic due to insufficient or corrupted swap space).

   In such a case, the integrity of the log volume's contents determines whether a startup-recovery will be successful or not. If the server was operated synchronously (with the SyncMode configuration activated) damage in the log volume is less likely than with asynchronous operation.

3. The database environment is corrupted by either a disk failure or an operating system problem (eg. overwriting parts of the data volume with invalid data). A failure like this usually causes the server to abort immediately (internal server panic).

   Typically, a subsequent startup-recovery will fail because the data and/or log volumes are either not accessible anymore or their contents have been corrupted.

A startup-recovery will be successful if all data and log volumes are present and accessible and if their contents have not been corrupted. In particular, if the operating system had a chance to write all pending modifications to the volumes after the server terminated abnormally, the subsequent startup-recovery will succeed.

If any of these conditions is not met, it is possible that a startup-recovery will fail. This normally requires restoration of all data volumes from the last (on-line or off-line) backup. The database server will probably start successfully afterwards.

However, any database modifications since the last backup would be lost unless they were contained in a forward-log. Combined with the last backup, the recovery from a forward-log would restore the contents of the entire database environment up to the very moment when the problem occurred.

In this way, a forward-log provides an additional safety mechanism in case the startup-recovery cannot be performed successfully.

Configuring Forward-logging

To enable forward-logging, the FwLog configuration entry must be defined:

FwLog

Configures the file, device or pipe to be used for forward-logging. Using the %N token in the file name activates automatic file management (see below, not possible for devices or pipes).
By default, forward-logging is inactive.

The examples below configure an automatically managed file and a pipe that compresses the data on-the-fly:

FwLog = /mnt/disk2/data/db-forward-%N.log
FwLog = |gzip -c >/mnt/disk2/data/db-forward.log.gz

Note: Using disk files with automatic file management via the %N token is recommended instead of using a pipe like in the above gzip-example (where a simple restart of the database server would destroy the previous forward log data).

Optionally, the configuration entries described below may be used to specify additional forward-logging and recovery properties:

FwRecovery

Configures the file, device or pipe to be used during forward recovery. If not set, the FwLog setting is used by default.

The example below configures a pipe that uncompresses the data on-the-fly:

FwRecovery = |gzip -dc /mnt/disk2/data/db-forward.log.gz

FwOnFailure

Configures the action to be taken in case the forward-log cannot be written, e.g. due to insufficient disk space. Possible values are disable or panic.
If set to disable, forward-logging will be disabled on failure. As soon as the problem is solved it can be manually enabled using dbctl. If set to panic, the eloqdb server will issue a panic and abort itself.
The default value is disable.

FwMaxSize

Limits the maximum size of automatically managed forward-log files (in megabytes). If not set or set to zero, the file size limit is 2 gigabytes.
The default value is 0 (not set).

EnableAudit

If set to a nonzero value, audit information is written to the forward-log in addition to recovery data.
The default value is 0 (disabled).
For details, please refer to the documentation of the database auditing functionality.

AuditOnly

If set to a nonzero value, audit information is written to the forward-log. However, no recovery data is written.
The default value is 0 (disabled).
For details, please refer to the documentation of the database auditing functionality.

GroupReadAccess

If set to a nonzero value, forward-log files are created with a permission that allows read access to the configured group (specified by [Server] GID config item). If set to zero, forward log files are created with a permission that restricts access to the configured owner (specified by [Server] UID config item).
The default value is 0 to permit owner access only.

Once the FwLog configuration entry has been defined (and the server restarted), forward-logging will take place at the specified location. Additionally, it can be temporarily disabled and re-enabled during server runtime. This is discussed in detail in the Forward-Log Management section below.

The FwLog configuration entry specifies where the forward-log is be written. The following options are available:

• A regular file: The database server will create this file and append its transaction records to it.

  For example:

  FwLog = /mnt/disk2/data/db-forward.log
  

  Note: The maximum size of a single forward-log file is limited to 2GB. Since a forward-log is supposed to hold all modifications of a particular database environment which occur between two (on-line or off-line) backups, a busy database server with many users attached may cause the forward-log to grow beyond this limit.

• Regular files with automatic management: The database server will automatically create a new file whenever necessary, for example on every server start, during on-line backup or whenever the current file's size would otherwise grow beyond the configured maximum size (which is 2GB by default).

  Automatic file management is the recommended mode for using FwLog.

  To enable this mode, the file name must contain the %N token. This token will be replaced by the volume generation count followed by a sequence number.

  For example:

  FwLog = /mnt/disk2/data/db-forward-%N.log
  

  Note: The %N token must be specified in the last path component (the file name itself). It is not evaluated if used in a directory name.

• A (tape) device file can be specified as well. Depending on the particular device's capacity, this could overcome any disk space limitations even on a busy database server with many users attached.

  For example:

  FwLog = /dev/rmt/c1t0d0BEST
  

  Note: Using a slow device will probably degrade the overall performance of the database server. Instead, it is preferable to create the forward-log file(s) on disk and later copy them to tape.

• If the file name starts with a pipe character (|) it is considered a command which is passed to the shell. This command is connected to the database server by means of a pipe where the forward-log is written to or read from.

  This could be useful if, for example, a single forward-log file is needed. To overcome the 2GB file size limit in such a scenario, the forward-log could be piped into the gzip command which would compress its contents on the fly.

  For example:

  FwLog = |gzip -c >/mnt/disk2/data/db-forward.log.gz
  

  This would cause the forward-log to be compressed on the fly into /mnt/disk2/data/db-forward.log.gz with the gzip -c command.

  Note: Since it is likely that a different command would be needed during recovery (where the forward-log's contents must be read from a pipe), an additional FwRecovery configuration entry may be specified.

  For example:

  FwRecovery = |gzip -dc /mnt/disk2/data/db-forward.log.gz
  

  This would cause the compressed /mnt/disk2/data/db-forward.log.gz to be decompressed with the gzip -dc command.

Choosing an appropriate forward-log location is vital. It should provide enough capacity so that all modifications of a particular database environment occurring between two (on-line or off-line) backups can safely be stored.

In addition, it should be separated physically from the database environment (the volumes). We recommend using a separate disk or a fast tape device. This ensures that the forward-log would still be usable even if one of the disks where the volumes are located suffers a hardware failure.

Forward-log management

• A new forward-log is created
  • whenever the server starts
  • during each on-line backup
  • when forward-logging is re-enabled after it was temporarily disabled
  • if a regular forward-log file with automatic management is about to grow beyond the configured maximum size (which is 2GB by default)

• If a regular forward-log file with automatic management has been configured, each new file will be created with the %N token replaced by the volume generation count followed by a sequence number.

• If an unmanaged regular file, a device file or a pipe has been configured, the new forward-log will be appended to the existing one.

For administrative purposes, forward-logging can be temporarily disabled and re-enabled during server runtime. You could, for example, disable it during a dbimport or dbrestore which normally would cause the forward-log to grow very quickly.

This is controlled by means of the dbctl utility:

• dbctl -u dba forwardlog enable
  This enables forward-logging if it has been previously disabled.

  Please note:

  • After re-enabling, the forward-log is not immediately created. A forward-log requires an on-line or off-line backup as a starting point. Therefore, creation of the forward-log is delayed until the next on-line backup.

  • A restart of the database server implicitly enables forward-logging and immediately creates a new forward-log.

• dbctl -u dba forwardlog disable
  This disables forward-logging if it has been previously enabled.

• dbctl -u dba forwardlog restart
  This forces a new forward-log to be created if forward-logging has been previously enabled (similar to disable followed by enable but performed in a single operation without requiring a new on-line or off-line backup).
  forwardlog restart and forwardlog changelog are synonyms.

• dbctl -u dba forwardlog status
  This displays the current forward-logging status and the name of the current forward-log file/device/pipe if any.

The dbrecover utility

1. All data volumes should be restored from the last (on-line or off-line) backup. Should the associated server configuration file (usually eloqdb.cfg) be damaged, it should be restored from this backup as well.

2. Next, the dbvolextend utility should be used to re-create the missing log volume(s).

   The dbvolextend utility supports the -R command line option to conveniently recreate all missing log volumes in a single run.

   For example:

   dbvolextend -v -R
   

3. Finally, the dbrecover utility should be used to apply all modifications which are reported in the forward-log to the database environment. See below for details.

4. Afterward, the server can be restarted and should operate normally. The state of the database environment should be current up to the very moment when the problem occurred.

The dbrecover utility's usage is:

dbrecover [options]
options:
 -t tmpdir    - directory to be used for temporary files
 -v           - verbose
 -d flags     - debug flags
 -c cfg       - configuration file name
 -b size      - Buffer cache size (MB)
 -T timestamp - recover until point in time (incl.)

timestamp formats:
 YYYY-MM-DD HH:MM:SS
 MM/DD/YYYY HH:MM:SS
 DD.MM.YYYY HH:MM:SS

The -b command line option may be used to specify the buffer cache size (in MB). If not set, it defaults to 5 MB. Older Eloquence versions used the BufferCache setting from the database server configuration file.

The -T option may be used to specify a point-in-time beyond dbrecover stops applying changes. Once this point is passed, dbrecover will exit. Specifying the time is optional (defaults to 00:00:00). Date and time may be separated with space or other characters (shell quoting might be required).

During recovery, incomplete user transactions are stored in temporary files until they have been completed. Because of this, an appropriate amount of temporary disk space is needed. By default, these temporary files are created in the current directory from where the dbrecover utility has been invoked. This can be overridden with the -t command line option.

dbrecover notes

With Eloquence B.08.00 (and recent B.07.10 patches) the dbrecover utility was enhanced to support incremental recovery. Whenever the recovery process fails (eg. due to insufficient temporary disk space), the dbrecover utility may be restarted and should be able to continue from the previous point. The original dbrecover utility required starting from the beginning, i.e. by restoring previous backup first.

Please note: When using incremental recovery the server process MUST NOT be started between dbrecover runs as this would change the volume files and/or generation count and make subsequent runs of dbrecover impossible or inconsistent (data corruption).

Also note that volume file operations outside transaction context, like dbvolextend or write mode of dbrepack, dbfsck or dbcfix make subsequent use of dbrecover impossible until a new baseline backup of the volume files is created.

After using dbrecover -T for performing a point-in-time recovery, you should move any unprocessed forward log files "out of the way" before starting the eloqdb server again. Otherwise the new server instance might fail to create new forward logs with conflicting generation numbers and either disable forward logging or panic (depending on the FwOnFailure setting in eloqdb.cfg).

Please note that it is not supported to use the dbrecover utility on a SLAVE server to perform a point-in-time recovery, then later resume replication. Doing so would very likely result in slave server data corruption and/or would cause the slave server to crash. For recovering a slave server to a specified point in time, you may restore a backup, start the eloqdb in slave role and then use dbrepl utility with option -T to apply transactions up to a given point in time. (these dbrepl -T runs may be done on the master or slave system, depending on where the relevant forward log files are available).