EQ Eloquence B.07.10 Release Notes - Database
Overview / Bulk Import

Bulk Import

The database server was enhanced to support bulk import of records. Bulk import uses a binary export file and makes use of existing chain information. Bulk loading also partially bypasses transaction logging. This has the potential of substantially improving data migration performance and retains current path order for all paths.

The TurboIMAGE data migration utility was enhanced for B.07.10 to support creating binary export files that may be used with the bulk import function. Please refer to the MPE data migration utility notes for further information.

Bulk import is currently not a seperate utility program but built into the server process and is used with the dbctl utility.

 dbctl bimport [options] database set bexp_file
The database argument specifies the database name, the set argument specifies the data set name or number and bexp_file spcifies the import file for this set (complete path is required).

Bulk import first erases the specified data set and then processes the specified import file. As the binary import files are directly accessed by the eloqdb6 server process the files must be readable by the user that is configured for the eloqdb6 and must reside on the same system.

The following options are supported with bimport:

/v - Display progress information. Specifying the option /v twice (/v /v) causes the import progress to be output to stderr for every 10,000 records.
/a - In case more than one bexp file is needed to import a data set this flag is specified with any subsequent bexp file.
/n - This option must be specified if additional bexp files exist for this data set.

As the current MPE dbexport only creates a single binary export file for each data set, the options /a and /n should not be used.

Note: bimport needs write access to the database content. The dba user by default has no access to the database content.


Example script

The example script below demonstrates the use of bimport to import a single database.

# bimport.sh - Eloquence example script to bimport a database
#
# This erases the target database and then attempts to bimport 
# any files that follow the pattern DBNAME.999.exb
#
# As the bexp files are directly accessed by the eloqdb6 server
# process the files must be readable by the user that is 
# configured for the eloqdb6 and must reside on the same system.
# 
# usage:
# bimport.sh database
#
# Note: currently the database name may not specify an Eloquence 
# instance. Please use the EQ_DBSERVER environment variable 
# instead to specify the server instance.
#
# Note: bimport needs write access to the database. Don't use
# the dba user with the dbctl bimport as it (by default) has no
# access to the database content.
#
# Note: Specifying the /v (or /v /v) option with bimport 
# provides a progress report that is printed to stderr.

if [ $# -lt 1 ]; then
   echo "usage: $0 database"
   exit
fi

db=$(echo $1|tr [a-z] [A-Z])
pwd=$(pwd)
#opt="/v /v"

# erase database
dberase $db

for bexp_file in $db.*.exb
do
 set=$(echo $bexp_file|cut -d. -f2)
 if [ -z "$opt" ]; then
  echo "$db.$set ... \c"
  dbctl bimport $db $set $pwd/$bexp_file
 else
  echo "$db.$set ..."
  dbctl bimport $opt $db $set $pwd/$bexp_file
 fi
done


How bulk import works

The bulk import uses a binary file format that contains both the original record number and the original chain pointers in addition to any data. bimport makes use of the chain information to improve the import process.

For master sets, bimport uses the chain pointers to any related detail sets.

For detail sets the original record numbers are retained in addition to the chain links. As Eloquence could make use of the existing chain pointers, substantial less effort is required to build the chain links.

The format of the bulk import file is not yet published but may be requested by contacting support@marxmeier.com.

Performance benefits:

Notes:



© 2005 Marxmeier Software AG - 2005-12-28