EQ Eloquence B.07.10 Release Notes - MPE database migration utilities
Overview / MPE database migration utilities

TurboIMAGE database migration utilities

The Eloquence MPE database migration utilities provide an easy and straight forward approach to move your TurboIMAGE databases to Eloquence.

Moving your database to Eloquence involves the following steps:

  1. Install the DBEXPORT utility on HP3000
  2. Export the database
  3. Transfer the schema file and the export files to the target system
  4. Create and import the database on Eloquence

Limitations:


Contents


Install the DBEXPORT utility on the HP3000

Eloquence comes with utilities to load and unload the database. Eloquence supports both using text files to hold the database content or using binary files.

An equivalent utility is also available for the HP3000 and may be used to unload your database. It can be downloaded from the Eloquence ftp server at the URL below:

The following files are available:

Note: When using ftp please make sure you download the archive files (.tar.Z) in binary format.

When using ftp to download an archive file to an MPE system you need to provide an alternative file name which is MPE compliant or prepend the file name on MPE with a ./ (dot slash) to make sure it is accepted as a POSIX file name, such as shown below:

ftp> bin
ftp> cd /eloq/B0710/mpe/migration
ftp> get hp3kbin.tar.Z ./hp3kbin.tar.Z

To install the tar archive on your system, please enter the POSIX shell and follow the procedure below:

$ tar -xvzopf hp3kbin.tar.Z
This restores the archive content in the current directory.

As an alternative, you may extract the archive file from the MPE shell with a command like below:

:/bin/tar "-xvzopf hp3kbin.tar.Z"
To use binary export (recommended) the DBEXPORT utility needs to run in privileged mode and must be installed in a privileged group. Also, the program capabilities must be changed with linkedit as below:
:linkedit "altprog DBEXPORT; cap=ia,ba,pm"
This adds the PM capability to DBEXPORT.

Compiling from source code

The source code to the Eloquence data migration utilities is available as an option. To build your own binary from source code you need a HP C compiler.

Download the source archive to MPE, unpack it with tar and run make -f Makefile.MPE (in a POSIX shell). As a result the file DBEXPORT and DBINFO are built.

Compiling was only tested with the HP C compiler on MPE 6.5 or newer. As the MPE 6.0 HP C compiler does not support the long long type you need to modify the Makefile to set the HAS_LONG_LONG=0 to get build on MPE 6.0.


The DBEXPORT utility

DBEXPORT is used to export the database contents to one or multiple text or binary files. To export to binary files, DBEXPORT must be run in PRIV mode.

When started without command line arguments (or with the -help option) the command line options are printed.

usage: DBEXPORT [options] database [set[.item] ...]
options:
 -help    - show usage (this list)
 -version - show version information
 -u user  - set user name
 -p pswd  - set password
 -o path  - set output directory (not single file)
 -v       - verbose output
 -c       - chained export
 -r       - create restructure information
 -s file  - output into single file, '-' = stdout
 -x       - exclude specified data sets
 -B       - binary export mode (needs PM)
 -S       - chained export by sort item
The MPE DBEXPORT utility is similar to but different from the Eloquence dbexport utility. While the options (if available) are the same not all options are available for both versions.

In addition, the database to be exported must be specified and optionally a list of data sets may be present (either data set name or number). If a list of data sets is provided only the specified data sets are processed. For a detail data set, a search item may also be specified to define the detail set export order.

The typically used options are:

Any data set name or number specified after the database name indicates that only the specified data sets should be exported. When the -x option is present, any specified data sets are not exported.

For example:

$ DBEXPORT -p secret -v sample
Export the database sample in text mode. The database is opened with the password "secret".
$ DBEXPORT -p secret -B -v sample
Export the database sample in binary mode.
$ DBEXPORT -p secret -v -s sample.exp sample
Export the database sample in text mode, the results are written to a single text file.
$ DBEXPORT -p secret sample customers parts
$ DBEXPORT -p secret sample 1 2
Export the data sets customers and parts in text mode.
$ DBEXPORT -p secret -x sample 1 2
Export all but the data sets customers and parts in text mode.

All the examples above used the POSIX shell. To use dbexport from the MPE shell you need to enclose the arguments in quotes

: DBEXPORT "-p secret -v sample"

See also:
http://eloquence.marxmeier.com/doc/html/manuals/dbms/dbms-57.html
http://eloquence.marxmeier.com/support/misc/export.html

How to use the text mode export

The text mode dbexport converts the database content to text. Unless the -s option is specified a separate text file is created for each data set (named DBNAME.999.exp).

Any empty data sets or automatic masters are skipped and no export file is created.

If the -c option is present, any detail records are exported in an order to retain the primary path.

Please note that DBEXPORT replaces any invalid numeric values by a zero value during export (eg. corrupted packed/zoned numbers).

See below for a workaround to overcome the 2 GB file size limitation on MPE for bytestream files.

How to use the binary export mode

The binary export mode is used when the -B (upper case B) command line option is specified.

The binary export mode creates a separate file for each dataset (unless it has zero entries). DBEXPORT uses the naming convention DBNAME.999.exb for each binary export file. If the size of the binary export file for a data set would exceed 1 GB instead multiple partial files are created named DBNAME.999.b## where ## is the sequential number of this file.

The binary export file includes a header to identify the data in the file and for each record the original record number and the chain pointers (besides the sysnonym chain information for a master set). This information may then be used by the Eloquence bimport to efficiently import the database into Eloquence, retaining the chain order and record number for any detail set.

See below for a workaround to overcome the 2 GB file size limitation on MPE for bytestream files.


The DBINFO utility

DBINFO lists the data sets for the specified database including data set name, type, number of entries and capacity. When started without command line arguments (or with the -help option) the command line options are printed.
usage: DBINFO [options] database [set ...]
options:
 -help    - show usage (this list)
 -version - show version information
 -u user  - set user name
 -p pswd  - set password
database is the name of the database.

Optionally, one or more data set names or numbers can be specified after the database name (separated by spaces). In this case only information on the specified data sets is returned. By default all accessible data sets are displayed.

:dbinfo testdb
Processing database : testdb

SET NAME              RECLEN CAPACITY ENTRIES
---------------- -- - ------ -------- --------
TEST             01 D    111      100        1

To use DBINFO from the MPE shell you need to enclose the arguments in quotes if there is more than one argument.


Exporting the database

It is typically preferable to export the database in binary mode as this usually results in better import performance and also retains the order for all paths.

Exporting the database in text mode is easier to use, the export files are typically smaller and source and target databases may differ to some extent.

Exporting the database in text mode:

When running the DBEXPORT utility from the POSIX shell the arguments are separated by a space.

$ DBEXPORT -p SECRET -v SAMPLE
When running from the MPE shell (CI) you need to enclose the arguments in quotes.
: DBEXPORT "-p SECRET -v SAMPLE"
This creates output files SAMPLE.001.exp ... (one for each non-empty data set, except for automatic masters).

Exporting the database in binary mode:

When running the DBEXPORT utility from the POSIX shell the arguments are separated by a space.

$ DBEXPORT -B -p SECRET -v SAMPLE
When running from the MPE shell (CI) you need to enclose the arguments in quotes.
: DBEXPORT "-B -p SECRET -v SAMPLE"
This creates output files SAMPLE.001.exb ... (one for each non-empty data set). Please note that automatic master sets are exported as well.


Transferring the database to the target system

ftp may be used to transfer the schema file and the export files to the target system.

To transfer the schema file, it must either be converted to bytestream format (/bin/tobyte [-a] -t) and then transferred in binary mode or you may transfer it in text mode. The ftp utility will then convert it to bytestream mode during the transfer.

Any export files (either text or binary export files) must be transferred in binary mode.


Creating the database on Eloquence

On Eloquence, first run the schema utility and the dbcreate utility.
dbschema schemafile
dbcreate database
Where schemafile is the file name of the schema file and database is the name of the database.

Note: dbschema is equivalent to schema -T. The -T option to schema indicates that the schema file follows the TurboIMAGE conventions. Eloquence schema files are similar to TurboIMAGE schema files but use different options and item sizes are specified in bytes instead of item type specific units.

Importing the database on Eloquence (text mode export)

When using text mode export, the Eloquence dbimport utility is used to populate the database from the export files.
$ dbimport [options] database {set [...]]
Typical options: A list of data set names or numbers can be can be specified after the database name. In this case only the specified data sets are processed. By default all accessible data sets are imported.

For example:

$ dbimport -v sampledb
$ dbimport -v -s sampledb.exp localhost:8208/sampledb

On the Windows and Linux platform you should use the -z roman8 command line option of dbimport to make sure any national charaters ("Umlaute") are converted properly.

Please note: Unless you import a trivial volume of data you may want to consider to set SyncMode = 0 in your eloqdb6.cfg configuration file during the dbimport and disable any forward logging. dbctl may be used for both:

dbctl -u dba syncmode off
dbctl -u dba forwardlog disable

How to use bimport on Eloquence (from binary mode export)

The new bimport function allows to efficiently import data from TurboIMAGE while retaining the chain information. For a detail set the record number is also retained.

While undocumented, the bimport function is included for testing purposes in the most recent Eloquence B.07.00 eloqdb6 patches for the HP-UX platform.

Currently, bimport is not a conventional utility program but builtin the server process and run with the dbctl utility.

dbctl bimport [options] database set bexp_file
This first erases the specified data set and then processes the specified import file. As the binary import files are directly accessed by the eloqdb6 server process the files must be readable by the user that is configured for the eloqdb6 server and must reside on the same system.

The following options are supported with bimport:

Note: As the current MPE dbexport only creates a single binary export file for each data set, the options /a and /n should not be used (these options require a separate header for each bexp file). See below for a workaround to overcome the 2 GB file size limitation on MPE for bytestream files.

For example:

$ dbctl bimport /v /v sample 1 $PWD/SAMPLE.001.exb
This imports the data set 1 of the database sample from the file SAMPLE.001.exb (an absolute path to the export file is specified).

An example script (bimport.sh) is shown subsequently in this document that may be used to automatically import a single database.

Completing the database migration

In addition to moving the database it might be necessary to update the database settings (eg. CIUPDATE flag) according to the original setting on MPE. This is done with dbutil.

In case a TPI package was used on MPE any indexes must be re-created. Eloquence includes limited support for TPI indexes and may be used to handle indexing. In this case the index is either created with dbutil after migration or added to the schema file before creating the database.

You may also want to consider changing the database security settings as by default the Eloquence schema creates databases that are accessible with the default "public" user.


Potential Issues

A number of subtle differences between Eloquence and TurboIMAGE may affect the database migration. The potential issues below were encountered in migration projects:

Item type U (upper case strings)

The item type U (upper case strings) is enforced with Eloquence. With TurboIMAGE the item type U is only used as an application hint. Eloquence will uppercase the item content when the record is modifed.

This immediately affects the text mode import (dbimport). While any item type U items are unmodified when using binary import the item contents will be changed subsequently when modified by the application.

It is recommended to change the item type U to X if lower case data are required in a string.

Binary values in strings

Binary values placed in a string item (type X or U) may result a subsequent problems. When the client and server use different character sets or byte orders Eloquence internally converts any item values on the fly as required. As Eloquence does not know about them any binary values placed in a string item will also be subject to a conversion, likely resulting in unexpected values.

Binary values in string items work as long as the new target system uses the same character set and byte order as MPE (eg. HP-UX on PA-RISC or Itanium). It will break when converting to a system using a different byte order or character set (eg. Windows on x86).

The recommended solution is to consider correcting the application to use separate items for binary values. As an option changing the item to item type B (binary) indicates to Eloquence that no conversion is allowed an item.

Value aliases (packed and zoned items used as search item)

Packed and zoned items could use a different encoded value to represent the same numerical value. For example, there could be a positive zero, a negative zero and an unsigned zero encoded value that all represent the same numeric value (zero) but use different binary representation.

TurboIMAGE consideres different binary values to be different, independendly if they represent the same numeric value. In a manual master multiple records could be present that have the same numeric value but a different binary encoded value as the search item.

Eloquence uses the numerical value for any comparison (not the binary encoded value) and thus considers records with the same numeric value for a search item a duplicate in a manual master. This might result in a problem when moving existing databases from TurboIMAGE as the database import fails with a status 43 due to a duplicate entry for a manual master while this was no problem on TurboIMAGE.

The suggested solution is to correct your data on MPE before moving it to Eloquence.

When using binary export it is recommended to check the database consistency (for example, using ADAGER or DBGENERL) before migration. With binary export mode any database inconsistencies will also affect Eloquence any may result in a problem during bimport or broken chains after the migration.


DBEXPORT B.07.10 enhancements

New -S option
The new option -S (upper case S) may be used to specify chained export mode and that a sort item has preference over the primary path. The option -S implies option -c (chained export mode). Please note that the -S option only affects the default path to be used for (text mode) exporting a detail set. A separate path may still be specified.

Specifying the search item in chained export mode
When using chained export mode (option -c or -S present) the search item for a detail set may be specified to override the default of using the primary path. The (detail) search item number may be specified with the dataset, seperated by a dot.

The examples below all specifiy to export the data set PO-RECEIPTS (set number 39) of the data base IOSDB, using chained export and the search item PART-NUMBER (item number 159).

DBEXPORT -c IOSDB.QEDDB 39
DBEXPORT -c IOSDB.QEDDB 39.PART-NUMBER
DBEXPORT -c IOSDB.QEDDB 39.129
DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.159
DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.PART-NUMBER

Binary export mode was enhanced to split bexp files
When using binary export mode (-B option present), a new export file is created for each GB of data. In this case no .exb file but multiple partial files are created that need to be combined to run bimport.

This allows binary exporting data sets up to a size of 100 GB with little extra effort by concatenating the files. See below for an example.

Please note that this is different from the bexport option to use multiple input files files (as specified with the options /v and /a). For bimport to accept multiple files for a data set each file must provide a valid BEXP header which is not supported by DBEXPORT at this time.


DBEXPORT workarounds

The following workarounds may be useful to overcome some commonly encountered limitations with the dbexport database migration tool.

Overcome the 2 GB filesize limitation with text mode export

To overcome the (MPE) 2 GB filesize limitation with text mode export files you may pipe the output to split.

DBEXPORT -s - database | split -b 1000m - exp.
This will split the export file into 1000 MB chunks named exp.aa, exp.ab ... On the target system you may combine the files with a command like below:
cat exp.* > database.exp
You may also pass the result to dbimport though stdin:
cat exp.* | dbimport -s - database

Performance problem during import with sorted chains (text mode export)

Sorted chains may cause a performance problem in text mode. It might help to export the database in sorted order. The -S options specifies to use chained export mode giving preference to any sort items (instead of primary path).

dbexport -S database
This attempts to retain the chain order for the first sort item which to reduce the overhead of the chain rebuild on the target platform.

Using the -x option (exclude) you may also export the database in two steps. In the first step, all "normal" data sets are processed, exclusing any sets that would benefit from different options. In the second step any data set is exported specifying specific options.

dbexport -c -x database 5
This exports the entire database except set #5
dbexport -c database 5.CUSTNO
This uses the chain associated with the item CUSTNO to export set #5.

Overcome the 2 GB filesize limitation with binary mode export

To overcome the (MPE) 2 GB filesize limitation with binary mode export files the dbexport utility automatically creates mupliple files when the export file for a data set would exceed 2 GB.

Binary export files are split into multiple segments for each GB of data. In this case no .exb file but multiple partial files are created that need to be combined on the target platform to run bimport.

For example, exporting set #5 could create the following partial binary export files: SAMPLE.005.b00 SAMPLE.005.b01 SAMPLE.005.b02 ...

The cat command may be used like below to combine these partial binary export files into a complete export file.

   cat SAMPLE.005.b* > SAMPLE.005.exb
While bimport has the option to process partial files (each with a complete bexp header) this approach is currently not supported by DBEXPORT.

Example shell script to bimport a database. This also combines any partial bexport files before importing.

# bimport.sh - Eloquence example script to bimport a database
#
# This erases the target database and then attempts to bimport 
# any files that follow the pattern DBNAME.999.exb
#
# As the bexp files are directly accessed by the eloqdb6 server
# process the files must be readable by the user that is 
# configured for the eloqdb6 and must reside on the same system.
#
# usage:
# bimport.sh database
#
# Note: currently the database name may not specify an Eloquence 
# instance. Please use EQ_DBSERVER as a workaround.
#
# Note: bimport needs write access to the database. Don't 
# specify the dba user with the dbctl bimport.
#
# Note: Specifying the /v (or /v /v) option with bimport 
# provides a progress report that is printed to stderr.

if [ $# -lt 1 ]; then
   echo "usage: $0 database"
   exit
fi

db=$(echo $1|tr [a-z] [A-Z])
pwd=$(pwd)
#opt="/v /v"

# erase database
dberase $db

# concatenate any partial files
for b00_file in $db.*.b00
do
 set=$(echo $b00_file|cut -d. -f2)
 if [ ! -f "$db.$set.bexp" ]; then
  echo "creating $db.$set.bexp from fragments ..."
  cat $db.$set.b[0-9][0-9] > $db.$set.bexp
 fi
done
exit

for bexp_file in $db.*.exb
do
 set=$(echo $bexp_file|cut -d. -f2)
 if [ -z "$opt" ]; then
  echo "$db.$set ... \c"
  dbctl bimport $db $set $pwd/$bexp_file
 else
  echo "$db.$set ..."
  dbctl bimport $opt $db $set $pwd/$bexp_file
 fi
done



© 2006 Marxmeier Software AG - 2006-01-12