Eloquence B.07.10 Release Notes - MPE database migration utilities
Moving your database to Eloquence involves the following steps:
Limitations:
An equivalent utility is also available for the HP3000 and may be used to unload your database. It can be downloaded from the Eloquence ftp server at the URL below:
The following files are available:
Note: When using ftp please make sure you download the archive files (.tar.Z) in binary format.
When using ftp to download an archive file to an MPE system you need to provide an alternative file name which is MPE compliant or prepend the file name on MPE with a ./ (dot slash) to make sure it is accepted as a POSIX file name, such as shown below:
ftp> bin ftp> cd /eloq/B0710/mpe/migration ftp> get hp3kbin.tar.Z ./hp3kbin.tar.Z
To install the tar archive on your system, please enter the POSIX shell and follow the procedure below:
$ tar -xvzopf hp3kbin.tar.ZThis restores the archive content in the current directory.
As an alternative, you may extract the archive file from the MPE shell with a command like below:
:/bin/tar "-xvzopf hp3kbin.tar.Z"To use binary export (recommended) the DBEXPORT utility needs to run in privileged mode and must be installed in a privileged group. Also, the program capabilities must be changed with linkedit as below:
:linkedit "altprog DBEXPORT; cap=ia,ba,pm"This adds the PM capability to DBEXPORT.
Download the source archive to MPE, unpack it with tar and run make -f Makefile.MPE (in a POSIX shell). As a result the file DBEXPORT and DBINFO are built.
Compiling was only tested with the HP C compiler on MPE 6.5 or newer. As the MPE 6.0 HP C compiler does not support the long long type you need to modify the Makefile to set the HAS_LONG_LONG=0 to get build on MPE 6.0.
When started without command line arguments (or with the -help option) the command line options are printed.
usage: DBEXPORT [options] database [set[.item] ...] options: -help - show usage (this list) -version - show version information -u user - set user name -p pswd - set password -o path - set output directory (not single file) -v - verbose output -c - chained export -r - create restructure information -s file - output into single file, '-' = stdout -x - exclude specified data sets -B - binary export mode (needs PM) -S - chained export by sort itemThe MPE DBEXPORT utility is similar to but different from the Eloquence dbexport utility. While the options (if available) are the same not all options are available for both versions.
In addition, the database to be exported must be specified and optionally a list of data sets may be present (either data set name or number). If a list of data sets is provided only the specified data sets are processed. For a detail data set, a search item may also be specified to define the detail set export order.
The typically used options are:
For example:
$ DBEXPORT -p secret -v sampleExport the database sample in text mode. The database is opened with the password "secret".
$ DBEXPORT -p secret -B -v sampleExport the database sample in binary mode.
$ DBEXPORT -p secret -v -s sample.exp sampleExport the database sample in text mode, the results are written to a single text file.
$ DBEXPORT -p secret sample customers parts $ DBEXPORT -p secret sample 1 2Export the data sets customers and parts in text mode.
$ DBEXPORT -p secret -x sample 1 2Export all but the data sets customers and parts in text mode.
All the examples above used the POSIX shell. To use dbexport from the MPE shell you need to enclose the arguments in quotes
: DBEXPORT "-p secret -v sample"
See also:
http://eloquence.marxmeier.com/doc/html/manuals/dbms/dbms-57.html
http://eloquence.marxmeier.com/support/misc/export.html
Any empty data sets or automatic masters are skipped and no export file is created.
If the -c option is present, any detail records are exported in an order to retain the primary path.
Please note that DBEXPORT replaces any invalid numeric values by a zero value during export (eg. corrupted packed/zoned numbers).
See below for a workaround to overcome the 2 GB file size limitation on MPE for bytestream files.
The binary export mode creates a separate file for each dataset (unless it has zero entries). DBEXPORT uses the naming convention DBNAME.999.exb for each binary export file. If the size of the binary export file for a data set would exceed 1 GB instead multiple partial files are created named DBNAME.999.b## where ## is the sequential number of this file.
The binary export file includes a header to identify the data in the file and for each record the original record number and the chain pointers (besides the sysnonym chain information for a master set). This information may then be used by the Eloquence bimport to efficiently import the database into Eloquence, retaining the chain order and record number for any detail set.
See below for a workaround to overcome the 2 GB file size limitation on MPE for bytestream files.
usage: DBINFO [options] database [set ...] options: -help - show usage (this list) -version - show version information -u user - set user name -p pswd - set password
Optionally, one or more data set names or numbers can be specified after the database name (separated by spaces). In this case only information on the specified data sets is returned. By default all accessible data sets are displayed.
:dbinfo testdb Processing database : testdb SET NAME RECLEN CAPACITY ENTRIES ---------------- -- - ------ -------- -------- TEST 01 D 111 100 1
To use DBINFO from the MPE shell you need to enclose the arguments in quotes if there is more than one argument.
Exporting the database in text mode is easier to use, the export files are typically smaller and source and target databases may differ to some extent.
Exporting the database in text mode:
When running the DBEXPORT utility from the POSIX shell the arguments are separated by a space.
$ DBEXPORT -p SECRET -v SAMPLEWhen running from the MPE shell (CI) you need to enclose the arguments in quotes.
: DBEXPORT "-p SECRET -v SAMPLE"This creates output files SAMPLE.001.exp ... (one for each non-empty data set, except for automatic masters).
Exporting the database in binary mode:
When running the DBEXPORT utility from the POSIX shell the arguments are separated by a space.
$ DBEXPORT -B -p SECRET -v SAMPLEWhen running from the MPE shell (CI) you need to enclose the arguments in quotes.
: DBEXPORT "-B -p SECRET -v SAMPLE"This creates output files SAMPLE.001.exb ... (one for each non-empty data set). Please note that automatic master sets are exported as well.
To transfer the schema file, it must either be converted to bytestream format (/bin/tobyte [-a] -t) and then transferred in binary mode or you may transfer it in text mode. The ftp utility will then convert it to bytestream mode during the transfer.
Any export files (either text or binary export files) must be transferred in binary mode.
dbschema schemafile dbcreate databaseWhere schemafile is the file name of the schema file and database is the name of the database.
Note: dbschema is equivalent to schema -T. The -T option to schema indicates that the schema file follows the TurboIMAGE conventions. Eloquence schema files are similar to TurboIMAGE schema files but use different options and item sizes are specified in bytes instead of item type specific units.
$ dbimport [options] database {set [...]]
Typical options:
For example:
$ dbimport -v sampledb $ dbimport -v -s sampledb.exp localhost:8208/sampledb
On the Windows and Linux platform you should use the -z roman8 command line option of dbimport to make sure any national charaters ("Umlaute") are converted properly.
Please note: Unless you import a trivial volume of data you may want to consider to set SyncMode = 0 in your eloqdb6.cfg configuration file during the dbimport and disable any forward logging. dbctl may be used for both:
dbctl -u dba syncmode off dbctl -u dba forwardlog disable
While undocumented, the bimport function is included for testing purposes in the most recent Eloquence B.07.00 eloqdb6 patches for the HP-UX platform.
Currently, bimport is not a conventional utility program but builtin the server process and run with the dbctl utility.
dbctl bimport [options] database set bexp_fileThis first erases the specified data set and then processes the specified import file. As the binary import files are directly accessed by the eloqdb6 server process the files must be readable by the user that is configured for the eloqdb6 server and must reside on the same system.
The following options are supported with bimport:
For example:
$ dbctl bimport /v /v sample 1 $PWD/SAMPLE.001.exbThis imports the data set 1 of the database sample from the file SAMPLE.001.exb (an absolute path to the export file is specified).
An example script (bimport.sh) is shown subsequently in this document that may be used to automatically import a single database.
In case a TPI package was used on MPE any indexes must be re-created. Eloquence includes limited support for TPI indexes and may be used to handle indexing. In this case the index is either created with dbutil after migration or added to the schema file before creating the database.
You may also want to consider changing the database security settings as by default the Eloquence schema creates databases that are accessible with the default "public" user.
Item type U (upper case strings)
The item type U (upper case strings) is enforced with Eloquence. With TurboIMAGE the item type U is only used as an application hint. Eloquence will uppercase the item content when the record is modifed.
This immediately affects the text mode import (dbimport). While any item type U items are unmodified when using binary import the item contents will be changed subsequently when modified by the application.
It is recommended to change the item type U to X if lower case data are required in a string.
Binary values placed in a string item (type X or U) may result a subsequent problems. When the client and server use different character sets or byte orders Eloquence internally converts any item values on the fly as required. As Eloquence does not know about them any binary values placed in a string item will also be subject to a conversion, likely resulting in unexpected values.
Binary values in string items work as long as the new target system uses the same character set and byte order as MPE (eg. HP-UX on PA-RISC or Itanium). It will break when converting to a system using a different byte order or character set (eg. Windows on x86).
The recommended solution is to consider correcting the application to use separate items for binary values. As an option changing the item to item type B (binary) indicates to Eloquence that no conversion is allowed an item.
Value aliases (packed and zoned items used as search item)
Packed and zoned items could use a different encoded value to represent the same numerical value. For example, there could be a positive zero, a negative zero and an unsigned zero encoded value that all represent the same numeric value (zero) but use different binary representation.
TurboIMAGE consideres different binary values to be different, independendly if they represent the same numeric value. In a manual master multiple records could be present that have the same numeric value but a different binary encoded value as the search item.
Eloquence uses the numerical value for any comparison (not the binary encoded value) and thus considers records with the same numeric value for a search item a duplicate in a manual master. This might result in a problem when moving existing databases from TurboIMAGE as the database import fails with a status 43 due to a duplicate entry for a manual master while this was no problem on TurboIMAGE.
The suggested solution is to correct your data on MPE before moving it to Eloquence.
When using binary export it is recommended to check the database consistency (for example, using ADAGER or DBGENERL) before migration. With binary export mode any database inconsistencies will also affect Eloquence any may result in a problem during bimport or broken chains after the migration.
The examples below all specifiy to export the data set PO-RECEIPTS (set number 39) of the data base IOSDB, using chained export and the search item PART-NUMBER (item number 159).
DBEXPORT -c IOSDB.QEDDB 39 DBEXPORT -c IOSDB.QEDDB 39.PART-NUMBER DBEXPORT -c IOSDB.QEDDB 39.129 DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.159 DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.PART-NUMBER
This allows binary exporting data sets up to a size of 100 GB with little extra effort by concatenating the files. See below for an example.
Please note that this is different from the bexport option to use multiple input files files (as specified with the options /v and /a). For bimport to accept multiple files for a data set each file must provide a valid BEXP header which is not supported by DBEXPORT at this time.
Overcome the 2 GB filesize limitation with text mode export
To overcome the (MPE) 2 GB filesize limitation with text mode export files you may pipe the output to split.
DBEXPORT -s - database | split -b 1000m - exp.This will split the export file into 1000 MB chunks named exp.aa, exp.ab ... On the target system you may combine the files with a command like below:
cat exp.* > database.expYou may also pass the result to dbimport though stdin:
cat exp.* | dbimport -s - database
Performance problem during import with sorted chains (text mode export)
Sorted chains may cause a performance problem in text mode. It might help to export the database in sorted order. The -S options specifies to use chained export mode giving preference to any sort items (instead of primary path).
dbexport -S databaseThis attempts to retain the chain order for the first sort item which to reduce the overhead of the chain rebuild on the target platform.
Using the -x option (exclude) you may also export the database in two steps. In the first step, all "normal" data sets are processed, exclusing any sets that would benefit from different options. In the second step any data set is exported specifying specific options.
dbexport -c -x database 5This exports the entire database except set #5
dbexport -c database 5.CUSTNOThis uses the chain associated with the item CUSTNO to export set #5.
Overcome the 2 GB filesize limitation with binary mode export
To overcome the (MPE) 2 GB filesize limitation with binary mode export files the dbexport utility automatically creates mupliple files when the export file for a data set would exceed 2 GB.
Binary export files are split into multiple segments for each GB of data. In this case no .exb file but multiple partial files are created that need to be combined on the target platform to run bimport.
For example, exporting set #5 could create the following partial binary export files: SAMPLE.005.b00 SAMPLE.005.b01 SAMPLE.005.b02 ...
The cat command may be used like below to combine these partial binary export files into a complete export file.
cat SAMPLE.005.b* > SAMPLE.005.exbWhile bimport has the option to process partial files (each with a complete bexp header) this approach is currently not supported by DBEXPORT.
# bimport.sh - Eloquence example script to bimport a database # # This erases the target database and then attempts to bimport # any files that follow the pattern DBNAME.999.exb # # As the bexp files are directly accessed by the eloqdb6 server # process the files must be readable by the user that is # configured for the eloqdb6 and must reside on the same system. # # usage: # bimport.sh database # # Note: currently the database name may not specify an Eloquence # instance. Please use EQ_DBSERVER as a workaround. # # Note: bimport needs write access to the database. Don't # specify the dba user with the dbctl bimport. # # Note: Specifying the /v (or /v /v) option with bimport # provides a progress report that is printed to stderr. if [ $# -lt 1 ]; then echo "usage: $0 database" exit fi db=$(echo $1|tr [a-z] [A-Z]) pwd=$(pwd) #opt="/v /v" # erase database dberase $db # concatenate any partial files for b00_file in $db.*.b00 do set=$(echo $b00_file|cut -d. -f2) if [ ! -f "$db.$set.bexp" ]; then echo "creating $db.$set.bexp from fragments ..." cat $db.$set.b[0-9][0-9] > $db.$set.bexp fi done exit for bexp_file in $db.*.exb do set=$(echo $bexp_file|cut -d. -f2) if [ -z "$opt" ]; then echo "$db.$set ... \c" dbctl bimport $db $set $pwd/$bexp_file else echo "$db.$set ..." dbctl bimport $opt $db $set $pwd/$bexp_file fi done