|
The Eloquence MPE database migration utilities provide an easy
and straight forward approach to move your TurboIMAGE databases
to Eloquence.
- The Eloquence schema processor understands TurboIMAGE schema
files so any schema file could simply be transferred to the target
system and used with the Eloquence schema processor.
- Eloquence provides a utility to extract the content of any
database to a set of flat files. These export files are transferred
to the target system and used to reload the database.
Moving your database to Eloquence involves the following steps:
-
Install the DBEXPORT utility on HP3000
-
Export the database
-
Transfer the schema file and the export files to the target system
-
Create and import the database on Eloquence
Limitations:
-
No support for Business Basic decimals
-
Export files are limited in size to 2GB (workaround available)
-
Binary export mode includes the IMAGE record number and chain
pointers along with a binary representation of the record.
Broken chains or corrupted data will show up in Eloquence.
Eloquence comes with utilities to load and unload the database.
Eloquence supports both using text files to hold the database content
or using binary files.
An equivalent utility is also available for the HP3000 and may be used
to unload your database. It can be downloaded from the Eloquence ftp
server at the URL below:
The following files are available:
- LICENSE - license for the migration utilities (BSD like)
- RELNOTES - release notes documenting recent changes
- hp3ksrc.tar.Z - source code
- hp3kbin.tar.Z - binary version, tar format (MPE 6.0 or newer)
Note:
When using ftp please make sure you download the archive files
(.tar.Z) in binary format.
When using ftp to download an archive file to an MPE system you need
to provide an alternative file name which is MPE compliant or prepend
the file name on MPE with a ./ (dot slash) to make sure it is
accepted as a POSIX file name, such as shown below:
ftp> bin
ftp> cd /eloq/B0710/mpe/migration
ftp> get hp3kbin.tar.Z ./hp3kbin.tar.Z
To install the tar archive on your system, please enter the POSIX
shell and follow the procedure below:
$ tar -xvzopf hp3kbin.tar.Z
This restores the archive content in the current directory.
As an alternative, you may extract the archive file from the
MPE shell with a command like below:
:/bin/tar "-xvzopf hp3kbin.tar.Z"
To use binary export (recommended) the DBEXPORT utility needs to
run in privileged mode and must be installed in a privileged group.
Also, the program capabilities must be changed with linkedit as below:
:linkedit "altprog DBEXPORT; cap=ia,ba,pm"
This adds the PM capability to DBEXPORT.
The source code to the Eloquence data migration utilities is
available as an option. To build your own binary from source
code you need a HP C compiler.
Download the source archive to MPE, unpack it with tar and run
make -f Makefile.MPE (in a POSIX shell).
As a result the file DBEXPORT and DBINFO are built.
Compiling was only tested with the HP C compiler on MPE 6.5 or newer.
As the MPE 6.0 HP C compiler does not support the long long type
you need to modify the Makefile to set the HAS_LONG_LONG=0 to get
build on MPE 6.0.
DBEXPORT is used to export the database contents to one or multiple
text or binary files. To export to binary files, DBEXPORT must be run
in PRIV mode.
When started without command line arguments (or with the -help option)
the command line options are printed.
usage: DBEXPORT [options] database [set[.item] ...]
options:
-help - show usage (this list)
-version - show version information
-u user - set user name
-p pswd - set password
-o path - set output directory (not single file)
-v - verbose output
-c - chained export
-r - create restructure information
-s file - output into single file, '-' = stdout
-x - exclude specified data sets
-B - binary export mode (needs PM)
-S - chained export by sort item
The MPE DBEXPORT utility is similar to but different from the
Eloquence dbexport utility. While the options (if available) are the
same not all options are available for both versions.
- The -help option prints a short overview on the command
line options.
- The -version option prints version information.
- The -u user option specifies the optional USERIDENT to open
the database (this is rarely used on TurboIMAGE).
- The -p pswd option specifies the password to open the
database.
- The -o option specifies the directory where export files
are created. By default export files are created in the current
directory.
- The -v option prints progress information during runtime
- The -c option uses the primary path to determine the order
of detail records (only relevant for text mode export).
By default, detail records are exported in serial order.
- The -r option causes dbexport to output additional information
in the export file on the data set structure (only relevant for text
mode export). This may be used by dbimport if the database structure
of the source and target database is different.
- The -s file option allows to specify a single file for text
mode export. If a - (dash) is used, dbexport will write to stdout.
If not specified, the output is written to multiple files in the
current directory or the directory specified with the -o option.
- The -x option may be used to indicate that any specified
data set should be excluded from export.
- The -B (upper case) option specifies to create binary export files.
This requires to run dbexport in PRIV mode.
- The -S (upper case) option may be used to specify using the first
sort item preferably to specify the export order of records in a detail set
(instead of using the primary path order or serial order). This may
be used to improve dbimport performance on detail sets with sorted chains
when the sort item is different from the primary path.
In addition, the database to be exported must be specified and optionally
a list of data sets may be present (either data set name or number).
If a list of data sets is provided only the specified data sets are
processed.
For a detail data set, a search item may also be specified to define
the detail set export order.
The typically used options are:
-
The -p passwd option specifies the database password used
to open the database.
-
the -B option specifies to use binary export mode.
-
The -s file option specifies that a single file should be used
which contains all data sets (text mode).
-
The -v (verbose) option displays the progress
Any data set name or number specified after the database name
indicates that only the specified data sets should be exported.
When the -x option is present, any specified data sets are not
exported.
For example:
$ DBEXPORT -p secret -v sample
Export the database sample in text mode. The database is opened
with the password "secret".
$ DBEXPORT -p secret -B -v sample
Export the database sample in binary mode.
$ DBEXPORT -p secret -v -s sample.exp sample
Export the database sample in text mode, the results are written
to a single text file.
$ DBEXPORT -p secret sample customers parts
$ DBEXPORT -p secret sample 1 2
Export the data sets customers and parts in text mode.
$ DBEXPORT -p secret -x sample 1 2
Export all but the data sets customers and parts in text mode.
All the examples above used the POSIX shell. To use dbexport from
the MPE shell you need to enclose the arguments in quotes
: DBEXPORT "-p secret -v sample"
See also:
http://eloquence.marxmeier.com/doc/html/manuals/dbms/dbms-57.html
http://eloquence.marxmeier.com/support/misc/export.html
The text mode dbexport converts the database content to text.
Unless the -s option is specified a separate text file is created
for each data set (named DBNAME.999.exp).
Any empty data sets or automatic masters are skipped and no
export file is created.
If the -c option is present, any detail records are exported
in an order to retain the primary path.
Please note that DBEXPORT replaces any invalid numeric values by
a zero value during export (eg. corrupted packed/zoned numbers).
See below for a workaround to overcome the
2 GB file size limitation on MPE for bytestream files.
The binary export mode is used when the -B (upper case B) command
line option is specified.
The binary export mode creates a separate file for each dataset
(unless it has zero entries). DBEXPORT uses the naming convention
DBNAME.999.exb for each binary export file. If the size of the
binary export file for a data set would exceed 1 GB instead
multiple partial files are created named DBNAME.999.b## where ## is
the sequential number of this file.
The binary export file includes a header to identify the data in the
file and for each record the original record number and the chain
pointers (besides the sysnonym chain information for a master set).
This information may then be used by the Eloquence bimport to
efficiently import the database into Eloquence, retaining the
chain order and record number for any detail set.
See below for a workaround to overcome the
2 GB file size limitation on MPE for bytestream files.
DBINFO lists the data sets for the specified database
including data set name, type, number of entries and capacity.
When started without command line arguments (or with the -help option)
the command line options are printed.
usage: DBINFO [options] database [set ...]
options:
-help - show usage (this list)
-version - show version information
-u user - set user name
-p pswd - set password
- The -help option prints a short overview on the command
line options.
- The -version option prints version information.
- The -p pswd option specifies the password to open the
database.
database is the name of the database.
Optionally, one or more data set names or numbers can be specified
after the database name (separated by spaces). In this case only
information on the specified data sets is returned. By default
all accessible data sets are displayed.
:dbinfo testdb
Processing database : testdb
SET NAME RECLEN CAPACITY ENTRIES
---------------- -- - ------ -------- --------
TEST 01 D 111 100 1
To use DBINFO from the MPE shell you need to enclose the arguments
in quotes if there is more than one argument.
It is typically preferable to export the database in binary mode
as this usually results in better import performance and also
retains the order for all paths.
Exporting the database in text mode is easier to use, the export
files are typically smaller and source and target databases may
differ to some extent.
Exporting the database in text mode:
When running the DBEXPORT utility from the POSIX shell the arguments
are separated by a space.
$ DBEXPORT -p SECRET -v SAMPLE
When running from the MPE shell (CI) you need to enclose the arguments
in quotes.
: DBEXPORT "-p SECRET -v SAMPLE"
This creates output files SAMPLE.001.exp ... (one for each non-empty
data set, except for automatic masters).
Exporting the database in binary mode:
When running the DBEXPORT utility from the POSIX shell the arguments
are separated by a space.
$ DBEXPORT -B -p SECRET -v SAMPLE
When running from the MPE shell (CI) you need to enclose the arguments
in quotes.
: DBEXPORT "-B -p SECRET -v SAMPLE"
This creates output files SAMPLE.001.exb ... (one for each non-empty
data set). Please note that automatic master sets are exported
as well.
ftp may be used to transfer the schema file and the export files
to the target system.
To transfer the schema file, it must either be converted to bytestream
format (/bin/tobyte [-a] -t) and then transferred in binary mode
or you may transfer it in text mode.
The ftp utility will then convert it to bytestream mode during the transfer.
Any export files (either text or binary export files) must be
transferred in binary mode.
On Eloquence, first run the schema utility and the dbcreate utility.
dbschema schemafile
dbcreate database
Where schemafile is the file name of the schema file and database
is the name of the database.
Note: dbschema is equivalent to schema -T. The -T option to
schema indicates that the schema file follows the TurboIMAGE
conventions. Eloquence schema files are similar to TurboIMAGE
schema files but use different options and item sizes are
specified in bytes instead of item type specific units.
When using text mode export, the Eloquence dbimport utility is
used to populate the database from the export files.
$ dbimport [options] database {set [...]]
Typical options:
-
The -v (verbose) option displays the progress
-
The -s file specifies that all data are contained in a single file.
-
database is the name of the database and may in addition specify
the server system and database instance.
A list of data set names or numbers can be can be specified
after the database name. In this case only the specified data
sets are processed. By default all accessible data sets are imported.
For example:
$ dbimport -v sampledb
$ dbimport -v -s sampledb.exp localhost:8208/sampledb
On the Windows and Linux platform you should use the
-z roman8 command line option of dbimport to make
sure any national charaters ("Umlaute") are converted properly.
Please note:
Unless you import a trivial volume of data you may want to consider
to set SyncMode = 0
in your eloqdb6.cfg configuration file during the dbimport and
disable any forward logging. dbctl may be used for both:
dbctl -u dba syncmode off
dbctl -u dba forwardlog disable
The new bimport function allows to efficiently import data from
TurboIMAGE while retaining the chain information. For a detail
set the record number is also retained.
While undocumented, the bimport function is included for testing
purposes in the most recent Eloquence B.07.00 eloqdb6 patches
for the HP-UX platform.
Currently, bimport is not a conventional utility program but builtin
the server process and run with the dbctl utility.
dbctl bimport [options] database set bexp_file
This first erases the specified data set and then processes the
specified import file. As the binary import files are directly
accessed by the eloqdb6 server process the files must be readable by
the user that is configured for the eloqdb6 server and must reside
on the same system.
The following options are supported with bimport:
-
/v - Display additional information.
Specifying the option /v twice (/v /v) causes the import
progress to be output to stderr for every 10,000 records.
-
/a - In case more than one bexp file is needed to import a data set
this flag is specified with any subsequent bexp file.
-
/n - This file must be specified if additional bexp files exist
for this data set.
Note: As the current MPE dbexport only creates a single binary export
file for each data set, the options /a and /n should not be used
(these options require a separate header for each bexp file).
See below for a workaround to overcome the
2 GB file size limitation on MPE for bytestream files.
For example:
$ dbctl bimport /v /v sample 1 $PWD/SAMPLE.001.exb
This imports the data set 1 of the database sample from the
file SAMPLE.001.exb (an absolute path to the export file is specified).
An example script (bimport.sh) is shown subsequently in this
document that may be used to automatically import a single database.
In addition to moving the database it might be necessary to update
the database settings (eg. CIUPDATE flag) according to the original
setting on MPE. This is done with dbutil.
In case a TPI package was used on MPE any indexes must be re-created.
Eloquence includes limited support for TPI indexes and may be used
to handle indexing. In this case the index is either created with dbutil
after migration or added to the schema file before creating the database.
You may also want to consider changing the database security settings
as by default the Eloquence schema creates databases that are accessible
with the default "public" user.
A number of subtle differences between Eloquence and TurboIMAGE may
affect the database migration. The potential issues below were
encountered in migration projects:
Item type U (upper case strings)
The item type U (upper case strings) is enforced with Eloquence.
With TurboIMAGE the item type U is only used as an application hint.
Eloquence will uppercase the item content when the record is modifed.
This immediately affects the text mode import (dbimport).
While any item type U items are unmodified when using binary import
the item contents will be changed subsequently when modified by the
application.
It is recommended to change the item type U to X if lower case data
are required in a string.
Binary values in strings
Binary values placed in a string item (type X or U) may result a
subsequent problems.
When the client and server use different character sets or byte
orders Eloquence internally converts any item values on the fly
as required.
As Eloquence does not know about them any binary values placed
in a string item will also be subject to a conversion,
likely resulting in unexpected values.
Binary values in string items work as long as the new target system
uses the same character set and byte order as MPE (eg. HP-UX on
PA-RISC or Itanium). It will break when converting to a system
using a different byte order or character set (eg. Windows on x86).
The recommended solution is to consider correcting the application to
use separate items for binary values. As an option changing the item
to item type B (binary) indicates to Eloquence that no conversion
is allowed an item.
Value aliases (packed and zoned items used as search item)
Packed and zoned items could use a different encoded value to
represent the same numerical value. For example, there could be
a positive zero, a negative zero and an unsigned zero encoded value
that all represent the same numeric value (zero) but use different
binary representation.
TurboIMAGE consideres different binary values to be different,
independendly if they represent the same numeric value.
In a manual master multiple records could be present that have
the same numeric value but a different binary encoded value
as the search item.
Eloquence uses the numerical value for any comparison (not the binary
encoded value) and thus considers records with the same numeric value
for a search item a duplicate in a manual master.
This might result in a problem when moving existing databases from
TurboIMAGE as the database import fails with a status 43 due to a
duplicate entry for a manual master while this was no problem
on TurboIMAGE.
The suggested solution is to correct your data on MPE before moving
it to Eloquence.
When using binary export it is recommended to check the database
consistency (for example, using ADAGER or DBGENERL) before migration.
With binary export mode any database inconsistencies will also affect
Eloquence any may result in a problem during bimport or broken chains
after the migration.
- New -S option
-
The new option -S (upper case S) may be used to specify chained
export mode and that a sort item has preference over the primary path.
The option -S implies option -c (chained export mode).
Please note that the -S option only affects the default path to be used
for (text mode) exporting a detail set. A separate path may still
be specified.
- Specifying the search item in chained export mode
-
When using chained export mode (option -c or -S present) the search
item for a detail set may be specified to override the default of
using the primary path. The (detail) search item number may be
specified with the dataset, seperated by a dot.
The examples below all specifiy to export the data set PO-RECEIPTS
(set number 39) of the data base IOSDB, using chained export and
the search item PART-NUMBER (item number 159).
DBEXPORT -c IOSDB.QEDDB 39
DBEXPORT -c IOSDB.QEDDB 39.PART-NUMBER
DBEXPORT -c IOSDB.QEDDB 39.129
DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.159
DBEXPORT -c IOSDB.QEDDB PO-RECEIPTS.PART-NUMBER
- Binary export mode was enhanced to split bexp files
-
When using binary export mode (-B option present), a new export
file is created for each GB of data. In this case no .exb file
but multiple partial files are created that need to be combined
to run bimport.
This allows binary exporting data sets up to a size of 100 GB with
little extra effort by concatenating the files.
See below for an example.
Please note that this is different from the bexport option to use
multiple input files files (as specified with the options /v and /a).
For bimport to accept multiple files for a data set each file must
provide a valid BEXP header which is not supported by DBEXPORT at
this time.
The following workarounds may be useful to overcome some commonly
encountered limitations with the dbexport database migration tool.
Overcome the 2 GB filesize limitation with text mode export
To overcome the (MPE) 2 GB filesize limitation with text mode export
files you may pipe the output to split.
DBEXPORT -s - database | split -b 1000m - exp.
This will split the export file into 1000 MB chunks named
exp.aa, exp.ab ... On the target system you may combine the
files with a command like below:
cat exp.* > database.exp
You may also pass the result to dbimport though stdin:
cat exp.* | dbimport -s - database
Performance problem during import with sorted chains (text mode export)
Sorted chains may cause a performance problem in text mode. It
might help to export the database in sorted order. The
-S options specifies to use chained export mode giving
preference to any sort items (instead of primary path).
dbexport -S database
This attempts to retain the chain order for the first sort
item which to reduce the overhead of the chain rebuild on
the target platform.
Using the -x option (exclude) you may also export the
database in two steps. In the first step, all "normal"
data sets are processed, exclusing any sets that would
benefit from different options. In the second step any
data set is exported specifying specific options.
dbexport -c -x database 5
This exports the entire database except set #5
dbexport -c database 5.CUSTNO
This uses the chain associated with the item CUSTNO to
export set #5.
Overcome the 2 GB filesize limitation with binary mode export
To overcome the (MPE) 2 GB filesize limitation with binary mode
export files the dbexport utility automatically creates mupliple
files when the export file for a data set would exceed 2 GB.
Binary export files are split into multiple segments for
each GB of data. In this case no .exb file but multiple
partial files are created that need to be combined on the
target platform to run bimport.
For example, exporting set #5 could create the following
partial binary export files:
SAMPLE.005.b00 SAMPLE.005.b01 SAMPLE.005.b02 ...
The cat command may be used like below to combine these
partial binary export files into a complete export file.
cat SAMPLE.005.b* > SAMPLE.005.exb
While bimport has the option to process partial files (each
with a complete bexp header) this approach is currently not
supported by DBEXPORT.
Example shell script to bimport a database. This also
combines any partial bexport files before importing.
# bimport.sh - Eloquence example script to bimport a database
#
# This erases the target database and then attempts to bimport
# any files that follow the pattern DBNAME.999.exb
#
# As the bexp files are directly accessed by the eloqdb6 server
# process the files must be readable by the user that is
# configured for the eloqdb6 and must reside on the same system.
#
# usage:
# bimport.sh database
#
# Note: currently the database name may not specify an Eloquence
# instance. Please use EQ_DBSERVER as a workaround.
#
# Note: bimport needs write access to the database. Don't
# specify the dba user with the dbctl bimport.
#
# Note: Specifying the /v (or /v /v) option with bimport
# provides a progress report that is printed to stderr.
if [ $# -lt 1 ]; then
echo "usage: $0 database"
exit
fi
db=$(echo $1|tr [a-z] [A-Z])
pwd=$(pwd)
#opt="/v /v"
# erase database
dberase $db
# concatenate any partial files
for b00_file in $db.*.b00
do
set=$(echo $b00_file|cut -d. -f2)
if [ ! -f "$db.$set.bexp" ]; then
echo "creating $db.$set.bexp from fragments ..."
cat $db.$set.b[0-9][0-9] > $db.$set.bexp
fi
done
exit
for bexp_file in $db.*.exb
do
set=$(echo $bexp_file|cut -d. -f2)
if [ -z "$opt" ]; then
echo "$db.$set ... \c"
dbctl bimport $db $set $pwd/$bexp_file
else
echo "$db.$set ..."
dbctl bimport $opt $db $set $pwd/$bexp_file
fi
done
|
