Database changes and limitations

Contents:

• Changes and fixed problems
  • Pre-release change history

• Known problems and limitations
• Backwards compatibility

• QUERY changes

Changes and fixed problems

• On HP-UX, Eloquence libraries are no longer installed in the Eloquence lib directory. A subdirectory has been introduced which separates different library versions.

  The following conventions apply:

  pa11_32	PA-RISC 1.1 compatible libraries (32 bit)
  pa20_32	PA-RISC 2.0 libraries (32 bit)
  pa20_64	PA-RISC 2.0 libraries (64 bit)
  hpux32	IA64 32 bit libraries (Itanium)
  hpux64	IA64 64 bit libraries (Itanium)

This is a summary of all changes and fixed problems relative to Eloquence version B.06.32.

• DBOPEN has been changed to return a more specific status when it fails to open a database (#471, #1694).
  • Status -21 is returned if the user has no access to the database. When the user is a member of an administrative group for the database (as the database creator is by default), opening the database succeeds although no access to any data set may be possible.

  • Status -92 is returned, when the database is not created. When the current user is a member of an administrative group (as the database creator is by default), opening the database succeeds although it has not been created.

  • Status -3 or -4 is returned if the user is either unknown to the server process or the wrong password was provided.

  • Status -2 is returned when the database cannot be opened in the specified mode (due to a concurrent open). The tenth status element returns additional information:
    • 1 - The database is opened concurrently in an exclusive mode
    • 2 - The database cannot be opened in an exclusive mode because it is in concurrent use
    • 3 - The database can only be opened in readonly mode (concurrent open in mode 8)
    • 4 - The database cannot be opened in read exclusive mode (mode 8) because it is opened concurrently in read/write mode
    This status is returned as -1 in the Eloquence language for compatibility reasons.

  • Status -1 is returned otherwise.

• Restoring a database will now place the executing user into the dba group to allow administrative access to the restored database (#799). Previously, restoring a database could fail if the user had no administrative access to the database to be restored.

• dbexport performance has been improved significantly (#1587). dbexport has been modified to use the more efficient bulk fetch access method (if supported by the server). The -b command line option can be used to fallback to the previous behavior.

• The dbimport utility has been changed to be more useful when using a single import file (-s option given). dbimport does now import in export file order. Furthermore, in restructure mode (-r option present) it is able to make use of the data set names embedded in the file. This also fixes a number of minor dbimport problems (#952,#1339,#1744).

• The NoDNS configuration item is now set by default (#1649). This change disables the reverse DNS lookup for incoming connections by default. The reverse lookup is known to cause eloqdb6 crashes with some glibc versions used by recent Linux distributions (Red Hat 8.x and SuSE Linux 8.1). It also may cause delays on all platforms in case of a wrong DNS configuration.

• Added support for dbctl "backup status" command which returns current status of on-line backup (#677). The following messages are returned:

    On-line backup mode is inactive.
    On-line backup mode is active.
  

• DBOPEN mode 8 did not allow a concurrent read-only DBOPEN mode (such as mode 8 or mode 9).

• The max. number of I/O threads has been increased to 64. The previous max. value of 8 I/O threads could result in performance issues with large databases and a big number of concurrent users.

• I/O scheduling and completion processing has been optimized to consume less CPU resources with bigger configurations. With previous versions a configuration with a large number of threads could cause a noticeable overhead.

• dbctl shutdown could result in an error message about a problem communicating with the database server (#1161).
  This was caused by a race condition where the server could terminate before dbctl could close the connection properly when IPC communication (EnableIPC) was used. An error message is like below could be output:

    $ dbctl -u dba shutdown
    Shutting down now.
    (14909) P0: Unable to up semaphore
    (14909) P0: semop(UP): Invalid argument (errno 22)
    (14909) P0: Server notification failed 
  

  A related problem could also cause a resource leak during shutdown where a shared memory segment was not freed.

• dbvolextend now requires that the volume type is specified explicitely on the command line using the -t command line option (#1299). Previous versions of dbvolexend used a default DATA volume type. This change was made to avoid possible mistakes where a DATA volume instead of a LOG volume is created.

• dbvolextend no longer issues warning messages when recreating a missing log volume after recovering from online backup (#826). LOG volumes may be omitted from an online backup and can be recreated with the dbvolextend utility if necessary.

• dbfsck refuses to execute in write mode if the server was not shutdown cleanly unless the -f command line option is specified. When the server is terminated unexpectedly important recovery information is saved in the log volume. Since dbfsck ignores any log volume content improper usage of dbfsck write mode may cause volume corruption.

• A problem was fixed which could cause the database server process to spin in the idle thread instead of blocking and consuming considerable CPU resources.

  This problem was caused by a corner case when the configured connections were exhausted and the HTTP status was not configured. Instead of rejecting additional incoming connections the database server could accept one extra connection but was unable to associate the network events properly (#1336).

• dbctl killthread now allows to terminate an idle session (#798). Previously, only blocked sessions could be terminated.

• When a log file is used (instead of syslog) eloqdb6 will prepend each line with the current date and time (#923).

• If a volume reaches its maximum size redundant log messages are suppressed (#1063). Previoiusly, a note was written to the log for each log volume which had reached its max. size. This note is only output once now.

• Fixed the HTTP status performance display. Average is estimated for the last 10 minutes (#1008). Bar graph images can now be cached in the web browser (#1342).

• The HTTP status now displays the server title in the page header if specified in the server configuration file (#1173).

• To avoid compiler warnings the eloqdb.h header file has been changed. The buffer arguments for the idb_get(), idb_update() and idb_put() prototypes have been changed from char* to void* (#1102).

• DBUPDATE mode 1 does no longer allow to change a sort item and a database status is returned (#986).

• DBUPDATE could fail to check if the updated record was covered by a concurrent lock. A status -12 is returned if the new record is covered by a concurrent lock (#986).

• Added support for default collating sequence which is used for master set indexes and sort items.

• If the schema processor is unable to create the database the reason (as returned by the server) is output (#1204).

   *** (FATAL) Database "SAMPLEDB" already exists
  

• prschema could return wrong access rights in the schema file (#668).

• The AllowSecondaryBlockingLock configuration item is now enabled by default (#1272).

• The Eloquence client side API and all database tools have been modified to support the EQ_DBSERVER, EQ_DBUSER and EQ_DBPASSWORD environment variables.
  The EQ_DBSERVER, EQ_DBUSER and EQ_DBPASSWORD environment variables can be used to specify the default database server, user and password. Please refer to the sections on Database Names and Authorization for a description.

• The dbdumpcat utility has been improved to support an additional output format which is easier to parse in a script and allows listing the catalog tables.

• dbimport on HP-UX and Linux handles export files which use CR/LF line terminators.

• New DBINFO mode 505 has been added to return a list of all data sets which contain a specific index item.

• New DBINFO mode 303 has been added to return sort item with path information. This supplements DBINFO mode 301 which does not support sort items.

• Index items can now be named identically to data items.

• Performance enhancements. For some write intensive workloads the performance has increased by 25%.

• Improved reliability in case of abnormal server termination.

Pre-release change history

Note: The following recent changes might affect you when updating from a previous Eloquence B.07.00 pre-release.

• As of PR#8 the forward-log configuration has changed. The relevant items are now configured in a separate section in the database server configuration file. If you use forward logging, please make the appropriate changes to your eloqdb6.cfg file.

• As of PR#12 additional forward-log configuration items are available to allow defining the eloqdb6 behavior in case of a failure to write to the forward-log file and configure the maximum size of automatically managed forward-log files.

B.07.00 Release:

• Support use of collating sequences for lookup on master sets using DBFIND modes 6 and 7 (#1360).

• Fixed a problem with the DBFIND mode 6 and 7 to enable usage with master sets and search items. Please note that this feature is currently not enabled on the server side.

• Handle SIGPIPE signal more gracefully. A configuration mistake in the server configuration file could result in a SIGPIPE signal and shutdown the server process.

• The eqdb database client library could pollute the application namespace. All internal functions are now prefixed with eq__ to avoid interaction with an application.

• DBOPEN has been changed to return a more specific status when it fails to open a database (#471, #1694).
  • Status -21 is returned if the user has no access to the database. When the user is a member of an administrative group for the database (as the database creator is by default), opening the database succeeds although no access to any data set may be possible.

  • Status -92 is returned, when the database is not created. When the current user is a member of an administrative group (as the database creator is by default), opening the database succeeds although it has not been created.

  • Status -3 or -4 is returned if the user is either unknown to the server process or the wrong password was provided.

  • Status -2 is returned when the database cannot be opened in the specified mode (due to a concurrent open). The tenth status element returns additional information:
    • 1 - The database is opened concurrently in an exclusive mode
    • 2 - The database cannot be opened in an exclusive mode because it is in concurrent use
    • 3 - The database can only be opened in readonly mode (concurrent open in mode 8)
    • 4 - The database cannot be opened in read exclusive mode (mode 8) because it is opened concurrently in read/write mode
    This status is returned as -1 in the Eloquence language for compatibility reasons.

  • Status -1 is returned otherwise.

• Restoring a database will now place the executing user into the dba group to allow administrative access to the restored database (#799). Previously, restoring a database could fail if the user had no administrative access to the database to be restored.

• dbexport performance has been improved significantly (#1587). dbexport has been modified to use the more efficient bulk fetch access method (if supported by the server). The -b command line option can be used to fallback to the previous behavior.

• dbexport in chained mode (-c option present) could fail with status -21 on DBINFO mode 302. Standalone detail data sets were not handled correctly, caused by a previous change.

• The dbimport utility has been changed to be more useful when using a single import file (-s option given). dbimport does now import in export file order. Furthermore, in restructure mode (-r option present) it is able to make use of the data set names embedded in the file. This also fixes a number of minor dbimport problems (#952,#1339,#1744).

• dbutil: The function key f7 did not work correctly in the "Manage Data Set Access" dialog (#1592).

• Restructuring a data set could cause the eloqdb6 process to abort with a panic message:

    server panic: Fatal problem detected in Node_FillItems
    Assertion failed: item->id
    server panic: Aborting on internal failure, file nodeilist.c, line 354
  

  This crash happened when no unused records were available (the number of records exactly matches the cluster boundary). In this case, a problem in the record restructuring was exposed (#1785).

• The NoDNS configuration item is now set by default (#1649). This change disables the reverse DNS lookup for incoming connections by default. The reverse lookup is known to cause eloqdb6 crashes with some glibc versions used by recent Linux distributions (Red Hat 8.x and SuSE Linux 8.1). It also may cause delays on all platforms in case of a wrong DNS configuration.

• If a forward-log cannot be written, e.g. due to insufficient disk space, the default action is now to disable forward-logging and continue normal server operation. Previously, the server issued a panic and stopped. If desired, the previous behavior can be configured (#1670).

• If a forward-log is re-enabled (dbctl -u dba forwardlog enable) creation of the new forward-log is delayed until the next on-line backup or server restart.

• Recovering from a forward-log could fail with the following message (#1726):

    panic: Fatal problem detected in Node_RecoverItemPage
    Assertion failed: rec->timestamp == ihead->timestamp + 1
    file B0700/eqdb/server/storage/nodeilist.c, line 1318
  

• On Windows, using LogLevel 3 could result in a server crash. The Eloquence database utilities were affected as well and using LogLevel 3 could cause the utility program to abort.

• DBLOGON could cause a server panic if an incorrectly encrypted password was transmitted (#1350):

    server panic: Fatal problem detected in decrypt_pswd
    Assertion failed: decrypt_pswd() failed on decrypt_msg_hex()
    server panic: Aborting on internal failure, file dbml.c, line 1037
  

  This has been solved by returning the status -4 (S_BADPSWD) in case the password cannot be decrypted but the user demands a password.

• Schema processor changes / fixes:
  • Fixed possible schema crash if capacity inc was specified (applies to TurboIMAGE compatible schema files).
  • The schema processor was modified to allow for 64 user classes. This improves compatibility to TurboIMAGE schema files (#1396)
  • The capacity was removed from the $CONTROL TABLES listing since it has no relevance for Eloquence (#1502)
  • Removed warning on a manual master with a search item other than first (#1525)
  • Schema now supports that the item type and size are separated by white space.

• Fixed a problem in the database client library which could cause the EQ_PASSWORD environment variable to be ignored (#1531)

• Fixed a problem in the Eloquence client side library related to search items other than first. You need the image3k library X.23.09.01 or above for this to work correctly.

• A number of problems have been fixed with dbutil and some missing functionality was added.
  • dbutil can now rename a database
  • dbutil now allows to insert an item in a data set entry instead of appending it.
  • dbutil can handle collating sequences

  Please make a backup before using dbutil on important data. The dbutil utility and database restructuring is new functionality and a bug may result in data corruption.

• The prschema utility has been improved to handle a default collating sequence more consistently.

• Added Linux glibc2.2 build.

• Changed library version code to A.07.00.02

• The forward-log configuration has changed. The relevant items are now configured in a separate section in the database server configuration file.

• The dbutil utility has been changed substantially and provides a number of new options:
  • An interactive user interface allows easy management of database security options and database properties.
  • The scripting abilities have been improved to support database structural maintenance and other adminitrative operations. Please refer to the dbutil syntax description for additional information.

• The restructuring functions have been enabled in the database server. This is used with the dbutil utility to perform database structural maintanance.

• A critical issue with dbimport was fixed which could import result in corrupted data in a specific case. With dbexport, numerical data is output in hex format (starting lower case x followed by hex digits, big endian) whenever an item type cannot be converted natively (ie. in decimal digit format). This is a rarely used function and would be used, for example, with 128 bit integer values. The dbimport utility had a bug in scanning these hex numbers which could result in corrupted information.

• The schema processor aborted with a segmentation violation if a TurboIMAGE schema file contained volume appendixes (,DISC).

• Added support for dbctl "backup status" command which returns current status of on-line backup (#677). The following messages are returned:

    On-line backup mode is inactive.
    On-line backup mode is active.
  

• dbimport has been fixed to perform data conversions during dbimport (dbimport restructuring mode, through the -r command line option). When enabled, dbimport will use a specification file or information optionally embedded in the export file to define value assignments and type conversions during dbimport.
  This change has only been lightly tested and we do not recommend to use the dbimport restructuring mode (-r command line option) except for testing purposes.

• A problem was fixed where passing an invalid item or data set number could result in a program abort (bus error). Qualifiers which do not meet the alignment requirements are now considered a string argument.

• DBOPEN mode 8 did not allow a concurrent read-only DBOPEN mode (such as mode 8 or mode 9).

• The max. number of I/O threads has been increased to 64. The previous max. value of 8 I/O threads could result in performance issues with large databases and a big number of concurrent users.

• I/O scheduling and completion processing has been optimized to consume less CPU resources with bigger configurations. With previous versions a configuration with a large number of threads could cause a noticeable overhead.

• A race condition was fixed which could result in a dbpurge or dberase failure or a server panic (#1165).

  A race condition with dberase or dbpurge and the internal checkpoint thread could cause dberase to fail in some cases if the database was modified since the last checkpoint operation. A message like below was logged:

    Node_Reset() failed: Node is still referenced ...
  

  On a related problem a dbpurge operation could result in a server panic:

   D0: server panic: Fatal problem detected in Node_Delete
   D0: Assertion failed: !node->ref_count
   D0: server panic: Aborting on internal failure, file nodecore.c, line 241
  

• An internal deadlock condition was fixed which could occur with the schema processor. The schema processor could hang while writing to the database catalog and the server would hang on shut down (#1237).

• A corruption problem was fixed which could happen after recovery from abnormal server termination (#1228).

• dbctl shutdown could result in an error message about a problem communicating with the database server (#1161).
  This was caused by a race condition where the server could terminate before dbctl could close the connection properly when IPC communication (EnableIPC) was used. An error message is like below could be output:

    $ dbctl -u dba shutdown
    Shutting down now.
    (14909) P0: Unable to up semaphore
    (14909) P0: semop(UP): Invalid argument (errno 22)
    (14909) P0: Server notification failed 
  

  A related problem could also cause a resource leak during shutdown where a shared memory segment was not freed.

• dbvolextend now requires that the volume type is specified explicitely on the command line using the -t command line option (#1299). Previous versions of dbvolexend used a default DATA volume type. This change was made to avoid possible mistakes where a DATA volume instead of a LOG volume is created.

• dbvolextend no longer issues warning messages when recreating a missing log volume after recovering from online backup (#826). LOG volumes may be omitted from an online backup and can be recreated with the dbvolextend utility if necessary.

• dbfsck refuses to execute in write mode if the server was not shutdown cleanly unless the -f command line option is specified. When the server is terminated unexpectedly important recovery information is saved in the log volume. Since dbfsck ignores any log volume content improper usage of dbfsck write mode may cause volume corruption.

• Support for search items in manual master sets which are not the first item in the entry has been added. Previous Eloquence versions required that the first item in a manual master set was used as the search item.
  Currently, this option must be enabled explicitely by specifying the -x command line option to the schema processor, otherwise an error message is issued by schema (#1230).

   *** Search item must be the first item in master sets
  

• A problem was fixed which could cause the database server process to spin in the idle thread instead of blocking and consuming considerable CPU resources.

  This problem was caused by a corner case when the configured connections were exhausted and the HTTP status was not configured. Instead of rejecting additional incoming connections the database server could accept one extra connection but was unable to associate the network events properly (#1336).

• dbctl killthread now allows to terminate an idle session (#798). Previously, only blocked sessions could be terminated.

• When a log file is used (instead of syslog) eloqdb6 will prepend each line with the current date and time (#923).

• If a volume reaches its maximum size redundant log messages are suppressed (#1063). Previoiusly, a note was written to the log for each log volume which had reached its max. size. This note is only output once now.

• Fixed the HTTP status performance display. Average is estimated for the last 10 minutes (#1008). Bar graph images can now be cached in the web browser (#1342).

• The HTTP status now displays the server title in the page header if specified in the server configuration file (#1173).

• To avoid compiler warnings the eloqdb.h header file has been changed. The buffer arguments for the idb_get(), idb_update() and idb_put() prototypes have been changed from char* to void* (#1102).

• DBUPDATE mode 1 does no longer allow to change a sort item and a database status is returned (#986).

• DBUPDATE could fail to check if the updated record was covered by a concurrent lock. A status -12 is returned if the new record is covered by a concurrent lock (#986).

• Added support for default collating sequence which is used for master set indexes and sort items.

• If the schema processor is unable to create the database the reason (as returned by the server) is output (#1204).

   *** (FATAL) Database "SAMPLEDB" already exists
  

• prschema could return wrong access rights in the schema file (#668).

• Support for shared memory communication (EnableIPC) with memory windows on the HP-UX platform has been implemented (#1259).

• The sysdbproperty catalog table has been added which is used to store database properties such as the default collating sequence.

• The AllowSecondaryBlockingLock configuration item is now enabled by default (#1272).

• An internal deadlock situation has been fixed which could result in a hang of the eloqdb6 server process (#1166).

• While the eloqdb6 shutdown has been initiated but not yet completed, the server could spin in the idle thread rather than blocking, causing 100% CPU load until the shutdown completes.

• The Eloquence client side API and all database tools have been modified to support the EQ_DBSERVER, EQ_DBUSER and EQ_DBPASSWORD environment variables.
  The EQ_DBSERVER, EQ_DBUSER and EQ_DBPASSWORD environment variables can be used to specify the default database server, user and password. Please refer to the sections on Database Names and Authorization for a description.

• The schema processor has been modified to improve TurboIMAGE syntax compatibility.

• The prschema utility has been improved to support the new B.07.00 functionality.

• The dbdumpcat utility has been improved to support an additional output format which is easier to parse in a script and allows listing the catalog tables.

• A problem with the DBFIND function was fixed to properly handle the situation where an index item has the same name as a search item but was not valid for the specified data set.

• A bug in processing sort items has been fixed which could result in an internal error.

  server panic: Fatal problem detected
  Assertion failed: ofs <= dset->rec_len
  server panic: Aborting on internal failure, file put.c, line 503
  

• The internal conversion buffer size for the dbimport utility has been increased to account for some corner cases.

• dbimport on HP-UX and Linux handles export files which use CR/LF line terminators.

• A failure during forward-logging, eg. due to a file system overflow, could cause the internal generation counters of the data and log volumes to differ after restarting the server. The server would refuse to start, requiring manual recreation of the log volumes with dbvolextend.

• The TurboIMAGE DBFIND TPI modes 1 and 21 did not work as expected when used with wildcard search patterns.

• The static Eloquence database library is provided for applications which do not allow dynamic linking. We, however, do not recommend linking your programs statically as this will require re-linking your application in case of a bug fix or new version.

• New DBINFO mode 505 has been added to return a list of all data sets which contain a specific index item.

• DBFIND and DBGET have been enhanced to support TurboIMAGE TPI modes.

• Native support for TurboIMAGE item types.
  The item types P (packed decimal), Z (zoned decimal), K (unsigned integer) and U (upper case string) are now fully supported as well as arbitrarily sized I (signed integer) type items.
  Eloquence is still fully compatible with previous versions. Item types not supported with previous versions are returned as type B (binary) or X (string).

• Database limits have been modified to follow or extend the current TurboIMAGE limits.

• New DBINFO mode 303 has been added to return sort item with path information. This supplements DBINFO mode 301 which does not support sort items.

• Index items can now be named identically to data items.

• The database utilities have been modified to support the new item types and the enhanced database limits.

• Forward logging has been integrated.
  This provides additional protection against system failure and speeds up recovery in case of a fatal problem.

• Performance enhancements. For some write intensive workloads the performance has increased by 25%.

• Improved reliability in case of abnormal server termination.

Known problems and limitations

• The forward logging functionality could require a large amount of disk space if used other than intended. Be sure to disable forward logging during dbimport or other unusual activities. This should not cause a problem during normal usage.

• On the Linux platform, eloqdb6 adds extra spaces to the command line as returned by the ps utility (#1062).

• dbutil currently does not allow to define a position in the schema, where a data set should be inserted. New data sets are always appended (with the exception of a master set which has a path to an existing detail).

• When restructuring a database, dbutil is blocked until the server operation is finshed. This could take considerable time, depending on the database size and operations required.

• On Red Hat Linux 8.0 and above, the IOthreads may not be visible in the process list (ps command). The option -m must be specified to output threads.

• On Red Hat Linux 8.0 and SuSE 8.1 reverse DNS lookup (setting NoDNS=0) can cause the eloqdb6 process to crash. This is related to glibc changes which uses excessive stack space for DNS lookup. By default the reverse lookup is now disabled.

• DBFIND modes 6 and 7 currently can not operate on master sets and search items but are restricted to index items.

• DBFIND modes 6 and 7 are currently not available from the Eloquence programming language.

Backwards compatibility

• Whenever you start the new B.07.00 database server or use any of the new B.07.00 database utilities such as dblogreset or dbfsck with an existing volume, the log volume format is changed, making the log volumes incompatible with earlier versions such as A.06.3x and B.06.32.

• In order to use your database environment with previous Eloquence versions you need to re-create the log volume(s) with the previous log file format. This requires at least Eloquence A.06.30 because earlier versions did not allow to recreate a log volume.

To restore the old log volume format, please follow the instructions below:

1. Run the B.07.00 dblogreset utility to make sure your log volumes are empty.

2. Delete the log volumes.

3. Use the A.06.3x or B.06.32 dbvolextend utility to re-create the log volumes with the old format:

   dbvolextend -t log /db/log/log01.vol
   dbvolextend -t log /db/log/log02.vol

4. Now you can use the volumes in your previous A.06.3x or B.06.32 environment.