The Eloquence A.06.00 Database

2 Introduction

The Eloquence A.06.00 Database

This chapter describes the operation and architecture of the Eloquence database.

NOTE: The new database is not binary compatible to the previous implementation and the database must be transferred by using dbexport/dbimport.

Introduction

When the original Eloquence database was implemented, it was intended as a compatible replacement of the HP260 Image database. With Eloquence release A.03.xx, index operations were added.

Since then, the amount of data stored in Eloquence databases has grown tremendously. While a typical database in 1990 had a size of less then 100 MB, Elquence databases today reach sizes of 4 GB and above. In addition Eloquence databases are used in a network environment.

It became evident, that the previous Eloquence database architecture would not allow to cope with the request for operating on ever increasing amounts of online data. Therefore Eloquence A.06.00 includes a new database system.

The main objectives for the new implementation were:

Compatibility: The new database is transparent to existing Eloquence programs.
Improved database security. This includes both, access control and protection against corruption due to program or system failures.
Better performance for large databases.
Transaction logging and recovery
Databases are kept in a database environment which is built of volumes. There are no longer hundreds of database related files.
A portable and extensible architecture

The new Eloquence database can be used as a building brick to efficiently realize relational and hierarchical database strcutures. In the sections below, we provide an overview on the Eloquence database architecture.

Database architecture

In former Eloquence releases, the database was implemented as a shared process. Each Eloquence process contained a common part of the database engine, while some central shared memory was used to coordinate database activities system wide.

The new Eloquence database now utilizes one process per database environment. This process controls and performs all database operations. This provides better performance, since less system resources are used (for the client processes), less inter-process synchronization is required and the database server process can use dedicated system resources allocated for it.

The new Eloquence database uses a layered architecture. The Diagram below shows the major database layers:

	---------------------------------------
	| Image      | Catalog | (SQL)        |  API Layer
	---------------------------------------
	| FixRec  | VarRec  | BTree  | . . .  |  Record Layer
	---------------------------------------
	| Node Management                     |  Node Layer
	---------------------------------------
	| Buffer Cache                        |  Buffer Layer
	---------------------------------------
	| Volume Management                   |  Storage Layer
	---------------------------------------

The storage layer

All databases are maintained in a database environment. A database environment consists of a group of related files (called database volumes) which actually contain the data and a configuration file. The storage layer is responsible for volume management and block allocation.

---------------------------------------------
| Storage Layer                             |   
|                                           |   
---------------------------------------------  
      |               |               |
-------------   -------------   ------------- 
| Volume #1 |   | Volume #2 |   | Volume #3 | 
| /mnt1/vol |   | /mnt2/vol |   | /mnt3/vol | 
|           |   |           |   |           | 
-------------   -------------   -------------

Each database volume contains a volume header, which describes the volume and its properties. It also has a block allocation map, which is used to locate available blocks in this volume. Below that are data blocks which can be allocated in groups for the various database objects.

A database environment can have up to 255 volumes, each volume up to 128 GB (this depends on the underlying operating system). A database volume can either be allocated with a fixed size or can grow by a specified amount when additional space is required. For example, you can have a volume which starts with a size of 100 MB, extends by 16 MB until it reaches a total size of 200 MB.

The Buffer Cache

The buffer cache is a memory area consisting of 8K blocks ("pages"). Each page is associated with a specific location on disc. A group of pages can be linked to a cluster to hold a consecutive disc area. This allows to read or write a group of related disc blocks in one operation.

Each buffer cache page consists of a buffer header, holding status and link information for this page and the buffer memory, which holds the data from/for a disk location.

       Buffer Header    Buffer Memory
       -----------      ------------------
       | Block   |      | Contents of    |  
       | #3      |----->| Block #3       |  
       |         |      |                |  
       |         |      |                |  
       -----------      ------------------  
                           ^                
       Database volume     |       
       ------------------------------------------------- 
       | Block | Block | Block | Block | ...   | Block |   
       | #1    | #2    | #3    | #4    |       | #n    | 
       -------------------------------------------------

The buffer memory is either allocated on server startup. Modified pages are written back to the disc, when either additional buffer space is required or due to database consistency requirements.

Node Management

A Database Node is a generic database object. Database resources are allocated to database nodes. Each node has an associated list of blocks allocated to it, besides that, it is free to do what it wants. A database node is like a file. You know its dimensions, but you don't know what it contains, unless you have a look inside.

          -------------   --------------
          | Node      |-->| Allocated  |   
          |           |   | Blocks     |
          -------------   --------------

Nodes are allocated dynamically, unlimited in number.

Record Layer

The Record layer implements different logical node types. It is responsible for the maintenance of logical records and is a building brick for the high level APIs (such as Image).

The following node types are currently available:

Fixed records (FixRec): Fixed records are identified by a constant record address (a 32 bit number) which is used to identify the block address in the volume which actually contains the data. Fixed records are allocated in clusters of up to 64k to minimize disc space overhead.

------------   --------------    ----------------
| Node     |-->| Block list |--->| Allocated    |   
------------   --------------    | blocks       |
               --------------    |              |
               | Free list  |    |              |
               --------------    ----------------

Fixed records use a list of allocated Blocks to map a record address to a data block (actually a cluster) in a database volume. The Free List is used to keep track of available record numbers.
BTree: This is an index. It contains an association of key values and data address.
SysCat: This is a special node type used internally to manage database meta data. For example, the former ROOT file is now just a collection of records in various system nodes. For example, a node "SYSTABLES" contains a list of all available tables in all databases.

API Layer

The database directory (called catalog) services is an internal API, which is used to maintain metadata on the database. For example, it associates a table name with a node id, and an index with a node id and the table and the indices. In former Eloquence database, database meta data were kept in a ROOT file.

There is no ROOT file anymore and you can have any number of databases in a database environment. All database meta information is now stored in catalog tables in the database environment. Of course, this kind of information is protected against modifications via Eloquence application programs.

If you think of the old ROOT file as a collection of various information specific to a single database, then this will become obvious:

Passwords: Database access passwords
Items: Data Items
Index Items: Index Items
Data sets: Database tables

All these informations are now simply stored in catalog tables inside the database.

For example:

SYSDB: List of databases contained in the database environment
SYSTABLES: List of tables
SYSCOLUMNS: Column (item) definitions
SYSINDICES: Index definitions

The Image API provides the Image functions such as DBPUT or DBGET. The new Image API additionally provides functionality for transaction management.

The new database is not limited to the Image API. While not included with Eloquence, another API such as SQL could be used to access the data simultaneously.

Compatibility

The new Eloquence database will be highly compatible with the previous implementation. From a programming point of view, it does not matter, how a DBPUT is implemented internally, as long as it works as before.

However database utilities such as SCHEMA, DBCREATE or DBUTIL are affected:

For example, the schema processor will no longer create a ROOT file, but it will send the database structural information to the database srever.

The new database is not binary compatible to the previous implementation and the database must be transferred by using dbexport/dbimport.

Eloquence version A.06.00 is able to access previous Eloquence databases by the usage of the eloqdb5 server.

Eloquence Database Manual - 19 DEC 2002