Welcome to the Eloquence dbobj beta test.

• Overview
• Database
• Application use
• Installation instructions

• eqdb specific calls
  • idb_obj_ctrl
  • idb_obj_fetch
  • idb_obj_delete
  • idb_obj_store
  • idb_obj_putdata
  • idb_obj_getdata
  • idb_obj_msg
• Status codes
• Example

• image3k specific calls
  • dbobjctrl
  • dbobjfetch
  • dbobjdelete
  • dbobjstore
  • dbobjputdata
  • dbobjgetdata

• Example schema
• Example application
• Changes

A database object is identified by a unique 32-bit value specified by the application. Object sizes beyond 100 MB are supported. However, the object size has an impact on disk space and compression efforts and objects are expected to be substantially smaller.

Database objects are not accessed with the usual database calls but use the calls implemented in the eqdbobj library. Depending on object size, data is transparently compressed and split into multiple records as necessary. The eqdbobj library functions handle the locking and compression of object data.

The eqdbobj library is currently considered a contributed function as it has not yet been decided whether the eqdbobj library will be integrated in the Eloquence product in its current form. We appreciate your feedback.

Please note: Please keep in mind that this is a beta test version. This test release may not meet the release criteria for quality or performance and is intended for test and evaluation use.

Object data are stored in a detail data set with a structure as shown below. The data set name and item names are arbitrary. The item type and order is mandatory and verified by the eqdbobj library.

item	type	description
id	K4	object id, must be a search item or index with the same name as the item
seq	K2	record sequence, a zero indicates the first record
len	K2	size of data in this record
data	B4070	holds a chunk of object data

The library equally accepts the I item type (signed) as an alternative to K (unsigned).

The size of the "data" item may be customized to optimize for a specific object type between 64 and 5112 bytes. A size of 4070 minimizes overhead for large objects. A smaller data size may reduce disk space but will result in a performance impact for larger objects.

Please note that it is currently not supported to change the "data" item size unless the set is empty.

The database and data set is passed to eqdbobj calls. An error status DBOBJ_ERR_SET (-2) is returned if the data set does not exist, is not of type detail or does not fit the requirements. Details are output to stderr in case of an error if the verbose mode is set.

The eqdbobj library may be used with the Eloquence eqdb database client library and/or the Eloquence image3k library.

The following compile/link command may be used on Linux to build the btest3k example.

cc -Wall -I /opt/eloquence/8.3/include -o btest3k btest3k.c \
 -L /opt/eloquence/8.3/lib64 -l image3k -l eqdbobj \
 -Wl,-rpath=/opt/eloquence/8.3/lib64

cc -Ae +DD32 -I /opt/eloquence/8.3/include -o btest3k btest3k.c \
 -L /opt/eloquence/8.3/lib/hpux32 -l image3k -l eqdbobj

The eqdbobj library is currently available for the Linux x86_64 and HP-UX IA64 platform. Other platforms supported by Eloquence B.08.30 will follow.

The current eqdbobj beta version is available as Eloquence 8.30 patch PE83-2106150.

Download:
/download/B0830/patch/patch-1810/beta/

Installation (as superuser):

cd /opt/eloquence/8.3
tar xzf /path/to/eqdbobj-linux-x86_64.tar.gz

This will install the eqdbobj.h header file and the libeqdbobj.so shared library.

eqdb calls

idb_obj_ctrl

int idb_obj_ctrl(int mode, int nval, int *oval);

The following modes are supported.

• mode 1 specifies a verbose logging level. A zero verbose level (default) disables any messages. A verbose level of one outputs additional details to stderr in case of a failure.

• mode 2 specifies the compression level when storing objects. A zero value disables object compression (typically not advisable unless storing compressed data). Values 1 to 9 specify the compression level with increasingly more effort spent on compression. The default compression level is 1. An object is transparently compressed if it exceeds one data block.

Return value:

idb_obj_ctrl returns zero on success or one of following error codes.

DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_BADARG - bad argument value

idb_obj_fetch

int idb_obj_fetch(
   int dbid, const void *set, unsigned int id, size_t *len_p);

len_p is used to return the object size. This may be used to allocate a sufficiently sized buffer.

The object data is held in memory and retrieved with the idb_obj_getdata call. Any memory is released once the object data was copied or on the next idb_obj_fetch call.

Return value:

idb_obj_fetch returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_ERR_LOCK - database locking failed
DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_CORRUPT - object data corrupted
DBOBJ_ERR_ZLIB - zlib call failed

idb_obj_delete

int idb_obj_delete(
   int dbid, const void *set, unsigned int id);

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Return value:

idb_obj_delete returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_ERR_LOCK - database locking failed

idb_obj_store

int idb_obj_store(
   int dbid, const void *set, int mode, unsigned int id);

Depending on the mode value an object may, must or may not exist.

• mode 1 adds the object if not already present. An error status is returned if the object id value already exists.

• mode 2 updates an existing object. An error status is returned if the object id value does not exist.

• mode 3 adds or updates the specified object

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Return value:

idb_obj_store returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_EXISTS - object already exists
DBOBJ_ERR_LOCK - database locking failed
DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_NODATA - no data passed
DBOBJ_ERR_LIMIT - data size exceeds limit

idb_obj_putdata

int idb_obj_putdata(
   int mode, const void *data, size_t data_sz);

• mode 0 releases previously allocated data. This may be used to release any memory occupied by a partially submitted object.

• mode 1 is used to specify the first block of object data. It resets any previously submitted data object data.

• mode 2 is used to specify an additional block of data.

Return value:

idb_obj_putdata returns zero on success or one of following error codes.

DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_NODATA - no data passed

idb_obj_getdata

int idb_obj_getdata(
   int mode, void *buf, size_t buf_sz, size_t *len);

• mode 0 resets any held data. This may be used to release any memory occupied by a partially retrieved object. The remaining arguments are ignored.

• mode 1 is used to obtain the next block of data.

len is used to return the number of bytes copied to the buffer. A zero len value indicates that that all data was sucessfully retrieved. This will also release any internal memory.

idb_obj_getdata is called until a status is returned indicating a problem or a zero len value is returned.

The buf_sz argument may be zero (and the buf argument NULL) if all data was retrieved. For example, if the buffer size matches or exceeds the remaining object size. idb_obj_getdata will transfer the data and return a non-zero len value that is less or equal the buffer size. It is still required to call idb_obj_getdata again to verify the data checksum and release the associated memory.
The buffer size may not be zero if more data is present. A DBOBJ_ERR_BADARG status code is returned in this case.

Return value:

idb_obj_getdata returns zero on success or one of following error codes.

DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_BADARG - buffer length may not be zero
DBOBJ_ERR_CORRUPT - object data corrupted

idb_obj_msg

const char *idb_obj_msg(int st);

DBOBJ_OK           |  0  |
DBOBJ_ERR_DB       | -1  | unexpected db status
DBOBJ_ERR_SET      | -2  | invalid or improper data set
DBOBJ_NOTFOUND     |  3  | object not found
DBOBJ_EXISTS       |  4  | object already exists
DBOBJ_ERR_LOCK     | -5  | database locking failed
DBOBJ_ERR_MEMALLOC | -6  | memory allocation failed
DBOBJ_ERR_CORRUPT  | -7  | object data inconsistent
DBOBJ_ERR_ZLIB     | -8  | zlib call failed
DBOBJ_ERR_BADMODE  | -9  | bad mode value
DBOBJ_ERR_BADARG   | -10 | bad argument value
DBOBJ_ERR_NODATA   |  11 | no data passed (sequence error)
DBOBJ_ERR_LIMIT    |  12 | data size exceeds limit

int dbid, status[10];
const char *set_name;
char *data;
size_t data_len, nbytes;

dbid = idb_open(db_name, NULL, 1, status);
dbstat("opening database", status);

idb_obj_ctrl(1, 1, NULL);   /* verbose */

set_name = "objd";
data = NULL;
cc = idb_obj_fetch(dbid, set_name, 123, &data_len);
if(cc)
   fprintf(stderr, "idb_obj_fetch: FAILED %d\n", cc);
else {
   data = malloc(data_len);
   if(data) {
      cc = idb_obj_getdata(1, data, data_len, &nbytes);
      if(!cc)
         cc = idb_obj_getdata(1, NULL, 0, &nbytes);
      if(cc)
         fprintf(stderr, "idb_obj_getdata: FAILED %d\n", cc);
   }
}
if(data && !cc)
   ...
free(data);

const char *str = "this is a test object";
cc = idb_obj_putdata(1, str, strlen(str));
if(!cc)
   cc = idb_obj_store(dbid, set_name, 1, 123);
if(cc)
   fprintf(stderr, "idb_obj_store: FAILED %d\n", cc);

static int read_file_obj(const char *fname)
{
   FILE *fp;
   char data[8192];
   size_t nbytes;
   int cc, mode;

   fp = fopen(fname, "rb");
   if(!fp) {
      perror(fname);
      goto failed;
   }

   mode = 1;
   while((nbytes = fread(data, 1, sizeof(data), fp)))
   {
      cc = idb_obj_putdata(mode, data, nbytes);
      if(cc) {
         fprintf(stderr, "idb_obj_putdata: FAILED %d\n", cc);
         goto failed;
      }
      mode = 2;
   }

   fclose(fp);
   return 0;

failed:
   idb_obj_putdata(0, NULL, 0);
   if(fp)
      fclose(fp);
   return -1;
}

if(   read_file_obj("123.xml") == 0
   && idb_obj_store(dbid, set_name, 1, 123) == 0)
   /* success */

image3k calls

dbobjctrl

void dbobjctrl(
  int16_t *mode, int16_t *status, 
  int16_t *nval, int16_t *oval);

• mode 1 specifies a verbose logging level. A zero verbose level (default) disables any messages. A verbose level of one outputs additional details to stderr in case of a failure.

• mode 2 specifies the compression level when storing objects. A zero value disables object compression (typically not advisable unless storing compressed data). Values 1 to 9 specify the compression level with increasingly more effort spent on compression. Object data is transparently compressed if it exceeds one data block.

status points to a 16-bit integer returning the completion status. dbobjctrl returns a zero status on success or an error code.

nval points to a 16-bit integer with the new setting value. oval is NULL or points to a 16-bit integer to return the previous setting.

dbobjctrl is equivalent to the idb_obj_ctrl call.

dbobjfetch

void dbobjfetch(
   char *base, char *dset, int16_t *status, 
   uint32_t *id, uint32_t *len);

status points to a 16-bit integer returning the completion status. dbobjfetch returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

len is NULL or points to a 32-bit integer returning the object size. This may be used to allocate a sufficiently sized buffer.

The object data is held in memory and retrieved with the dbobjgetdata call. Any memory is released once the object data was copied or on the next dbobjfetch call.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found

dbobjdelete

void dbobjdelete(
   char *base, char *dset, int16_t *status, uint32_t *id);

status points to a 16-bit integer returning the completion status. dbobjdelete returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found

dbobjstore

void dbobjstore(
   char *base, char *dset, int16_t *mode,
   int16_t *status, uint32_t *id);

mode points to a 16-bit integer holding the mode value. Depending on the mode value an object may, must or may not exist.

• mode 1 adds the object if not already present. An error status is returned if the object id value is already exists.

• mode 2 updates an existing object. An error status is returned if the object id value does not exist.

• mode 3 adds or updates the specified object

status points to a 16-bit integer returning the completion status. dbobjstore returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

The object data must be passed with the dbobjputdata call before calling dbobjstore. Any data held by dbobjputdata is released by the dbobjstore call.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found
DBOBJ_EXISTS     - object already exists
DBOBJ_ERR_NODATA - no data (sequence error)

dbobjputdata

void dbobjputdata(
  int16_t *mode, int16_t *status, 
  void *data, uint32_t *len);

mode points to a 16-bit integer holding the mode value.

• mode 0 may used to reset any partially submitted data. The data and len arguments are ignored.

• mode 1 is used to pass the first block of data and will reset any previous data.

• mode 2 is used to pass a subsequent block of data.

status points to a 16-bit integer returning the completion status. dbobjputdata returns a zero status on success or an error code.

data points to a buffer and len to a 32-bit integer specifying the buffer size holding the object data.

dbobjputdata is equivalent to the idb_obj_putdata call.

dbobjgetdata

void dbobjgetdata(
  int16_t *mode, int16_t *status, 
  void *buf, uint32_t *buf_sz, uint32_t *len);

mode points to a 16-bit integer holding the mode value.

• mode 0 may be used to release any partially obtained data. The buf, buf_sz and len arguments are ignored.

• mode 1 is used to return the next block of data.

buf points to a buffer and buf_sz to a 32-bit integer specifying the buffer size in bytes. This is a buffer provided by the application and used by dbobjgetdata to return the next block of object data.

len points to a 32-bit integer. It is used to return the number of bytes copied to the buffer. A zero len value indicates that all data was completely retrieved. This will also release any internal memory.

dbobjgetdata is equivalent to the idb_obj_getdata call.

The schema below uses a separate database dedicated to holding objetcs. The first variant uses a btree index to index the object id. The second variant uses a master/detail.

begin database objdb;

passwords:

items:

  obj-id, k4;
  obj-seq, k2;
  obj-len, k2;
  << block size is customizable >>
  obj-data, b4070;

iitems:

  obj-id = obj-id;

sets:

n: objd, d(/0);
e: obj-id,
   obj-seq,
   obj-len,
   obj-data;
i: obj-id;

end.

sets:

n: obja, a(/0);
e: obj-id;

n: objd, d(/0);
e: obj-id(obja),
   obj-seq,
   obj-len,
   obj-data;

The btest3k application allows a simple way to store a file as a database object, extract a database object into a file or delete a database object.

usage: btest3k [opt] db set id {op} [file]
 opt: -v[vv] -z{0..9}
 op: add|del|upd|get|store

store the file file.xml as the database object id 123

btest3k -v objdb objd 123 store file.xml

btest3k -v objdb objd 123 get /tmp/file.xml

btest3k -v objdb objd 123 del

• eqdbobj beta version 1.
• Initial release

We appreciate your feedback. Please send any feedback to support@marxmeier.com.

Enjoy!
The Eloquence Team