.
Eloquence Database Binary Objects contact contact

Documentation / Eloquence dbobj

Eloquence Database Binary Objects

 
.
  Document revision: 2024-06-04
Refers to eqdbobj version: B.08.40


Table of contents


The eqdbobj library adds support for binary objects (blob) to the Eloquence database. Objects are maintained in the database in one or multiple records. The data is compressed as necessary.

A database object is identified by a unique 32-bit value specified by the application. Object sizes beyond 100 MB are supported. However, the object size has an impact on disk space and compression efforts and objects are expected to be substantially smaller.

Database objects are not accessed with the usual database calls but use the calls implemented in the eqdbobj library. Depending on object size, data is transparently compressed and split into multiple records as necessary. The eqdbobj library functions handle the locking and compression of object data.


The eqdbobj library does not depend on a specific database. The table to hold the object data can also be integrated into an existing database. Also, multiple tables may be used for different object categories.

Object data are stored in a detail data set with a structure as shown below. The data set name and item names are arbitrary. The item type and order is mandatory and verified by the eqdbobj library.

item type description
id K4 object id, must be a search item or index with the same name as the item
seq K2 record sequence, a zero indicates the first record
len K2 size of data in this record
data B4070 holds a chunk of object data

The library equally accepts the I item type (signed) as an alternative to K (unsigned).

The size of the "data" item may be customized to optimize for a specific object type between 64 and 5112 bytes. A size of 4070 minimizes overhead for large objects. A smaller data size may reduce disk space but will result in a performance impact for larger objects.

Please note that it is currently not supported to change the "data" item size unless the set is empty.

The database and data set is passed to eqdbobj calls. An error status DBOBJ_ERR_SET (-2) is returned if the data set does not exist, is not of type detail or does not fit the requirements. Details are output to stderr in case of an error if the verbose mode is set.


The eqdbobj.h header file specifies the eqdbobj library function prototypes and constants. The application includes the eqdbobj.h header file and links against the libeqdbobj.so library.

The eqdbobj library may be used with the Eloquence eqdb database client library and/or the Eloquence image3k library.

The following compile/link command may be used on Linux to build the btest3k example.

cc -Wall -I /opt/eloquence/8.4/include -o btest3k btest3k.c \
 -L /opt/eloquence/8.4/lib64 -l image3k -l eqdbobj \
 -Wl,-rpath=/opt/eloquence/8.4/lib64
HP-UX, IA64 32-bit:
cc -Ae +DD32 -I /opt/eloquence/8.4/include -o btest3k btest3k.c \
 -L /opt/eloquence/8.4/lib/hpux32 -l image3k -l eqdbobj


eqdb calls

The following calls are used with the eqdb library. Equivalent calls exist for use with the image3k library.

idb_obj_ctrl

idb_obj_ctrl is used to change settings for the eqdbobj library.
int idb_obj_ctrl(int mode, int nval, int *oval);
The mode argument specifies the parameter, the nval argument the new setting value and oval may be NULL or returns the previous value.

The following modes are supported.

  • mode 1 specifies a verbose logging level. A zero verbose level (default) disables any messages. A verbose level of one outputs additional details to stderr in case of a failure.

  • mode 2 specifies the compression level when storing objects. A zero value disables object compression (typically not advisable unless storing compressed data). Values 1 to 9 specify the compression level with increasingly more effort spent on compression. The default compression level is 1. An object is transparently compressed if it exceeds one data block.

Return value:

idb_obj_ctrl returns zero on success or one of following error codes.

DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_BADARG - bad argument value


idb_obj_fetch

idb_obj_fetch reads an object from the database.
int idb_obj_fetch(
   int dbid, const void *set, unsigned int id, size_t *len_p);
dbid specifies the database id as returned by idb_open, set specifies the data set name or number and id specifies the object id.

len_p is used to return the object size. This may be used to allocate a sufficiently sized buffer.

The object data is held in memory and retrieved with the idb_obj_getdata call. Any memory is released once the object data was copied or on the next idb_obj_fetch call.

Return value:

idb_obj_fetch returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_ERR_LOCK - database locking failed
DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_CORRUPT - object data corrupted
DBOBJ_ERR_ZLIB - zlib call failed


idb_obj_delete

idb_obj_delete is used to delete the specified object.
int idb_obj_delete(
   int dbid, const void *set, unsigned int id);
dbid specifies the database id as returned by idb_open, set specifies the data set name or number and id specifies the object id.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Return value:

idb_obj_delete returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_ERR_LOCK - database locking failed


idb_obj_store

idb_obj_store call is used to store an object in the database.
int idb_obj_store(
   int dbid, const void *set, int mode, unsigned int id);
dbid specifies the database id as returned by idb_open, set specifies the data set name or number and id specifies the object id value.

Depending on the mode value an object may, must or may not exist.

  • mode 1 adds the object if not already present. An error status is returned if the object id value already exists.

  • mode 2 updates an existing object. An error status is returned if the object id value does not exist.

  • mode 3 adds or updates the specified object
The object data must be passed with the idb_obj_putdata call before calling idb_obj_store. Any data held by idb_obj_putdata is released by the idb_obj_store call.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Return value:

idb_obj_store returns zero on success or one of following error codes.

DBOBJ_ERR_DB - unexpected db status
DBOBJ_ERR_SET - invalid or improper data set
DBOBJ_NOTFOUND - object not found
DBOBJ_EXISTS - object already exists
DBOBJ_ERR_LOCK - database locking failed
DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_NODATA - no data passed
DBOBJ_ERR_LIMIT - data size exceeds limit


idb_obj_putdata

idb_obj_putdata is used to pass data to the idb_obj_store call.
int idb_obj_putdata(
   int mode, const void *data, size_t data_sz);
Object data passed to idb_obj_putdata is kept in memory (in compressed form, as necessary) until the next idb_obj_store call. Data may be passed in multiple calls as necessary.

  • mode 0 releases previously allocated data. This may be used to release any memory occupied by a partially submitted object.

  • mode 1 is used to specify the first block of object data. It resets any previously submitted data object data.

  • mode 2 is used to specify an additional block of data.
data points to a buffer of data_sz bytes holding the object data. The data is copied to an internal holding area that is reset by the next idb_obj_store call.

Return value:

idb_obj_putdata returns zero on success or one of following error codes.

DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_NODATA - no data passed


idb_obj_getdata

idb_obj_getdata is used to return the object data read by the idb_obj_fetch call.
int idb_obj_getdata(
   int mode, void *buf, size_t buf_sz, size_t *len);
Object data read by idb_obj_fetch is kept in memory (in compressed form) until the object data is copied or the next idb_obj_fetch call.

  • mode 0 resets any held data. This may be used to release any memory occupied by a partially retrieved object. The remaining arguments are ignored.

  • mode 1 is used to obtain the next block of data.
buf points to a buffer of buf_sz bytes. This is a buffer provided by the application and used by idb_obj_getdata to return the next block of object data.

len is used to return the number of bytes copied to the buffer. A zero len value indicates that that all data was sucessfully retrieved. This will also release any internal memory.

idb_obj_getdata is called until a status is returned indicating a problem or a zero len value is returned.

The buf_sz argument may be zero (and the buf argument NULL) if all data was retrieved. For example, if the buffer size matches or exceeds the remaining object size. idb_obj_getdata will transfer the data and return a non-zero len value that is less or equal the buffer size. It is still required to call idb_obj_getdata again to verify the data checksum and release the associated memory.
The buffer size may not be zero if more data is present. A DBOBJ_ERR_BADARG status code is returned in this case.

Return value:

idb_obj_getdata returns zero on success or one of following error codes.

DBOBJ_ERR_MEMALLOC - memory allocation failed
DBOBJ_ERR_ZLIB - zlib call failed
DBOBJ_ERR_BADMODE - bad mode value
DBOBJ_ERR_BADARG - buffer length may not be zero
DBOBJ_ERR_CORRUPT - object data corrupted


idb_obj_msg

idb_obj_msg returns a pointer to a static error text for the status code passed as argument.
const char *idb_obj_msg(int st);
"unknown" is returned for unknown status codes.


The following data codes are used by the eqdbobj library:
DBOBJ_OK           |  0  |
DBOBJ_ERR_DB       | -1  | unexpected db status
DBOBJ_ERR_SET      | -2  | invalid or improper data set
DBOBJ_NOTFOUND     |  3  | object not found
DBOBJ_EXISTS       |  4  | object already exists
DBOBJ_ERR_LOCK     | -5  | database locking failed
DBOBJ_ERR_MEMALLOC | -6  | memory allocation failed
DBOBJ_ERR_CORRUPT  | -7  | object data inconsistent
DBOBJ_ERR_ZLIB     | -8  | zlib call failed
DBOBJ_ERR_BADMODE  | -9  | bad mode value
DBOBJ_ERR_BADARG   | -10 | bad argument value
DBOBJ_ERR_NODATA   |  11 | no data passed (sequence error)
DBOBJ_ERR_LIMIT    |  12 | data size exceeds limit


The following example obtains the object with the id 123 into an allocated buffer.
int dbid, status[10];
const char *set_name;
char *data;
size_t data_len, nbytes;

dbid = idb_open(db_name, NULL, 1, status);
dbstat("opening database", status);

idb_obj_ctrl(1, 1, NULL);   /* verbose */

set_name = "objd";
data = NULL;
cc = idb_obj_fetch(dbid, set_name, 123, &data_len);
if(cc)
   fprintf(stderr, "idb_obj_fetch: FAILED %d\n", cc);
else {
   data = malloc(data_len);
   if(data) {
      cc = idb_obj_getdata(1, data, data_len, &nbytes);
      if(!cc)
         cc = idb_obj_getdata(1, NULL, 0, &nbytes);
      if(cc)
         fprintf(stderr, "idb_obj_getdata: FAILED %d\n", cc);
   }
}
if(data && !cc)
   ...
free(data);
The following example adds an object with the id 123.
const char *str = "this is a test object";
cc = idb_obj_putdata(1, str, strlen(str));
if(!cc)
   cc = idb_obj_store(dbid, set_name, 1, 123);
if(cc)
   fprintf(stderr, "idb_obj_store: FAILED %d\n", cc);
The following example uses idb_obj_putdata() calls to pass the content of a file as an object.
static int read_file_obj(const char *fname)
{
   FILE *fp;
   char data[8192];
   size_t nbytes;
   int cc, mode;

   fp = fopen(fname, "rb");
   if(!fp) {
      perror(fname);
      goto failed;
   }

   mode = 1;
   while((nbytes = fread(data, 1, sizeof(data), fp)))
   {
      cc = idb_obj_putdata(mode, data, nbytes);
      if(cc) {
         fprintf(stderr, "idb_obj_putdata: FAILED %d\n", cc);
         goto failed;
      }
      mode = 2;
   }

   fclose(fp);
   return 0;

failed:
   idb_obj_putdata(0, NULL, 0);
   if(fp)
      fclose(fp);
   return -1;
}

if(   read_file_obj("123.xml") == 0
   && idb_obj_store(dbid, set_name, 1, 123) == 0)
   /* success */


image3k calls

The calls used with the image3k library are substantially similar to the eqdb calls but follow image3k conventions.

dbobjctrl

dbobjctrl is used to change settings for the eqdbobj library.
void dbobjctrl(
  int16_t *mode, int16_t *status, 
  int16_t *nval, int16_t *oval);
mode points to a 16-bit integer holding the mode value.
  • mode 1 specifies a verbose logging level. A zero verbose level (default) disables any messages. A verbose level of one outputs additional details to stderr in case of a failure.

  • mode 2 specifies the compression level when storing objects. A zero value disables object compression (typically not advisable unless storing compressed data). Values 1 to 9 specify the compression level with increasingly more effort spent on compression. Object data is transparently compressed if it exceeds one data block.

status points to a 16-bit integer returning the completion status. dbobjctrl returns a zero status on success or an error code.

nval points to a 16-bit integer with the new setting value. oval is NULL or points to a 16-bit integer to return the previous setting.

dbobjctrl is equivalent to the idb_obj_ctrl call.


dbobjfetch

dbobjfetch reads an object from the database.
void dbobjfetch(
   char *base, char *dset, int16_t *status, 
   uint32_t *id, uint32_t *len);
base specifies the database. dset specifies the data set name or number.

status points to a 16-bit integer returning the completion status. dbobjfetch returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

len is NULL or points to a 32-bit integer returning the object size. This may be used to allocate a sufficiently sized buffer.

The object data is held in memory and retrieved with the dbobjgetdata call. Any memory is released once the object data was copied or on the next dbobjfetch call.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found
dbobjfetch is equivalent to the idb_obj_fetch call.


dbobjdelete

dbobjdelete deletes an object from the database.
void dbobjdelete(
   char *base, char *dset, int16_t *status, uint32_t *id);
base specifies the database. dset specifies the data set name or number.

status points to a 16-bit integer returning the completion status. dbobjdelete returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found
dbobjdelete is equivalent to the idb_obj_delete call.


dbobjstore

dbobjstore call is used to store an object in the database.
void dbobjstore(
   char *base, char *dset, int16_t *mode,
   int16_t *status, uint32_t *id);
base specifies the database. dset specifies the data set name or number.

mode points to a 16-bit integer holding the mode value. Depending on the mode value an object may, must or may not exist.

  • mode 1 adds the object if not already present. An error status is returned if the object id value is already exists.

  • mode 2 updates an existing object. An error status is returned if the object id value does not exist.

  • mode 3 adds or updates the specified object

status points to a 16-bit integer returning the completion status. dbobjstore returns a zero status on success or an error code.

id points to a 32-bit integer holding the object id.

The object data must be passed with the dbobjputdata call before calling dbobjstore. Any data held by dbobjputdata is released by the dbobjstore call.

The database must be opened in a write mode or a DBOBJ_ERR_DB status is returned.

Relevant status codes:

DBOBJ_NOTFOUND   - object not found
DBOBJ_EXISTS     - object already exists
DBOBJ_ERR_NODATA - no data (sequence error)
dbobjstore is equivalent to the idb_obj_store call.


dbobjputdata

dbobjputdata call is used to pass the object data to the dbobjstore call.
void dbobjputdata(
  int16_t *mode, int16_t *status, 
  void *data, uint32_t *len);
Object data passed to dbobjputdata is kept in memory (in compressed form, as necessary) until the next dbobjstore call. Data may be passed in multiple calls as necessary.

mode points to a 16-bit integer holding the mode value.

  • mode 0 may used to reset any partially submitted data. The data and len arguments are ignored.

  • mode 1 is used to pass the first block of data and will reset any previous data.

  • mode 2 is used to pass a subsequent block of data.

status points to a 16-bit integer returning the completion status. dbobjputdata returns a zero status on success or an error code.

data points to a buffer and len to a 32-bit integer specifying the buffer size holding the object data.

dbobjputdata is equivalent to the idb_obj_putdata call.


dbobjgetdata

dbobjgetdata is used to return the object data read by the dbobjfetch call.
void dbobjgetdata(
  int16_t *mode, int16_t *status, 
  void *buf, uint32_t *buf_sz, uint32_t *len);
Object data read by dbobjfetch is kept in memory (in compressed form) until the object data is copied or the next dbobjfetch call.

mode points to a 16-bit integer holding the mode value.

  • mode 0 may be used to release any partially obtained data. The buf, buf_sz and len arguments are ignored.

  • mode 1 is used to return the next block of data.
status points to a 16-bit integer returning the completion status. dbobjgetdata returns a zero status on success or an error code.

buf points to a buffer and buf_sz to a 32-bit integer specifying the buffer size in bytes. This is a buffer provided by the application and used by dbobjgetdata to return the next block of object data.

len points to a 32-bit integer. It is used to return the number of bytes copied to the buffer. A zero len value indicates that all data was completely retrieved. This will also release any internal memory.

dbobjgetdata is equivalent to the idb_obj_getdata call.


The eqdbobj library does not depend on a specific database. The table to hold the object data can be integrated into existing databases. Also, multiple tables may be used for different objects.

The schema below uses a separate database dedicated to holding objetcs. The first variant uses a btree index to index the object id. The second variant uses a master/detail.

begin database objdb;

passwords:

items:

  obj-id, k4;
  obj-seq, k2;
  obj-len, k2;
  << block size is customizable >>
  obj-data, b4070;

iitems:

  obj-id = obj-id;

sets:

n: objd, d(/0);
e: obj-id,
   obj-seq,
   obj-len,
   obj-data;
i: obj-id;

end.
Alernative using chains:
sets:

n: obja, a(/0);
e: obj-id;

n: objd, d(/0);
e: obj-id(obja),
   obj-seq,
   obj-len,
   obj-data;


The btest3k example demonstrates the use of the eqdbobj library functionality. It uses the image3k database calls.

The btest3k application allows a simple way to store a file as a database object, extract a database object into a file or delete a database object.

usage: btest3k [opt] db set id {op} [file]
 opt: -v[vv] -z{0..9}
 op: add|del|upd|get|store
For example:

store the file file.xml as the database object id 123

btest3k -v objdb objd 123 store file.xml
extract the database object id 123 into file /tmp/file.xml
btest3k -v objdb objd 123 get /tmp/file.xml
delete the database object id 123
btest3k -v objdb objd 123 del
The source code of the btest3k example application is available here.


B.08.40
  • Integrated into B.08.40 release
PE83-2106150
  • Initial release (beta)


 
 
 
  Privacy | Webmaster | Terms of use | Impressum Revision: 2024-06-04  
  Copyright © 1995-2021 Marxmeier Software AG