.
contact contact


Database Encryption

 
.
  Contents:

Overview

The Eloquence database encryption feature allows to configure dataset fields with sensitive contents for encrypted storage. It helps protecting database volume files, forward logs, as well as dbstore files and backups against unauthorized access to sensitive data.

Using database encryption does NOT protect against weak database passwords or incorrect database security settings (like user, group, member or dataset privileges). It is also NOT suitable to protect client connections across insecure networks, for example.

When using encryption, the database server maintains a list of one or more data encryption keys per database. Each dataset entry containing encrypted items uses one of those data encryption keys to protect the item contents. New or updated entries always use the most recently created data key. This allows "rolling data key changes" without the need for a database downtime to re-encrypt all data.

The data encryption keys are stored (in encrypted form) inside the database volume files, protected by so-called "master keys".

Master keys are created outside the database server and stored in special text files, protected by individual passphrases. The master key(s) must be submitted to the database server so that it can decrypt its internal data encryption keys and use those for accessing encrypted dataset fields.

The master key submission has to be performed after each start of the database server. Without the master keys, the database server is unable to read or write encrypted items.

Similar to the database server, tools like dbbexp or fwaudit, which access encrypted information directly also need to be provided the master keys before they are able to process encrypted dataset fields. On the other hand, utilities like dbrecover or dbrepl do not need to know the master keys, as their processing does not involve "unpacking" of encrypted item content.

Master keys may optionally be created as multi-part keys. With multi-part keys, all parts have to be submitted to the database server to be active. Using multi-part keys allows authorization schemes where keys are spread across independent key owners and all the involved parties are required for successful key submission.


Configuration

This section describes how to configure dataset fields for encrypted storage.

Please note that after configuring one or more dataset fields for encryption, the database server always needs the associated master key(s) before being able to read or write encrypted data items. It is therefore crucial that you carefully keep the key files and their passphrases. Loss of the keys or their passphrases will result in the irrecoverable loss of the associated encrypted database (and forward log) contents.

Step 1: generate master key with dbkeyutil
Before the database server can create and store any data encryption keys, it needs to be provided with a master key. You generate such a master key with dbkeyutil.

Here is an example:

$ dbkeyutil keygen mykey
Enter passphrase for mykey:
Confirm passphrase:
The master key is generated, protected using the passphrase and stored in the local text file eqdb.key by default. You may use the -k option of dbkeyutil to specify a different keyring file.

Note: Please make a copy of the key file and document the passphrase. Place this in a secure location. Loss of either the key file or the passphrase will result in losing access to encrypted data.

Step 2: submit master key to the database server with dbkeyutil
The dbkeyutil submit command is used to submit the master key to the running database server process. The utility prompts for the passphrase required to access the key in your keyring file (eqdb.key by default). It will require dba or operator privileges on the database server side.

Here is an example:

$ dbkeyutil -u dba submit mykey
Enter passphrase for mykey:
Passphrase is valid
Master key submitted successfully
You may use the -h and -s options or the EQ_DBSERVER environment variable to specify host and port.

The dbkeyutil status command may be used to review the list of active master keys on the database server, as shown in the example below:

$ dbkeyutil status
idx  master key checksum              stat type     ts
---- -------------------------------- ---- -------- -------------------
1    9f452bd41b7d669d1e2da1a4fdea8522 ACTV AES 128  2009-11-25 15:23:50
Step 3: use dbutil to create data encryption keys
Before you can configure selected dataset fields for encrypted storage, the database server needs to have the internal data encryption keys for the respective databases.

Use dbutil to trigger the creation of data encryption keys. These keys are stored inside the database volume files, protected by a master key. You reference the desired master key in your dbutil command by specifying its checksum (available from your keyring file or dbkeyutil status output).

Here is an example:

$ dbutil -v -

ELOQUENCE DBUTIL (C) Copyright 2010 Marxmeier Software AG (B.08.10)
Processing script ...

database "mydb";
create encryption key using "9f452bd41b7d669d1e2da1a4fdea8522";
exit;
done
You may optionally specify a key type (like AES128 or AES256) for the creation of the data encryption keys. The default is AES (AES 128 bit).

Note that Step 3 and Step 4 may be combined in a single dbutil session.

Step 4: use dbutil to configure dataset fields as encrypted
After preparing the database server with master keys and data encryption keys for the database(s) of interest, you may now configure selected sets or items for encrypted storage.

Use dbutil to apply the changes. It will restructure the associated sets.

Here is an example:

$ dbutil -v -

ELOQUENCE DBUTIL (C) Copyright 2010 Marxmeier Software AG (B.08.10)
Processing script ...

database "mydb";
change set "invoices" set encrypted;
change item "order-date" in "orders" set encrypted;
change item "price" set encrypted;
exit;
Checking database consistency ...
Consistency check completed successfully

Database restructure analysis:
 PRODUCTS
  * Record reorganized due encryption change
 INVOICES
  * Record reorganized due encryption change
 ORDERS
  * Record reorganized due encryption change
Data restructure process required.

Uploading modified schema ...
Restructuring database ...
done
The above example configured all items of set INVOICES, a single item of dataset ORDERS and the item PRICE in all sets to become encrypted.

Note that Step 3 and Step 4 may be combined in a single dbutil session.

Please note that after configuring encrypted items, the database server must be provided with the valid master key after every server restart. Otherwise it will not be able to access the contents of encrypted items.

To remove item encryption, you may use the above dbutil commands with an "unset encrypted" clause. However, this also will only be possible if the server has been provided with valid master key via dbkeyutil submit.


Operation

When using databases with encrypted items, it is neccessary to submit the respective master keys to the database server on every restart. Otherwise the server is unable to access the encrypted content and will operate in a "degraded mode" where only dbopen mode 8 is permitted, reading content of encrypted items is impossible, and (depending on schema relationship of the encrypted items) chained or indexed access may also be impacted.

When using databases with encrypted items in conjunction with replication, you also need to submit the respective master keys to the slave server(s) if or when access to the slave server databases is desired. The data replication itself also works without master keys being present in the slave server(s).

Database auditing is also affected by the use of encrypted items. Access to encrypted data is only possible when supplying the relevant master keys in the fwaudit invocation. If the contents of encrypted items are not of interest, fwaudit may also be used without specifying any master keys; it will then return blank or zero values for the inaccessible encrypted items.

Please note that the fwutil library does not currently support the use of master keys. Consequently, encrypted information is not available.

Note that you may need "historic" master keys when attempting to perform database auditing using older forward logs or when trying to access databases restored from older backups. It is thus advisable to carefully document and archive the "history" of master keys, especially when changing master keys in periodic intervals for security reasons.

To change the passphrases of master keys (periodically), you may use the dbkeyutil chpass command. Note that archived copies of the keyfile will still need the previous passphrases to be used.

To change master keys for specific databases (periodically), you may use the "change all encryption keys using ..." command in dbutil. Note that older database backups and forward logs will still need the previous master keys to be used.

To change the data encryption keys for specific databases (periodically), you may use the "create encryption key" command in dbutil. Note that this only affects the encryption of database entries added or updated subsequently. To also re-encrypt older entries and retire older data encryption keys from the database, you may use the dbctl dbkeyupdate command (which may run for a while, consuming CPU and disk I/O resources).

Note that changing or retiring data encryption keys is independent of any changes to the associated master keys or their passphrases.


Backwards compatibility

Configuring and using item encryption requires an internal database catalog table "syskey" that did not exist in Eloquence versions before B.08.10. For databases created on older Eloquence versions, it is necessary to upgrade the catalog tables with the "upgrade database" command in dbutil before using encryption.

Encrypted databases are not compatible to previous Eloquence versions. Attempting to access data or an index that holds encrypted data with a previous Eloquence version is likely to result in a server abort or corrupted data. When encrypting data, the record size and layout differs and previous Eloquence versions are currently not able to detect this.

Databases are fully compatible to with previous versions when no encryption is used.


dbkeyutil utility

dbkeyutil is an Eloquence utility that is used to create, maintain and upload database master keys. A master key is used by the Eloquence database to encrypt/decrypt the actual data encryption keys. Without the appropriate master key the database server is unable to access encrypted content. The master keys are maintained outside the database server volume files.

Find more information in the dbkeyutil document.


dbutil script syntax

All commands below are only valid inside a database context:
DATABASE "example";
The following command upgrades the database catalog for databases created on older Eloquence versions (see "Compatibility" above):
UPGRADE DATABASE;
To use encryption a data encryption key must be generated for the database. As data encryption keys are encrypted with a master key a master key must be uploaded to the server before creating a new encryption key.

The following command is used to create a data encryption key:

CREATE ENCRYPTION KEY [ ("type") ] 
USING [MASTER KEY] "master key";
Type specifies the data encryption algorithm associated with this key. The default encryption algorithm is AES.

Supported encryption types are:

  • AES - AES 128 bit
  • AES128 - AES 128 bit
  • AES256 - AES 256 bit
The master key argument specifies the checksum of the master key that is used to encrypt the data encryption key. The master key must have been submitted to the server before. The keywords "MASTER KEY" are optional and may be omitted.

For example:

CREATE ENCRYPTION KEY("AES256")
USING MASTER KEY "330d6de453136a93fd99c1638fd645fd";
This creates a new data encryption key with the encryption algorithm AES256 and encrypts it with the designated master key.

The following command may be used to remove data encryption keys if the database does not use encrypted information. It fails if encryption is used in the database.

DELETE ALL ENCRYPTION KEYS;
Please notice that this command must be performed in a separate session after removing encrpytion from a database.

Data encryption keys may be changed to use a new master key. This may be used to replace a master key. To succeed, both the previous and the new master keys need to be present in the server.

The following command may be used to assign all data encryption keys of a database to the designated master key:

CHANGE ALL ENCRYPTION KEYS USING [MASTER KEY] "master key";
The master key argument specifies the checksum of the master key that is used to encrypt the data encryption key. The keywords "MASTER KEY" are optional and may be omitted.

The commands below may be used to designate a field as encrypted or unencrypted:

CHANGE ITEM item-name [IN set-name] SET ENCRYPTED;
CHANGE ITEM item-name [IN set-name] UNSET ENCRYPTED;
The item-name argument specifies the item. The set-name argument specifies the name of a data set. If no data set is specified, the item is changed in all data sets. If a data set is specified, the field is only changed in this data set.

Please note: encryption status is maintained for each field (item in a data set). When adding a field to a data set, it defaults to unencrypted.

The following commands may be used to designate all fields in a data set as encrypted or unencrypted:

CHANGE SET set-name SET ENCRYPTED;
CHANGE SET set-name UNSET ENCRYPTED;
The set-name argument specifies a set name. Then all items in this set are changed accordingly.

The SET or UNSET ENCRYPTED may also be used in a change set context (enclosed in brackets).

For example:

CHANGE SET customers {
  UNSET ENCRYPTED;
  CHANGE ITEM turnover SET ENCRYPTED;
}
This would mark all fields in the customers record as unencrypted and then change the turnover field to be maintained in encrypted form.


dbkeyupdate command

The dbkeyupdate command in dbctl provides a means to retire old data encryption keys from a database. It triggers a scan for the specified database covering all datasets and indexes using encrypted items. It searches for data encrypted with an encryption key older than a given key ID or key creation date. If necessary it re-encrypts any entries found (using the currently active key). It finally discards the old and now unused data encryption keys from the catalog.

Find more information in the dbkeyupdate document.


database catalog table

The data encryption keys configured with dbutil script commands are maintained in the catalog table "syskey" for the respective database. They may be reviewed with dbdumpcat as needed (e.g. to identify the associated master key required, or help with other troubleshooting).

For example:

$ dbdumpcat -t 13 mydb

---------------------------------------------------------------------------------
#13 syskey (1 entries)
---------------------------------------------------------------------------------
|kid |tableid|type|tskey     |key             |kcheck          |mcheck          |
---------------------------------------------------------------------------------
|1   |0      |2   |2009-11-25|16:4b50922ca12f |16:e6da66d666f1 |16:9f452bd41b7d |
---------------------------------------------------------------------------------
The "kid" column shows the key ID. The "tskey" column shows the data encryption key's creation timestamp (in seconds since 1970, if using the -n option). The "mcheck" checksum shows the leading part of associated master key's checksum, which may be used to identify the respective master key in the keyfile.

Use the dbdumpcat -n option to show the "mcheck" column in full length.

They "key" and "kcheck" columns contain the data encryption key and a checksum. The data encryption key is, of course, stored encrypted with its associated master key.


Risks and pitfalls

Loss of keys or passphrases
After configuring one or more dataset fields for encryption, the database server always needs the associated master key(s) before being able to read or write encrypted data items. It is therefore crucial that you carefully keep the key files and their passphrases. Loss of the master keys or their passphrases will result in the irrecoverable loss of the associated encrypted database (and forward log) contents.

Please keep in mind that the master key and its pass phrase is required as long as backups need to be accessible.

Encryption overhead
Using data encryption changes the record size and layout. To minimize the encryption overhead, all encrypted fields are grouped together. As encryption uses a block size of 16 bytes the combined size of all encrypted fields is enlarged to be a multiple of 16 bytes. In addition, 4 bytes are used to record the id of the encryption key used to encrypt the data. This allows to use multiple generations of encryption keys. In addition, another 4 bytes are used to maintain an individual initialization value for this record ("salt") to ensure encrypted data have a different binary representation for identical values. Consequently, using encrypted data will add an overhead of 8 to 22 bytes per record, depending on the record layout.

Enabling encryption requires additional CPU resources, depending on the amount of encrypted items and the frequency records holding encrypted data are accessed. In our testing we encountered about 20% additional CPU resources required in a benchmark when encrypting the entire database.


 
 
.
 
 
  Privacy | Webmaster | Terms of use | Impressum Revision:  2010-07-29  
  Copyright © 2010 Marxmeier Software AG