jx_database is a persistent database for keeping track of a set of json objects, each indexed by a unique key and described by a set of arbitrary name value pairs. More...

#include "jx.h"

Functions
struct jx_database *	jx_database_create (const char *logdir)
	Create a new database, recovering state from disk if available. More...

void	jx_database_insert (struct jx_database db, const char key, struct jx *j)
	Insert or update an object into the database. More...

struct jx *	jx_database_lookup (struct jx_database db, const char key)
	Look up an object in the database. More...

struct jx *	jx_database_remove (struct jx_database db, const char key)
	Remove an object from the database. More...

void	jx_database_firstkey (struct jx_database *db)
	Begin iteration over all keys in the database. More...

int	jx_database_nextkey (struct jx_database db, char key, struct jx *j)
	Continue iteration over the database. More...

Detailed Description

jx_database is a persistent database for keeping track of a set of json objects, each indexed by a unique key and described by a set of arbitrary name value pairs.

The current state of the database is kept in memory for fast queries, while a history of all modifications is logged to disk to enable recovering the state of the database at any point in the past.

The history function is secondary to the online access function, so as a general rule, errors in accessing the on-disk history are ignored in order to keep the online access going.

The on-disk history works as follows:

For each day of the year, a checkpoint file is created which is an exact snapshot of the table at the beginning of the day. For updates received during that day, a corresponding log file records the individual changes.

The state of the table at any time can be obtained by starting with the proper daily checkpoint, then playing the log of updates until the desired time is reached. This could be used to creating arbitrary snapshots in time, or for running more complex queries that show the change of a single entity over time.

The log directory is broken down by year and day-of-year, so that each checkpoint file is named DIR/YEAR/DAY.ckpt and the corresponding log file is named DIR/YEAR/DAY.log

The checkpoint file is simply a json object containing the keys and values of all the objects in the database.

The log file consists of a series of entries, each one a json array in the following formats:

T [time]               - Indicates the current time in Unix epoch format.
C [key] [object]       - Create a new object with the given key.
D [key] [object]       - Delete an object with the given key.
U [key] [name] [value] - Update a named property with a new value.
R [key] [name]         - Remove a property with the given name.

In the examples above, time is a JSON integer,
key and name are JSON strings, object is a JSON object,
and value can be any JSON value.

As of 2012, with approx 300 entities reporting to the catalog, each day results in 20MB of log data and 150KB of checkpoint data, totalling under 8GB data per year.

Function Documentation

struct jx_database* jx_database_create ( const char * logdir )

Create a new database, recovering state from disk if available.

Parameters

logdir A directory to contain the database on disk. If it does not exist, it will be created. If null, no disk storage will be used.

Returns: A pointer to a newly created history table.

void jx_database_insert	(	struct jx_database *	db,
		const char *	key,
		struct jx *	j
	)

Insert or update an object into the database.

If an object with the same primary key exists in the database, it will generate update (U) records in the log, otherwise a create (C) record is generated against the original object.

Parameters

db	The database to access.
key	The primary key to associate with the object.
j	The object in the form of an jx expression.

struct jx* jx_database_lookup	(	struct jx_database *	db,
		const char *	key
	)

Look up an object in the database.

Parameters

db	The database to access.
key	The primary key of the desired object.

Returns: A pointer to the matching object, which should not be modified or deleted. Returns null if no match found.

struct jx* jx_database_remove	(	struct jx_database *	db,
		const char *	key
	)

Remove an object from the database.

Causes a delete (D) record to be generated in the log.

Parameters

db	The database to access.
key	The primary key of the desired object.

Returns: A pointer to the matching object, which should be discarded with jx_delete when done Returns null if no match found.

void jx_database_firstkey ( struct jx_database * db )

Begin iteration over all keys in the database.

This function begins a new iteration over the database. allowing you to visit every primary key in the database. Next, invoke jx_database_nextkey to retrieve each value in order.

Parameters

db	The database to access.

int jx_database_nextkey	(	struct jx_database *	db,
		char **	key,
		struct jx **	j
	)

Continue iteration over the database.

This function returns the next primary key and object in the iteration.

Parameters

db	The database to access.
key	A pointer to an unset char pointer, which will be made to point to the primary key.
j	A pointer to an unset jx pointer, which will be made to point to the next object.

Returns: Zero if there are no more elements to visit, non-zero otherwise.

Functions

Detailed Description

Function Documentation