cctools
Functions
chirp_matrix.h File Reference

Stores very large distributed matrices. More...

#include "chirp_types.h"

Go to the source code of this file.

Functions

struct chirp_matrix * chirp_matrix_create (const char *host, const char *path, int width, int height, int element_size, int nhosts, time_t stoptime)
 Create a new distributed matrix. More...
 
struct chirp_matrix * chirp_matrix_open (const char *host, const char *path, time_t stoptime)
 Open an existing matrix. More...
 
int chirp_matrix_get_row (struct chirp_matrix *matrix, int y, void *data, time_t stoptime)
 Get all values in a row. More...
 
int chirp_matrix_set_row (struct chirp_matrix *matrix, int y, const void *data, time_t stoptime)
 Set all values in a row. More...
 
int chirp_matrix_get_col (struct chirp_matrix *matrix, int x, void *data, time_t stoptime)
 Get all values in a column. More...
 
int chirp_matrix_set_col (struct chirp_matrix *matrix, int x, const void *data, time_t stoptime)
 Set all values in a column. More...
 
int chirp_matrix_get_range (struct chirp_matrix *matrix, int x, int y, int width, int height, void *data, time_t stoptime)
 Get a range of data. More...
 
int chirp_matrix_set_range (struct chirp_matrix *matrix, int x, int y, int width, int height, const void *data, time_t stoptime)
 Set a range of data. More...
 
int chirp_matrix_get (struct chirp_matrix *matrix, int x, int y, void *data, time_t stoptime)
 Get a single element. More...
 
int chirp_matrix_set (struct chirp_matrix *matrix, int x, int y, const void *data, time_t stoptime)
 Set a single element. More...
 
int chirp_matrix_setacl (const char *host, const char *path, const char *subject, const char *rights, time_t stoptime)
 Set the acls on a matrix. More...
 
int chirp_matrix_width (struct chirp_matrix *matrix)
 Get the width of a matrix. More...
 
int chirp_matrix_height (struct chirp_matrix *matrix)
 Get the height of a matrix. More...
 
int chirp_matrix_element_size (struct chirp_matrix *matrix)
 Get the element size of a matrix. More...
 
int chirp_matrix_nhosts (struct chirp_matrix *matrix)
 Get the number of hosts used by a matrix. More...
 
int chirp_matrix_nfiles (struct chirp_matrix *matrix)
 Get the number of files used by a matrix. More...
 
void chirp_matrix_fsync (struct chirp_matrix *matrix, time_t stoptime)
 Force all data to disk. More...
 
void chirp_matrix_close (struct chirp_matrix *matrix, time_t stoptime)
 Close a matrix and free all related resources. More...
 
int chirp_matrix_delete (const char *host, const char *path, time_t stoptime)
 Delete a matrix. More...
 

Detailed Description

Stores very large distributed matrices.

This module manages the storage of very large matrices (ranging from gigabytes to terabytes) by distributing the data across multiple Chirp servers. By harnessing the aggregate memory and storage of multiple machines, the time to process data is dramatically reduced.

Function Documentation

struct chirp_matrix* chirp_matrix_create ( const char *  host,
const char *  path,
int  width,
int  height,
int  element_size,
int  nhosts,
time_t  stoptime 
)

Create a new distributed matrix.

The host and path specified here will be used to store a small index file that contains the configuration of the matrix. The actual pieces of the matrix will be spread across multiple hosts. To tell Chirp where to store those pieces, you must create a hosts file that is a simple list of host names separated by newlines. Chirp will look for this file in $CHIRP_HOSTS, then in $HOME/.chirp/hosts. If neither is available, this call will fail.

Parameters
hostThe hostname and optional port of the index file.
pathThe path to the index file.
widthThe number of elements in one row.
heightThe number of elements in one column.
element_sizeThe size in bytes of each element in the matrix.
nhostsThe number of hosts on which to spread the data.
stoptimeThe absolute time at which to abort.
Returns
On success, a pointer to a struct chirp_matrix. On failure, returns zero and sets errno appropriately.
See Also
chirp_matrix_open, chirp_matrix_delete
struct chirp_matrix* chirp_matrix_open ( const char *  host,
const char *  path,
time_t  stoptime 
)

Open an existing matrix.

Parameters
hostThe hostname and optional port of the index file.
pathThe path to the index file.
stoptimeThe absolute time at which to abort.
Returns
On success, a pointer to a struct chirp_matrix. On failure, returns zero and sets errno appropriately.
See Also
chirp_matrix_close
int chirp_matrix_get_row ( struct chirp_matrix *  matrix,
int  y,
void *  data,
time_t  stoptime 
)

Get all values in a row.

This is the most efficient way to access data in a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
yThe y position of the row.
dataA pointer to a buffer where to store the data.
stoptimeThe absolute time at which to abort.
int chirp_matrix_set_row ( struct chirp_matrix *  matrix,
int  y,
const void *  data,
time_t  stoptime 
)

Set all values in a row.

This is the most efficient way to access data in a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
yThe y position of the row.
dataA pointer to a buffer containing the data to write.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_get_col ( struct chirp_matrix *  matrix,
int  x,
void *  data,
time_t  stoptime 
)

Get all values in a column.

Note that accessing columns is not as efficient as accessing rows. If possible, use chirp_matrix_get_row instead.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe x position of the column.
dataA pointer to a buffer where to store the data.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_set_col ( struct chirp_matrix *  matrix,
int  x,
const void *  data,
time_t  stoptime 
)

Set all values in a column.

Note that accessing columns is not as efficient as accessing rows. If possible, use chirp_matrix_set_row instead.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe x position of the column.
dataA pointer to a buffer containing the data to write.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_get_range ( struct chirp_matrix *  matrix,
int  x,
int  y,
int  width,
int  height,
void *  data,
time_t  stoptime 
)

Get a range of data.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe starting x position of the range.
yThe starting y position of the range;
widthThe width of the range in cells.
heightThe width of the range in cells.
dataA pointer to a buffer where to store the data.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_set_range ( struct chirp_matrix *  matrix,
int  x,
int  y,
int  width,
int  height,
const void *  data,
time_t  stoptime 
)

Set a range of data.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe starting x position of the range.
yThe starting y position of the range;
widthThe width of the range in cells.
heightThe width of the range in cells.
dataA pointer to a buffer containing the data to write.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_get ( struct chirp_matrix *  matrix,
int  x,
int  y,
void *  data,
time_t  stoptime 
)

Get a single element.

Note: Reading a single element at a time is very inefficient. If possible, get multiple elements at once using chirp_matrix_get_row.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe x position of the element.
yThe y position of the element.
dataA pointer to a buffer where to store the element.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_set ( struct chirp_matrix *  matrix,
int  x,
int  y,
const void *  data,
time_t  stoptime 
)

Set a single element.

Note: Writing a single element at a time is very inefficient. If possible, set multiple elements at once using chirp_matrix_set_row.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
xThe x position of the element.
yThe y position of the element.
dataA pointer to a buffer where to store the element.
stoptimeThe absolute time at which to abort.
Returns
Greater than or equal to zero on success, negative on failure.
int chirp_matrix_setacl ( const char *  host,
const char *  path,
const char *  subject,
const char *  rights,
time_t  stoptime 
)

Set the acls on a matrix.

int chirp_matrix_width ( struct chirp_matrix *  matrix)

Get the width of a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
Returns
The width of the matrix, measured in elements.
int chirp_matrix_height ( struct chirp_matrix *  matrix)

Get the height of a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
Returns
The height of the matrix, measured in elements.
int chirp_matrix_element_size ( struct chirp_matrix *  matrix)

Get the element size of a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
Returns
The size of each element in the matrix, measured in bytes.
int chirp_matrix_nhosts ( struct chirp_matrix *  matrix)

Get the number of hosts used by a matrix.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
Returns
The number of hosts used by the matrix.
int chirp_matrix_nfiles ( struct chirp_matrix *  matrix)

Get the number of files used by a matrix.

This value might be greater than the number of hosts, if the matrix is very large and it is necessary to break it into multiple files of one gigabyte or less.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
Returns
The number of files used by the matrix.
void chirp_matrix_fsync ( struct chirp_matrix *  matrix,
time_t  stoptime 
)

Force all data to disk.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
stoptimeThe absolute time at which to abort.
void chirp_matrix_close ( struct chirp_matrix *  matrix,
time_t  stoptime 
)

Close a matrix and free all related resources.

Parameters
matrixA pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
stoptimeThe absolute time at which to abort.
int chirp_matrix_delete ( const char *  host,
const char *  path,
time_t  stoptime 
)

Delete a matrix.

Parameters
hostThe hostname and optional port of the index file.
pathThe path to the index file.
stoptimeThe absolute time at which to abort.