BXGrid User's Manual

Overview

Beside an easy to use web portal, which let users browse,validate, share, and download biometrics datasets, we also provide a more powerful command line tool for expert users. This manual shows you how to use BXGRid command line tool to perform basic and advance operations.

You need a BXGrid account to use the tool. Please contact Dr. Douglas Thain: dthain at nd dot edu or Dr. Patrick Flynn: flynn at nd dot edu.

To use the command line tool, add the bxgrid directory to your path. If you use bash:

export PATH=/afs/crc.nd.edu/group/ccl/software/bxgrid/bin:${PATH}

Or if you use tcsh:

export PATH /afs/crc.nd.edu/group/ccl/software/bxgrid/bin:${PATH}

Basic Operations

[hbui@cclws01]$ bxgrid
Use:
 bxgrid [options] <database> <function> <argument(s)> ...
Options are:
 -G Enable Chirp  debug mode
 -D Enable detail mode, show import/export information
 -M Enable Mysql  debug mode
 -u specify username
 -p specify password
 -s specify database hostname
 -o  output format for metadata. Options are -o line or -o csv
 -m export symlink instead of actual data file
 -n number of files to audit or repair. (default 1000)
Avaliable functions:
 login    Stores username and password on local computer.
          e.g. bxgrid biometrics login
 logout   Deletes username and password on local computer.
          e.g. bxgrid biometrics logout
 import   <table%gt;  <metadata file>
          Import recordings from metadata file to <table>.
          e.g. bxgrid biometrics import faces_still 200801.datafile data
 export   <table> to <metadata file> <output dir> as <dir/name schema> where <conditions>
          Export metadata + data files from <table> with conditions.
          e.g. bxgrid biometrics export irises_still to test.xml /tmp as /race/sequenceid metadata subjectid,sequenceid where TRUE limit 100
 query   <table> to <metadata file> where <condition>
          Query metadata from <table> with conditions.
          e.g. bxgrid biometrics query irises_still to test.xml where TRUE limit 100
 delete   <table>  where <conditions>
          Delete recordings from <table> with conditions.
          e.g. bxgrid biometrics delete faces_still where subjectid = 'nd1S04473'
 benchmark   <table>  <metadata file>  <data path>
          Benchmark bxgrid performance.
          e.g. bxgrid biometrics benchmark faces_still 200801-facestills.datafile~ccl/software/bxgrid/testdata/dat
 transform_set   <setname>  <function> <number of file per condor job>
          Transform a set using Condor.
          e.g. bxgrid biometrics transform_set AP100 20
 audit    Evaluate database for consistency and redundancy.
          e.g. bxgrid biometrics audit -n 1000
 repair   Like audit, but also repair damage as needed.
          e.g. bxgrid biometrics repair -n 1000

Because we are hosting multiple databases, you need to specify the database for each operation, following by the operation: import,export,query bxgrid will prompt you to enter your username and pasword everytime or you can include your username and password from command line using -u and -p options

[hbui@cclws01]$ bxgrid -u hbui -p mypassword biometrics export irises_still to test.xml /tmp as /race/sequenceid metadata subjectid,sequenceid where TRUE limit 100

Here are some bxgrid basic operations:

1. login/logout: To avoid having to type in your username/password all the time, you can use bxgrid login operation to save your username and password

   [hbui@cclws01]$ bxgrid biometrics login mysql login: hbui password:

   To delete your saved password, use bxgrid logout

   [hbui@cclws01]$ bxgrid biometrics logout

2. export: Simply run bxgrid <database> export <datatype> to <metadata file> <output dir> where <conditions>

   [hbui@cclws01]$ bxgrid biometrics export irises_still to test.data /tmp where subjectid = \'nd1S04473\'

   There are main 5 data types you can export: irises_still, faces_still, irises_mov, faces_mov and faces_3d. Default option will export all metadata and using sequenceid(if avaliable) as export filenames. You can specify metadata you need using keyword metadata, and choose output data directory and naming schema using keyword as

   [hbui@cclws01]$ bxgrid biometrics export irises_still to test.data /tmp as /race/fileid metadata subjectid,sequenceid where TRUE limit 100

   Output data is seperated by race and bxgrid use fileid as file name. The metadata file will look like these

   subjectid string nd1S04473 sequenceid string 04473d468 file file /tmp/White/160461.tiff subjectid string nd1S04853 sequenceid string 04853d578 file file /tmp/Asian/160456.tiff

3. query query usage is very similar to export usage. You want to use query if you only want metadata.

   [hbui@cclws01]$ bxgrid biometrics query irises_still to test.data metadata subjectid,sequenceid where TRUE limit 100

   The metadata file will look like these.

   subjectid string nd1S04473 sequenceid string 04473d468 file file xmlonly/04473d466.tiff subjectid string nd1S04853 sequenceid string 04853d578 file file xmlonly/04473d466.tiff

   Again, default option without keyword metadata will query all metadata.

Advanced Operations

1. import: Use import to ingest new data to BXGrid.
   Previously using import <table%gt; <metadata file> <data path> we need to specify a metadata file and a directory where the actual data file is. Metadata file is in line format (one line per recording). An example of a metadata file

   2008-253-042-1_L-lg4000.tiff 2463 Left tiff Brown 09/09/2008 No nd4T00015 Inside nd4E00054 nd4N00020 nd4I00013 1 2008-253-042-1_R-lg4000.tiff 2463 Right tiff Brown 09/09/2008 No nd4T00015 Inside nd4E00054 nd4N00020 nd4I00013 1 2008-253-042-2_L-lg4000.tiff 2463 Left tiff Brown 09/09/2008 No nd4T00015 Inside nd4E00054 nd4N00020 nd4I00013 2

   Now we are moving to a new format for metadata file which is name value pair

   shotid string 2008-253-042-1_L-lg4000.tiff subjectid string nd1S02463 eye string Left format string tiff color string Brown date string 2008-09-09 00:00:00 glasses string No stageid string nd4T00015 weather string Inside environmentid string nd4E00054 sensorid string nd4N00020 illuminantid1 string nd4I00013 shot string 1 state string unvalidated file file 2008-253-042-1_L-lg4000.tiff shotid string 2008-253-042-1_R-lg4000.tiff subjectid string nd1S02463 eye string Right format string tiff color string Brown date string 2008-09-09 00:00:00 glasses string No stageid string nd4T00015 weather string Inside environmentid string nd4E00054 sensorid string nd4N00020 illuminantid1 string nd4I00013 shot string 1 state string unvalidated file file 2008-253-042-1_R-lg4000.tiff

   You only need to sepcify a metadata file: import <table%gt; <metadata file>

   bxgrid -D biometrics import faces_still test.dat /tmp/hbui/05213d305.NEF checksum: e4a46c6c6c93347b5d677db174b28382 file id 1080247 replica 2347150 sc0-14.cse.nd.edu bxgridtest/2/8/1080247_2347150.NEF replica 2347151 sc0-11.cse.nd.edu bxgridtest/8/1/1080247_2347151.NEF replica 2347152 sc0-04.cse.nd.edu bxgridtest/6/7/1080247_2347152.NEF /tmp/hbui/05213d306.JPG checksum: 2d402ebebdf219b59c48e5a3c3c93b32 file id 1080248 replica 2347153 sc0-15.cse.nd.edu bxgridtest/0/1/1080248_2347153.JPG replica 2347154 sc0-24.cse.nd.edu bxgridtest/8/6/1080248_2347154.JPG replica 2347155 sc0-08.cse.nd.edu bxgridtest/1/1/1080248_2347155.JPG /tmp/hbui/05213d307.NEF checksum: 10ba05351013682f74ae74c16f40866e file id 1080249 replica 2347156 sc0-21.cse.nd.edu bxgridtest/5/7/1080249_2347156.NEF replica 2347157 sc0-28.cse.nd.edu bxgridtest/7/2/1080249_2347157.NEF replica 2347158 sc0-01.cse.nd.edu bxgridtest/7/5/1080249_2347158.NEF imported 3 recording into temp_collectionid 1253734194 faces_still table

   Option -D will give you more detail about ingesting process such as checksum of data file, location of replicas,.. And more important, the unique temporary collectionid (1253734194) in this case. The temporary collectionid is an unique number for each ingestion batch. You can use this to assgign validation task, or remove a bad batch using delete operation.

   In order to convert old one line per recording format to new name value pair format, please use a perl script here:

   /afs/crc.nd.edu/group/software/bxgrid/script/l2nvp.pl Usage: l2nvp.pl <datatype> <metadata file> <new metadata file> For example: [hbui@cclws01]$ l2nvp.pl irises_still 200807-irisstills.datafile test.dat

2. delete: Use delete to remove data from BXGrid.
   

3. audit: Use audit to check data integrity.
   

4. repair: Use repair after audit to correct inconsistent data.