HOWTO: Allow Condor Jobs to Access a Chirp Filesystem using Parrot

Contributed by Patrick LaFratta, 29 July 2009

For large Condor clusters that produce large output files, routing output to a global filesystem such as Chirp can be quite convenient, especially when used with the convenience of the Parrot interface. Allowing applications submitted as jobs to the Condor batch system to access files on a Chirp filesystem with Parrot requires some setup that may not be straightforward to the non-expert user of Condor and Chirp. This article explains one series of steps that will allow you to get Condor to Chirp with Parrot happily.

Basic Approach

A first thought for directing output of a Condor job running application myApp to a file on a Chirp system might be to set the executable command in the Condor submission file to "parrot" with myApp as the first argument. As this does not result in the desired behavior, we must seek an alternative route. In order submit application myApp to Condor and allow it to access /chirp/chirpServ.cse.nd.edu/myDir/, first create the directory and set the permissions of the directory for the compute nodes appropriately:

	% mkdir /chirp/chirpServ.cse.nd.edu/myDest/
	% parrot_setacl /chirp/chirpServ.cse.nd.edu/myDir/ hostname:*.nd.edu rwdl

Now, write a Condor submission file, submit.job, in a location readable by the public (e.g. /afs/nd.edu/user/mypid/Public/), and set the executable command to submit.sh:

	# Condor submission file submit.job
	universe = vanilla
	executable = /afs/nd.edu/user/mypid/Public/submit.sh
	arguments = someArgument1 someArgument2 /chirp/chirpServ.cse.nd.edu/myDir/outputFile
	queue

Create the submit.sh script in the same location. The primary purpose of submit.sh is to run myApp (assuming myApp is in the same location as submit.sh) within Parrot.

	#!/bin/bash
	arg1 = $1
	arg2 = $2
	output = $3
	parrot_run bash -c "myApp $arg1 $arg2 > $output"

Now, submit the job:

	% condor_submit submit.job

Submitting Iterative Clusters

One frequent use for Condor Chirping with Parrot is to iterate over large parameter sets for a given executable. To do this, create a third script, /afs/nd.edu/user/mypid/Public/runIter.sh, which will iteratively call "condor_submit submitIter.job". Notice that you must export the loop variables on each iteration so that they can be read as environment variables by the Condor submission file:

	#!/bin/bash
	export chirpDir=/chirp/chirpServ.cse.nd.edu/myDir/
	for fileName in $(ls /afs/nd.edu/user/mypid/files/)
	do
		for ((param1=0;param1<=9;param1++))
		do
			export fileName param1
			condor_submit submitIter.job
		done
	done

The Condor submission file, submitIter.job, would appear as follows:

	# Condor submission file submitIter.job
	outputFile = $ENV(chirpDir)output.$ENV(fileName).$ENV(param1)
	universe = vanilla
	executable = /afs/nd.edu/user/mypid/Public/submitIter.sh
	arguments = $ENV(fileName) $ENV(param1) $(outputFile)
	queue

And the submit script, submitIter.sh:

	#!/bin/bash
	file = $1
	aParam = $2
	output = $3
	parrot_run bash -c "myApp $file $aParam > $output"

And submit the cluster by running runIter.sh:

	% ./runIter.sh

Notes

It has not yet been determined how to allow Condor to direct logging, errors, and output to files residing on a Chirp filesytem through the log, error, and output commands in the Condor submission file. [an error occurred while processing this directive]