Sofware Development Values

Collaboration We work closely with our user community to identify problems and solutions of mutual benefit. Our work must have value to both the end users and to the research interests of the CCL.

Responsiveness When our software is known to have a defect that affects our users, we drop everything else to fix it. When a collaborator has a problem or an idea, we jump at the opportunity to learn from their perspective.

Minimality We strive to select the simplest effective solution to the problem at hand. It is much better to solve a small problem comprehensively than a large problem incompletely.

Portability We rely on widely deployed languages and systems, so that the software is usable on the widest variety of computing platforms with modest effort. We avoid depending on new technologies that are not essential to the task at hand.

Minimum Privilege Most of our users work in computing facilities where they do not have special privileges. Modying the operating system, the filesystem, the firewall, and other such facilities is simply impossible at scale. We seek to provide tools that can be built, deployed, and used with the least privilege possible.

Interoperability We gladly interoperate with the other systems and software important our user community.

Development Workflow

We use git and github to manage our source code. The authoritative source code repository is known as main-line and is stored here:

We recommend that all developers create an account on github and then fork the cctools into their own private repository, which would be located here:

Senior members of the group (committers) will have commit rights to main-line. For junior members of the group and external contributors, the expected work-flow is:

1. Clone the main-line repository from github.
2. Make your changes.
3. Commit to your personal repository.
4. Create a pull request against main-line.
5. Ask a committer review your pull request.

When new members or external contributors want to incorporate a change into main-line, either from the local master or some other local branch, they will submit a code review request. Ben is the main person responsible for code reviews, but any of the senior members may resolve them. With a github account, the code review request is simply a pull request. Without a github account, a patch (man git-format-patch) should be sent to Ben.

We encourage the creation of branches, leaving the local master branch always as a mirror of main-line.

Guidance for the Developer

When submitting code for review, please consider the following:

• Above all, write code so that you can be proud of what you wrote!
• Split a large function into smaller logical pieces. When is a function too long? When it is bigger than your head, both literally and otherwise. When a function is large (bigger than your head literally) that is a sign that there are many lines to read and understand, and many intermediate results to keep track of (bigger than your head otherwise). Strive to solve the smallest problems that are still interesting.
• Write self-documenting code. The name of a function should tell what it does. Use comments to describe how it does it in general terms. Even if a function is only going to be called once, it is a good idea to write it as a self-documenting capsule, rather than write the corresponding code as part of another function.
• Gradually shape the source code with small commits, do not alter it with one-shot large commits. It makes easier for the other members of the group to track what the changes are, and helps everybody to find which change introduce a bug.
• Use the debug facility from dttools to display intermediate and internal information.
• If assumptions and side-effects are not obvious, write a comment.
• Be aware of the assumptions of running in a particular operating system (e.g., does your code only runs on a recent version of Linux? or it runs on Linux, but not on Darwin?). In general, strive for POSIX compliance and C99 code.
• If the limits of the function are not obvious, write a comment explaining why the limit could be broken (e.g. does a function run when the input size is ten, but not a million?).

Guidance for the Code Reviewer

Code reviews should be done with an eye towards high level issues of structure, functionality, and usability. Proposed changes to an existing, widely used tool require a higher level of scrutiny than newly created features. Avoid nitpicking on non-functional issues. The code reviewer should ask the following questions:

• Is the code suitably organized, well named, and broken into modules of appropriate sizes?
• Does the system detect and report all expected failure modes?
• Has the software been tested in the configurations important to our user community?
• Are external dependencies avoided where possible and documented where necessary?
• Are the user interface and documentation clear, consistent, and concise?
• Are any background processes stranded in the tests?

Guidance for the Release Manager

About four weeks before a planned release, a branch will be created from main-line, with the name of the release. New features will not be added to this 'release candidate' from this point on. The release wrangler will manually incorporate bug fixes added to main-line to the release candidate from this point on to the date of the release. The release wrangler will have two weeks to test the release candidate. This testing should include at least:

• Compile the release with the most strict settings of the compiler.
• Run every executable with valgrind, to detect memory leaks, uninitialized variables, etc.
• Run makeflow with the shrimp, BLAST and BWA work-flows, each with -Tsge, -Tcondor, and -Twq. In the case of -Twq, three different runs will be made, uniquely with SGE workers, uniquely with condor workers, and a mixture of SGE, condor, and local Work Queue workers. (Scripts will be developed to make this automatic.)

After this testing, the release candidate will be deployed internally in Notre Dame, in which it will be tested at least two weeks before release. The release wrangler will incorporate any bug fixes for the issues appearing in this time period.