CCL | Software | Download | Manuals | Forum | Papers
CCL Home

Research

Software Community Operations

Recent News in the CCL


Congrads to Ph.D Graduates

Congratulations to all of our 2017 Ph.D. graduates in Computer Science and Engineering,
and especially to Dr. Haiyan Meng who is moving on to a position at Google, Inc.

Mon, 22 May 2017 13:37:00 +0000

Announcement: CCTools 6.1.0. released

The Cooperative Computing Lab is pleased to announce the release of version 6.1.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, Umbrella, Prune, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a major which adds several features and bug fixes. Among them:

  • [General]  IPv4 and IPv6 mode handling fixes. (Tim Shaffer)
  • [Grow]  Fuse module for GROW-FS. (Tim Shaffer)
  • [Makeflow]  Updated manual. (Douglas Thain)
  • [Makeflow]  Support for Mesos. (Charles Zheng)
  • [Makeflow]  Support for Singularity. (Kyle Sweeney)
  • [Makeflow]  --shared-fs option. (Nick Hazekamp)
  • [Makeflow]  --preserve option for per rule caching. (Pierce Cunneen)
  • [Parrot]  Fix ld.so and exec bugs. (Tim Shaffer)
  • [Parrot]  Fix handling of namespace symlinks. (Tim Shaffer)
  • [Parrot]  Bind/connect on AF_UNIX sockets. (Tim Shaffer)
  • [Parrot]  Several bug fixes. (Tim Shaffer)
  • [Parrot]  parrot_namespace to use parrot inside parrot. (Tim Shaffer)
  • [Parrot]  Add fixed and warped PIDs. (Douglas Thain)
  • [Prune]     Several bug fixes. (Peter Ivie)
  • [ResourceMonitor]  Resource per task snapshots, --snapshot-file. (Ben Tovar)
  • [Umbrella]  Several bug fixes. (Haiyan Meng)
  • [WorkQueue]  Resource per task snapshots, q.enable_monitoring_snapshots. (Ben Tovar)
  • [WorkQueue]  Several bug fixes. (Ben Tovar)
  • [WorkQueue]  Custom environments for wq factory. (Kyle Sweeney)
  • [WorkQueue]  Several fixes to wq factory. (Nate Kremer-Herman)
  • [WQ_Maker]   Several bug fixes, update to latest version of maker. (Nick Hazekamp)

Thanks goes to the contributors for many features, bug fixes, and tests:

  • Jakob Blomer
  • Pierce Cunneen
  • Patrick Donnelly
  • Nathaniel Kremer-Herman
  • Nicholas Hazekamp
  • Peter Ivie
  • Haiyan Meng
  • Tim Shaffer
  • Douglas Thain
  • Ben Tovar
  • Kyle Sweeney
  • Chao Zheng

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy! Wed, 17 May 2017 16:55:00 +0000

Makeflow and Mesos Paper at CCGrid 2017

Charles Zheng will present the paper Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos at the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2017) in May 15, 2017 at Madrid, Spain. In this paper we consider how to launch workflow system on container schedulers with minimal performance loss and higher system efficiency. As examples of current technology, we use Makeflow,  Work Queue, Resource Monitor and Mesos. We observe that using Work Queue and Resource Monitor not only reduces the task turnaround time but also achieves higher resource usage rate. Following is the system architecture.

Fri, 05 May 2017 17:28:00 +0000

Workflow Reproducibility Paper at ICCS 2017

Haiyan Meng will present a paper titled Facilitating the Reproducibility of Scientific Workflows with Execution Environment Specifications at the International Conference on Computational Science (ICCS) 2017 this June at Zurich, Switzerland. This paper explores the challenges in reproducing scientific workflows, and proposes a framework for facilitating the reproducibility of scientific workflows at the task level by giving scientists complete control over the execution environments of the tasks in their workflows and integrating execution environment specifications into scientific workflow systems.



Mon, 01 May 2017 17:58:00 +0000

Ph.D. Defense: Haiyan Meng

Haiyan Meng successfully defended her dissertation titled "Improving the Reproducibility of Scientific Applications with Execution Environment Specifications"  Congratulations!


Wed, 22 Mar 2017 18:04:00 +0000

Makeflow Examples Archive

We recently updated our archive of example Makeflows so that they are significantly easier to download, execute, and reshape to various sizes.   For each one, we have instructions on how to obtain the underlying binary program, generate some sample data, and then create a workload of arbitrary size.  This allows you to experiment with Makeflow at small scale, and then dial things up when you are ready run on on thousands of nodes:

https://github.com/cooperative-computing-lab/makeflow-examples


Thu, 09 Mar 2017 18:32:00 +0000

Big CMS Data Analysis at Notre Dame

Analyzing the data produced by the Compact Muon Solenoid (CMS), one of the experiments at the Large Hadron Collider, requires a collaboration of physicists, computer scientists to harness hundreds of thousands of computers at universities and research labs around the world.  The contribution of each site to the global effort, whether small or large, is reported out on a regular basis.

This recent graph tells an interesting story about contributions to CMS computing in late 2016.  Each color in the bargraph represents the core-hours provided by a given site over the course of a week:

The various computing sites are divided into tiers:

  • Tier 0 is CERN, which is responsible for providing data to the lower tiers.
  • Tier 1 contains the national research labs like Fermi National Lab (FNAL), Rutherford Appleton Lab in in UK, and so forth, that facilitate analysis work for universities in their countries.
  • Tier 2 contains universities like Wisconsin, Purdue, and MIT, that have significant shared computing facilities dedicated to CMS data analysis.
  • Tier 3 is everyone else performing custom data analysis, sometimes on private clusters, and sometimes on borrowed resources. Most of those are so small that they are compressed into black at the bottom of the graph.

Now, you would think that the big national sites would produce most of the cycles, but there are a few interesting exceptions at the top of the list.

First, there are several big bursts in dark green that represent the contribution of the HEPCloud prototype, which is technically a Tier-3 operation, but is experimenting with consuming cycles from Google and Amazon.  This has been successful at big bursts of computation, and the next question is whether this will be cost-effective over the long term.

Next, the Tier-2 at the University of Wisconsin consistently produces a huge number of cycles from their dedicated facility and opportunistic resources from the Center for High Throughput Computing.  This group works closely with the HTCondor team at Wisconsin to make sure every cycle gets used, 365 days a year.

Following that, you have the big computing centers at CERN and FNAL, which is no surprise.

And, then the next contributor is our own little Tier-3 at Notre Dame, which frequently produces more cycles than most of the Tier-2s and some of the Tier-1s!  The CMS group at ND harnesses a small dedicated cluster, and then adds to that unused cycles from our campus Center for Research Computing by using Lobster and the CCL Work Queue software on top of HTCondor.

The upshot is, on a good day, a single grad student from Notre Dame can perform data analysis at a scale that rivals our national computing centers!


Fri, 17 Feb 2017 14:37:00 +0000

IceCube Flies with Parrot and CVMFS


IceCube is a neutrino detector built at the South Pole by instrumenting about a cubic kilometer of ice with 5160 light sensors. The IceCube data is analyzed by a collaboration of about 300 scientists from 12 countries. Data analysis relies on the precise knowledge of detector characteristics, which are evaluated by vast amounts of Monte Carlo simulation.  On any given day, 1000-5000 jobs are continuously running.

Recently, the experiment began using Parrot to get their code running on GPU clusters at XSEDE sites (Comet, Bridges, and xStream) and the Open Science Grid.  IceCube relies on software distribution via CVMFS, but not all execution sites provide the necessary FUSE modules.  By using Parrot, jobs can attach to remote software repositories without requiring special privileges or kernel modules.

- Courtesy of Gonzalo Merino, University of Wisconsin - Madison Thu, 02 Feb 2017 21:25:00 +0000

Reproducibility Papers at eScience 2016

CCL students presented two papers at the IEEE 12th International Conference on eScience on the theme of reproducibility in computational science:
Congrads to Haiyan for winning a Best of Conference award for her paper:
    Tue, 25 Oct 2016 13:47:00 +0000

    CCL Workshop 2016

    The 2016 CCL Workshop on Scalable Scientific Computing was held on October 19-20 at the University of Notre Dame.  We offered tutorials on Makeflow, Work Queue, and Parrot. and gave highlights of the many new capabilities relating to reproducibility and container technologies. Our user community gave presentations describing how these technologies are used to accelerate discovery in genomics, high energy physics, molecular dynamics, and more. 
    Everyone got together to share a meal, solve problems, and generate new ideas. Thanks to everyone who participated, and see you next year

    Tue, 25 Oct 2016 13:26:00 +0000

    NSF Grant to Support CCTools Development

    We are pleased to announce that our work will continue to be supported by the National Science Foundation through the division of Advanced Cyber Infrastructure.

    The project is titled "SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools"  It will advance the development of the Cooperative Computing Tools to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale.  We will continue to focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds. 

    Tue, 20 Sep 2016 15:11:00 +0000

    Announcement: CCTools 6.0.0. released

    The Cooperative Computing Lab is pleased to announce the release of version 6.0.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, Umbrella, Prune, SAND, All-Pairs, Weaver, and other software.

    The software may be downloaded here:
    http://ccl.cse.nd.edu/software/download

    This is a major which adds several features and bug fixes. Among them:

    • [Catalog]   Automatic fallback to a backup catalog server. (Tim Shaffer)
    • [Makeflow]  Accept DAGs in JSON format. (Tim Shaffer)
    • [Makeflow]  Multiple documentation omission bugs. (Nick Hazekamp and Haiyan Meng)
    • [Makeflow]  Send information to catalog server. (Kyle Sweeney)
    • [Makeflow]  Syntax directives (e.g. .SIZE for to indicate file size). (Nick Hazekamp)
    • [Parrot] Fix cvmfs logging redirection. (Jakob Blomer)
    • [Parrot] Multiple bug-fixes. (Tim Shaffer, Patrick Donnelly, Douglas Thain)
    • [Parrot] Timewarp mode for reproducible runs. (Douglas Thain)
    • [Parrot] Use new libcvmfs interfaces if available. (Jakob Blomer)
    • [Prune]     Use SQLite as backend. (Peter Ivie)
    • [Resource Monitor] Record the time where a resource peak occurs. (Ben Tovar)
    • [Resource Monitor] Report the peak number of cores used. (Ben Tovar)
    • [Work Queue] Add a transactions log. (Ben Tovar)
    • [Work Queue] Automatic resource labeling and monitoring. (Ben Tovar)
    • [Work Queue] Better capacity worker autoregulation. (Ben Tovar)
    • [Work Queue] Creation of disk allocation per tasks. (Nate Herman-Kremer)
    • [Work Queue] Extensive updates to wq_maker. (Nick Hazekamp)
    • [Work Queue] Improvements in computing master's task capacity. (Nate Herman-Kremer).
    • [Work Queue] Raspberry Pi compilation fixes. (Peter Bui)
    • [Work Queue] Throttle work_queue_factory with --workers-per-cycle. (Ben Tovar)
    • [Work Queue] Unlabeled tasks are assumed to consume 1 core, 512 MB RAM and 512 MB disk. (Ben Tovar)
    • [Work Queue] Worker disconnects when node does not longer have the resources promised. (Ben Tovar)
    • [Work Queue] work queue statistics clean up (see work_queue.h for deprecated names). (Ben Tovar)
    • [Work Queue] work_queue_status respects terminal column settings. (Mathias Wolf)

    We will have tutorials on the new features in our upcoming workshop, October 19 and 20. Refer to http://ccl.cse.nd.edu/workshop/2016 for more information. We hope you can join us!

    Thanks goes to the contributors for many features, bug fixes, and tests:

    • Jakob Blomer
    • Peter Bui
    • Patrick Donnelly
    • Nathaniel Kremer-Herman
    • Kenyi Hurtado-Anampa
    • Peter Ivie
    • Kevin Lannon
    • Haiyan Meng
    • Tim Shaffer
    • Douglas Thain
    • Ben Tovar
    • Kyle Sweeney
    • Mathias Wolf
    • Anna Woodard
    • Chao Zheng

    Please send any feedback to the CCTools discussion mailing list:

    http://ccl.cse.nd.edu/community/forum

    Enjoy! Thu, 15 Sep 2016 17:55:00 +0000

    Summer REU Projects in Data Intensive Scientific Computing

    We recently wrapped up the first edition of the summer REU in Data Intensive Scientific Computing at the University of Notre Dame.  Ten undergraduate students came to ND from around the country and worked on projects encompassing physics, astronomy, bioinformatics, network sciences, molecular dynamics, and data visualization with faculty at Notre Dame.

    To learn more, see these videos and posters produced by the students:

           


    Fri, 19 Aug 2016 13:24:00 +0000

    Simulation of HP24stab with AWE and Work Queue


    The villin headpiece subdomain "HP24stab" is a recently discovered 24-residue stable supersecondary structure that consists of two helices joined by a turn. Simulating 1μs of motion for HP24stab can take days or weeks depending on the available hardware, and folding events take place on a scale of hundreds of nanoseconds to microseconds.  Using the Accelerated Weighted Ensemble (AWE), a total of 19us of trajectory data were simulated over the course of two months using the OpenMM simulation package. These trajectories were then clustered and sampled to create an AWE system of 1000 states and 10 models per state. A Work Queue master dispatched 10,000 simulations to a peak of 1000 connected 4-core workers, for a total of 250ns of concurrent simulation time and 2.5μs per AWE iteration. As of August 8, 2016, the system has run continuously for 18 days and completed 71 iterations, for a total of 177.5μs of simulation time. The data gathered from these simulations will be used to report the mean first passage time, or average time to fold, for HP24stab, as well as the major folding pathways.  - Jeff Kinnison and Jesús Izaguirre, University of Notre Dame
    Wed, 10 Aug 2016 20:58:00 +0000

    ND Leads DOE Grant on Virtual Clusters for Scientific Computing

    Prof. Douglas Thain is leading a new $2.2M DOE-funded project titled "VC3: Virtual Clusters for Community Computation" in an effort to make our national supercomputing facilities more effective for collaborative scientific computing.  The project team brings together researchers from the University of Notre Dame, the University of Chicago, and Brookhaven National Lab.



    Our current NSF and DOE supercomputers are very powerful, but they each have different operating systems and software configurations, which makes it difficult and time consuming for new users to deploy their codes and share results.  The new service will create virtual clusters on the existing machines that have the custom software and other services needed to easily run advanced scientific codes from fields such as high energy physics, bioinformatics, and astrophysics.  If successful, users of this service will be able to easily move applications between university and national supercomputing facilities.





    Wed, 10 Aug 2016 20:30:00 +0000

    2016 DISC Summer Session Wraps Up

    Congratulations to our first class of summer students participating in the Data Intensive Scientific Computing research experience!  Twelve students from around the country came to Notre Dame to learn how computing drives research in high energy physics, climatology, bioinformatics, astrophysics, and molecular dynamics.

    At our closing poster session in Jordan hall (along with several other REU programs) students presented their work and results to faculty and guests across campus.

    If you are excited to work at the intersection of scientific research and advanced computing, we invite you to apply to the 2017 DISC summer program at Notre Dame! Fri, 29 Jul 2016 18:00:00 +0000

    New Work Queue Visualization

    Nate Kremer-Herman has created a new, convenient way to lookup information of Work Queue masters. This new visualization tool provides real-time updates on the status of each Work Queue master that contacts our catalog server. We hope that this new tool will serve to both facilitate our users' understanding of what their Work Queue masters are doing and assist the user in determining when it may be time to take corrective action.

    A Comparative View


    A Specific Master


    In part, this tool provides our users with measurements on their tasks currently running, the number of tasks waiting to be run, and the total capacity of tasks that could be running. As an example, a user could find that they have a large number of tasks waiting, a small number of tasks running, and a task capacity that is somewhere in between. A recommendation we could make to a user who is seeing something like this would be to ask for more workers. Our hope is that users will take advantage of this new way to view and manage their work.

    Wed, 22 Jun 2016 18:57:00 +0000

    Work Queue from Raspberry Pi to Azure at SPU

    "At Seattle Pacific University we have used Work Queue in the CSC/CPE 4760 Advanced Computer Architecture course in Spring 2014 and Spring 2016.  Work Queue serves as our primary example of a distributed system in our “Distributed and Cloud Computing” unit for the course.  Work Queue was chosen because it is easy to deploy, and undergraduate students can quickly get started working on projects that harness the power of distributed resources."

    The main project in this unit had the students obtain benchmark results for three systems: a high performance workstation; a cluster of 12 Raspberry Pi 2 boards, and a cluster of A1 instances in Microsoft Azure.  The task for each benchmark used Dr. Peter Bui’s Work Queue MapReduce framework; the students tested both a Word Count and Inverted Index on the Linux kernel source. In testing the three systems the students were exposed to the principles of distributed computing and the MapReduce model as they investigated tradeoffs in price, performance, and overhead.
     - Prof. Aaron Dingler, Seattle Pacific University. 

    Thu, 26 May 2016 12:52:00 +0000

    Lifemapper analyzes biodiversity using Makeflow and Work Queue

    Lifemapper is a high-throughput, webservice-based, single- and multi-species modeling and analysis system designed at the Biodiversity Institute and Natural History Museum, University of Kansas. Lifemapper was created to compute and web publish, species distribution models using available online species occurrence data.  Using the Lifemapper platform, known species localities georeferenced from museum specimens are combined with climate models to predict a species’ “niche” or potential habitat availability, under current day and future climate change scenarios. By assembling large numbers of known or predicted species distributions, along with phylogenetic and biogeographic data, Lifemapper can analyze biodiversity, species communities, and evolutionary influences at the landscape level.

    Lifemapper has had difficulty scaling recently as our projects and analyses are growing exponentially.  For a large proof-of-concept project we deployed on the XSEDE resource Stampede at TACC, we integrated Makeflow and Work Queue into the job workflow.  Makeflow simplified job dependency management and reduced job-scheduling overhead, while Work Queue scaled our computation capacity from hundreds of simultaneous CPU cores to thousands.  This allowed us to perform a sweep of computations with various parameters and high-resolution inputs producing a plethora of outputs to be analyzed and compared.  The experiment worked so well that we are now integrating Makeflow and Work Queue into our core infrastructure.  Lifemapper benefits not only from the increased speed and efficiency of computations, but the reduced complexity of the data management code, allowing developers to focus on new analyses and leaving the logistics of job dependencies and resource allocation to these tools.


    Information from C.J. Grady, Biodiversity Institute and Natural History Museum, University of Kansas.

    Tue, 24 May 2016 15:30:00 +0000

    Condor Week 2016 presentation

    We presented in Condor Week 2016 our approach to create a comprehensive resource feedback loop to execute tasks of unknown size. In this feedback look, tasks are monitored and measured in user-space as they run; the resource usage is collected into an online archive; and further instances are provisioned according to the application's historical data to avoid resource starvation and minimize resource waste. We present physics and bioinformatics case studies consisting of more than 600,000 tasks running on 26,000 cores (96% of them from opportunistic resources), where the proposed solution leads to an overall increase in throughput (from 10% to 400% across different workflows), and a decrease in resource waste compared to workflow executions without the resource feedback-loop.

    condor week 2016 slides Tue, 24 May 2016 12:32:00 +0000

    Containers, Workflows, and Reproducibility

    The DASPOS project hosted a workshop on Container Strategies for Data and Software Preservation that Promote Open Science at Notre Dame on May 19-20, 2016.  We had a very interesting collection of researchers and practitioners, all working on problems related to reproducibility, but presenting different approaches and technologies.

    Prof. Thain presented recent work by CCL grad students Haiyan Meng and Peter Ivie on Combining Containers and Workflow Systems for Reproducible Execution.


    The Umbrella tool created by Haiyan Meng allows for a simple, compact, archival representation of a computation, taking into account hardware, operating system, software, and data dependencies.  This allows one to accurately perform computational experiments and give each one a DOI that can be shared, downloaded, and executed.


    The PRUNE tool created by Peter Ivie allows one to construct dynamic workflows of connected tasks, each one precisely specified by execution environment.  Provenance and data are tracked precisely, so that the specification of a workflow (and its results) can be exported, imported, and shared with other people.  Think of it like git for workflows.



    Mon, 23 May 2016 18:44:00 +0000

    Balancing Push and Pull in Confuga, an Active Storage Cluster File System for Scientific Workflows

    Patrick Donnelly has published a journal article in Concurrency and Computation: Practice and Experience on the Confuga active cluster file system. The journal article presents the use of controlled transfers to distribute data within the cluster without destabilizing the resources of storage nodes:


    Patrick Donnelly and Douglas Thain, Balancing Push and Pull in Confuga, an Active Storage Cluster File System for Scientific Workflows, Concurrency and Computation: Practice and Experience, May 2016.

    Confuga is a new active storage cluster file system designed for executing regular POSIX workflows. Users may store extremely large datasets on Confuga in a regular file system layout, with whole files replicated across the cluster. You may then operate on your dataset using regular POSIX applications, with defined inputs and outputs.


    Jobs execute with full data locality with all whole-file dependencies available in its own private sandbox. For this reason, Confuga will first copy a job's missing data to the target storage node prior to dispatching the job. This journal article examines two transfer mechanisms used in Confuga to manage this data movement: push and pull.

    A push transfer is used to direct a storage node to copy a file to another storage node. Pushes are centrally managed by the head node which allows it to schedule transfers in a way that avoids destabilizing the cluster or individual storage nodes. To avoid some potential inefficiencies with centrally managed transfers, Confuga also uses pull transfers which resemble file access in a typical distributed file system. A job will pull its missing data dependencies from other storage nodes prior to execution.

    This journal article examines the trade-offs of the two approaches and settles on a balanced approach where pulls are used for transfers of smaller files and pushes are used for larger files. This results in significant performance improvements for scientific workflows with large data dependencies. For example, two bioinformatics workflows we studied, a Burrows-Wheeler Aligner (BWA) workflow and an Iterative Alignments of Long Reads (IALR) workflow, achieved 48% and 77% reductions in execution time compared to using either an only push or only pull strategy.

    For further details, please check out our journal article here. Confuga is available as part of the Cooperative Computing Toolset distributed under the GNU General Public License. For usage instructions, see the Confuga manual and man page.

    See also our last blog post on Confuga which introduced Confuga. Mon, 09 May 2016 14:20:00 +0000

    Interships at Red Hat and CERN

    Two CCL graduate students will be off on internships in summer 2016:

    Haiyan Meng will be interning at Red Hat, working on container technologies.


    Tim Shaffer will be a summer visitor at CERN, working with the CVMFS group on distributed filesystems for HPC and high energy physics.
    Tue, 03 May 2016 14:36:00 +0000

    Ph.D. Defense: Patrick Donnelly

    Patrick Donnelly successfully defended his Ph.D. titled "Data Locality Techniques in an Active Cluster Filesystem for Scientific Workflows". 

    Congratulations to Dr. Donnelly!


    The continued exponential growth of storage capacity has catalyzed the broad acquisition of scientific data which must be processed. While today's large data analysis systems are highly effective at establishing data locality and eliminating inter-dependencies, they are not so easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address the needs of scientific computing, I propose the design of an active storage cluster file system which allows for execution of regular unmodified applications with full data
    locality.

    This dissertation analyzes the potential benefits of exploiting the structural information already available in scientific workflows -- the explicit dependencies -- to achieve a scalable and stable system. I begin with an outline of the design of the Confuga active storage cluster file system and its applicability to scientific computing. The remainder of the dissertation examines the techniques used to achieve a scalable and stable system. First, file system access by jobs is scoped to explicitly defined dependencies resolved at job dispatch. Second, workflow's structural information is harnessed to direct and control necessary file transfers to enforce cluster stability and maintain performance. Third, control of transfers is selectively relaxed to improve performance by limiting any negative effects of centralized transfer management.

    This work benefits users by providing a complete batch execution platform joined with a cluster file system. The user does not need to redesign their workflow or provide additional consideration to the management of data dependencies. System stability and performance is managed by the cluster file
    system while providing jobs with complete data locality. - See more at: http://cse.nd.edu/events/phd-defense-patrick-donnelly#sthash.4BLYgJYM.dpuf
    DATA LOCALITY TECHNIQUES IN AN ACTIVE CLUSTER FILE SYSTEM DESIGNED FOR
    SCIENTIFIC WORKFLOWS - See more at: http://cse.nd.edu/events/phd-defense-patrick-donnelly#sthash.4BLYgJYM.dpuf
    DATA LOCALITY TECHNIQUES IN AN ACTIVE CLUSTER FILE SYSTEM DESIGNED FOR
    SCIENTIFIC WORKFLOWS - See more at: http://cse.nd.edu/events/phd-defense-patrick-donnelly#sthash.4BLYgJYM.dpuf
    Wed, 06 Apr 2016 18:38:00 +0000

    Searching for Exo-Planets with Makeflow and Work Queue

    Students at the University of Arizona made use of Makeflow and Work Queue to build an image processing pipeline on the Chameleon cloud testbed at TACC.

    The course project was to build an image processing pipeline to accelerate the research of astronomer Jared Males, who designs instruments to search for exo-planets by observing the changes in appearance of a star.  This results in hundreds of thousands of images of a single star, which must then be processed in batch to eliminate noise and align the images.

    The students built a solution (Find-R) which consumed over 100K CPU-hours on Chameleon, distributed using Makeflow and Work Queue.

    Read more here:
    https://www.tacc.utexas.edu/-/in-search-of-a-planet
    Tue, 22 Mar 2016 19:14:00 +0000