Home > Events > PhD Defense - Patrick Donnelly

PhD Defense - Patrick Donnelly

Start: 4/6/2016 at 12:00PM
End: 4/6/2016 at 3:30PM
Location: 257G Fitzpatrick Hall
Attendees: Faculty and students are welcome to attend the presentation portion of the defense. Light refreshments will be served.
Add to calendar:
iCal vCal

Patrick Donnelly

April 6, 2016

12:00 pm

257G Fitzpatrick Hall

Adviser:  Dr. Douglas Thain

Committee Members:  Dr. Scott Emrich     Dr. Collin McMillan     Dr. Christian Poellabauer

Title

“DATA LOCALITY TECHNIQUES IN AN ACTIVE CLUSTER FILE SYSTEM DESIGNED FOR
SCIENTIFIC WORKFLOWS”

Abstract

The continued exponential growth of storage capacity has catalyzed the broad acquisition of scientific data which must be processed. While today's large data analysis systems are highly effective at establishing data locality and eliminating inter-dependencies, they are not so easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address the needs of scientific computing, I propose the design of an active storage cluster file system which allows for execution of regular unmodified applications with full data
locality.

This dissertation analyzes the potential benefits of exploiting the structural information already available in scientific workflows -- the explicit dependencies -- to achieve a scalable and stable system. I begin with an outline of the design of the Confuga active storage cluster file system and its applicability to scientific computing. The remainder of the dissertation examines the techniques used to achieve a scalable and stable system. First, file system access by jobs is scoped to explicitly defined dependencies resolved at job dispatch. Second, workflow's structural information is harnessed to direct and control necessary file transfers to enforce cluster stability and maintain performance. Third, control of transfers is selectively relaxed to improve performance by limiting any negative effects of centralized transfer management.

This work benefits users by providing a complete batch execution platform joined with a cluster file system. The user does not need to redesign their workflow or provide additional consideration to the management of data dependencies. System stability and performance is managed by the cluster file
system while providing jobs with complete data locality.