Work Queue: A Flexible Master/Worker Framework

Work Queue is a framework for building large master/worker solutions that can mix multicore, cluster, cloud, and grid resources. The framework is defined by a an API that allows users to build master programs that complete their work by creating tasks, submitting the tasks to a queue, and then receiving back the results of the tasks. The tasks are executed by an included component of the Work Queue framework, the workers, which are started by the user on available resources and negotiate work allocations with the master process. The Work Queue handles the data transfer and caching.

Work Queue workers can be executed on a variety of systems, including individual machines to which the user has access, computing clusters, or a distributed Condor batch system. The master/worker paradigm is the same in each case, so you can easily grow your application from one machine up to thousands.

For More Information

Publications