SCOOP (Scalable Concurrent Operations in Python) is a
Python software module for distributing
concurrent tasks on various environments, from
heterogeneous grids of workstations to supercomputers.
It uses
ØMQ and th
Greenletpackage as building blocks to encapsulate and distribute tasks (named a
Future
The future is the time after the past and present. Its arrival is considered inevitable due to the existence of time and the laws of physics. Due to the apparent nature of reality and the unavoidability of the future, everything that currently ...
) between processes and/or systems. Its interface is inspired from th
PEP-3148proposal.
SCOOP is targeted towards scientific applications that require execution of many loosely coupled tasks using all available hardware resources. These resources need to be accessible through
SSH.
History
SCOOP was initiated by Yannick Hold and Marc Parizeau at the Computer Vision and Systems Laboratory of
Université Laval. It is an iterative step over the now deprecated DTM module of the DEAP framework for developing
evolutionary algorithm. While DTM used
MPI
MPI or Mpi may refer to:
Science and technology Biology and medicine
* Magnetic particle imaging, an emerging non-invasive tomographic technique
* Myocardial perfusion imaging, a nuclear medicine procedure that illustrates the function of the hear ...
for its communications, SCOOP uses instead ØMQ.
Network topology
SCOOP uses the Broker Architecture
to distribute its Futures. It is based on a central element, called the Broker, that dispatches work to its workers. The main difference between this pattern and a
Master/slave topology reside in the Future origin. In the Broker architecture, the Futures emanate from a worker, which is located on the periphery of the topology, instead of the master in the Master/slave architecture. This allows higher reliability in regard to worker faults and generally better performances due to the generic function of the Broker. Since he doesn't need to serialize nor deserialize any Future to route them around the grid, his workload consists of networking or interprocess I/O and almost no CPU processing time. This lowers the bottleneck of the Broker topology.
The Broker architecture won't stress the networking fabric and elements as much as a totally distributed topology since there is only one connection required on every worker.
Example
An introductory parallel "Hello, world!" example is implemented this way:
from scoop import futures
def hello_world(value) -> str:
return "Hello World from Future #".format(value)
if __name__ "__main__":
return_values = futures.map(hello_world, range(16))
print("\n".join(return_values))
References
External links
* {{Official website, https://github.com/soravux/scoop/
Official documentation
Distributed computing
Free science software
Python (programming language) scientific libraries
Python (programming language) software