dispy : Python Framework for Distributed and Parallel Computing

dispyscheduler

As mentioned in dispy and Project Page, if jobs are submitted with JobCluster instead of SharedJobCluster, then dispyscheduler is not needed; JobCluster includes the scheduler. If there is a need to use same node by more than one cluster at the same time (e.g., multiple client programs need to run simultaneously), then SharedJobCluster must be used, along with running 'dispyscheduler' on a node. Usually, no options are needed when invoking this program.

Below are various options to invoking dispyscheduler:

NAT/Firewall Forwarding

As explained in dispy and dispynode documentation, 'ext_ip_addr' can be used to use services behind NAT firewall/gateway. This option can be used with dispyscheduler, too. This is especially useful if there are many nodes in a network behind NAT firewall/gateway (otherwise, as explained in dispynode documentation, each dispynode should be started with a different port and all those ports forwarded appropriately). Assuming that dispyscheduler is to run on a node with (private) IP address 192.168.20.55 and it is behind NAT firewall/gateway at (public) IP address a.b.c.d, dispyscheduler can be invoked as dispynode.py -i 192.168.20.55 --ext_ip_addr a.b.c.d and setup NAT to forward UDP and TCP ports 51347 and TCP port 51349 to 192.168.20.55. Then dispy clients can use nodes in this network with cluster = SharedJobCluster(compute, nodes=['*'], scheduler_node='a.b.c.d')