Module pprocess
[hide private]
[frames] | no frames]

Module pprocess

source code


A simple parallel processing API for Python, inspired somewhat by the thread
module, slightly less by pypar, and slightly less still by pypvm.

Copyright (C) 2005, 2006, 2007, 2008, 2009 Paul Boddie <paul@boddie.org.uk>

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Software Foundation; either version 3 of the License, or (at your option) any
later version.

This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
details.

You should have received a copy of the GNU Lesser General Public License along
with this program.  If not, see <http://www.gnu.org/licenses/>.


Version: 0.5

Classes [hide private]
  Undefined
  AcknowledgementError
  Channel
A communications channel.
  PersistentChannel
A persistent communications channel which can handle peer disconnection, acting as a server, meaning that this channel is associated with a specific address which can be contacted by other processes.
  Exchange
A communications exchange that can be used to detect channels which are ready to communicate.
  Persistent
A mix-in class providing methods to exchanges for the management of persistent communications.
  ManagedCallable
A callable managed by an exchange.
  PersistentCallable
A callable which sets up a persistent communications channel.
  BackgroundCallable
A callable which sets up a persistent communications channel, but is unmanaged by an exchange.
  Map
An exchange which can be used like the built-in 'map' function.
  Queue
An exchange acting as a queue, making data from created processes available in the order in which it is received.
  MakeParallel
A wrapper around functions making them able to communicate results.
  MakeReusable
A wrapper around functions making them able to communicate results in a reusable fashion.
  PersistentExchange
An exchange which manages persistent communications.
  PersistentQueue
A queue which manages persistent communications.
Functions [hide private]
 
BackgroundQueue(address)
Connect to a process reachable via the given 'address', making the results of which accessible via a queue.
source code
 
pmap(callable, sequence, limit=None)
A parallel version of the built-in map function with an optional process 'limit'.
source code
 
_get_number_of_cores()
Return the number of distinct, genuine processor cores.
source code
 
_get_number_of_cores_solaris()
Return the number of cores for OpenSolaris 2008.05 and possibly other editions of Solaris.
source code
 
create_socketpair()
Create a new process, returning a communications channel to both the creating process and the created process.
source code
 
create_pipes()
Create a new process, returning a communications channel to both the creating process and the created process.
source code
 
create()
Create a new process, returning a communications channel to both the creating process and the created process.
source code
 
get_number_of_cores()
Return the number of distinct, genuine processor cores.
source code
 
create_persistent(address)
Create a new process, returning a persistent communications channel between the creating process and the created process.
source code
 
connect_persistent(address)
Connect via a persistent channel to an existing created process, reachable at the given 'address'.
source code
 
exit(channel)
Terminate a created process, closing the given 'channel'.
source code
 
start(callable, *args, **kw)
Create a new process which shall start running in the given 'callable'.
source code
 
start_persistent(address, callable, *args, **kw)
Create a new process which shall be reachable using the given 'address' and which will start running in the given 'callable'.
source code
 
close_streams()
Close streams which keep the current process attached to any creating processes.
source code
 
waitall()
Wait for all created processes to terminate.
source code
Variables [hide private]
  _cpuinfo_fields = ('processor', 'physical id', 'core id')
  __package__ = None
Function Details [hide private]

pmap(callable, sequence, limit=None)

source code 

A parallel version of the built-in map function with an optional process
'limit'. The given 'callable' should not be parallel-aware (that is, have a
'channel' parameter) since it will be wrapped for parallel communications
before being invoked.

Return the processed 'sequence' where each element in the sequence is
processed by a different process.

_get_number_of_cores()

source code 

Return the number of distinct, genuine processor cores. If the platform is
not supported by this function, None is returned.

create_pipes()

source code 

Create a new process, returning a communications channel to both the
creating process and the created process.

This function uses pipes instead of a socket pair, since some platforms
seem to have problems with poll and such socket pairs.

get_number_of_cores()

source code 

Return the number of distinct, genuine processor cores. If the platform is
not supported by this function, None is returned.

create_persistent(address)

source code 

Create a new process, returning a persistent communications channel between
the creating process and the created process. This channel can be
disconnected from the creating process and connected to another process, and
thus can be used to collect results from daemon processes.

In order to be able to reconnect to created processes, the 'address' of the
communications endpoint for the created process needs to be provided. This
should be a filename.

start(callable, *args, **kw)

source code 

Create a new process which shall start running in the given 'callable'.
Additional arguments to the 'callable' can be given as additional arguments
to this function.

Return a communications channel to the creating process. For the created
process, supply a channel as the 'channel' parameter in the given 'callable'
so that it may send data back to the creating process.

start_persistent(address, callable, *args, **kw)

source code 

Create a new process which shall be reachable using the given 'address' and
which will start running in the given 'callable'. Additional arguments to
the 'callable' can be given as additional arguments to this function.

Return a communications channel to the creating process. For the created
process, supply a channel as the 'channel' parameter in the given 'callable'
so that it may send data back to the creating process.

Note that the created process employs a channel which is persistent: it can
withstand disconnection from the creating process and subsequent connections
from other processes.