NAME
    Win32-ProcFarm - system for parallelization of code under Win32

OVERVIEW
  What is Win32::ProcFarm?

    "Win32::ProcFarm" is the code I wrote to speed up tasks that are limited
    by network latency, but not by network bandwidth or local computer
    power. For instance, say you want to ping every address on a subnet. The
    simple approach (excluding pinging the broadcast address) is to
    sequentially ping every address on the subnet. If only 30% of the
    addresses are in use and you wait 1 second before deciding an address is
    not in use, it will take roughly 3 minutes to ping a class C subnet. The
    limitation here is obviously not the local CPU or even network
    bandwidth, but rather latency. One solution would be to break up the
    task. Unfortunately, the thread support in Perl doesn't work with
    ActivePerl, and in any event the support is currently experimental.
    Another approach would be to spin off 10 processes, have each take 25
    addresses, and funnel the information back into a single process for
    reporting.

    This is the approach "Win32::ProcFarm" takes, but it is somewhat more
    sophisticated. A "pool" of processes is created that communicate with
    the parent process using TCP sockets. The parent process communicates
    with the child processes using a "RPC" style library to assign tasks to
    the child processes and to retrieve the return data from those tasks.

    Each child process is comprised of a library file that includes the
    communications routines, as well as whatever subroutines pertain to the
    problem at hand. The parent process spins off the child process, which
    then connects back to the parent process through a TCP port. The parent
    process uses "Data::Dumper" to package up the desired subroutine name
    along with any associated parameters and ships it off to the child
    process. The child process then executes that subroutine and uses
    "Data::Dumper" to package up the return values and send them back to the
    parent. What makes the library useful is that the child process can
    operate asynchronously from the parent; the parent simply calls
    "execute" to instruct the child process to execute a subroutine. The
    parent process can then periodically call "get_state", which will return
    "wait" while the child process is still executing the subroutine. When
    the child process finishes and ships the return values back up the
    socket, the "get_state" method call on the parent object will return the
    "fin" state. The parent then calls "get_retval" to obtain the returned
    values, and the child process can then be used to execute another task.

    The pool system is based upon this simplistic "RPC" system. To use the
    "Win32::ProcFarm::Pool" object, one simply creates a new pool, passing
    it the number of child processes to start as well as the name of the
    child process and a few other parameters. Once the pool has been
    created, one adds jobs to the waiting pool. This might be a list of IP
    addresses to ping, for instance. Then one tells the
    "Win32::ProcFarm::Pool" object to execute all the jobs. The pool assigns
    a job to each of the child processes until all the child processes are
    busy. It then checks the child processes periodically to see if they
    have finished with the task. If they have, it places the return values
    into a hash, identified by an ID passed when the job was created, and
    sends the child process another job. When all the jobs have finished,
    one simply requests the hash of return values and proceeds on.

  Process Farm Advantages

    Speed
        By farming the work out over a large number of processes (I
        typically use from 5 to 30), large speedup factors can be achieved
        fairly easily.

    Reuse
        The process farm system is designed to be fairly easy to use. Simply
        write the function of use, include it in a child process, and add
        roughly 10 lines of boilerplate code to the parent.

    Efficiency in face of variable length jobs
        Because jobs are assigned one-by-one to the child processes as they
        come free, jobs are allocated as efficiently as possible given the
        constraint that the job execution time cannot be predicted.

    Low probability of child process orphaning
        Because the code to kill the child processes when everything is over
        is implemented in the "DESTROY" for the parent, orphaning is a rare
        event.

  Process Farm Limitations

    The Process Farm code is very useful in certain situations, but it has a
    number of limitations that should be kept in mind.

    Child Process Startup Time
        On a dual Pent-Pro/200 with 128MB of RAM, child process startup time
        is roughly 1/3rd of a second. This means spinning off 30 child
        processes takes 10 seconds. The code already uses asynchronous
        startup, and I believe the major limitation remaining is the time
        necessary to start up a Perl process and create the TCP socket.

    Child Process Memory Utilization
        By keeping an eye on total memory utilization, it appears that each
        bare child process uses roughly 2.3MB of memory. A child process
        that also uses "Net::Ping" to implement a ping function uses roughly
        2.6MB of memory. If you spin off 30 of these processes, that's 75MB
        of RAM. If you start swapping, the thrash of 30 processes running
        simultaneously is going to kill any speed benefit, so keep memory
        utilization in mind when selecting the number of child processes to
        use.

  Real World Results

    Despite the limitations, I have found the Process Farm system to be very
    useful. In the previous example of pinging a range of IP addresses, with
    roughly 10% coverage on a Class C, and 31 child processes, total ping
    time runs roughly 21 seconds, a speed up of a factor of 10 on a problem
    that otherwise takes an obnoxious amount of time.

  Further Information

    Please see the "tutorial" in "Docs/tutorial.pod" for more information,
    as well as the POD contained within the actual Perl modules.