PICL supports both host/node and hostless programming models. The PICL
versions of the two programming models are described below. Most of the
material on the host/node programming model is a simplified version of what
is available in creference.shar and userguide.shar. It is included here to
provide background for the explanations of the hostless model.

HOST/NODE
---------

HOST: In the host/node programming model, a host process is used to allocate
node processors in the multiprocessor and to load node executables onto these
processors. A host process first calls open0 to do the allocation, calls
load0 to load a node executable onto one or all allocated nodes, and calls
close0 to release the processors. In order to enable tracing and to define a
file where trace data should be saved, the host prcess calls traceenable. If
the host process itself is to be traced, then tracehost and tracelevel must
also be called. A common structure for a host program is the following. (See
creference.shar and/or userguide.shar for descriptions of these commands.)

_______________________________________________________________________________

/* get user input, including tracing parameters and number of processors */
/* to allocate: tracing, filename, verbose, hosttracing, tracesize, flush, */
/* picl, user, trace, nprocs */
...

/* if tracing is desired, enable it */
if (tracing) traceopen(filename, verbose);

/* if tracing on the host is desired, start tracing */
if (hosttracing){
  tracehost(tracesize, flush);
  tracelevel(picl,user,trace);
  };

/* allocate processors */
numprocs = nprocs;
open0(&numprocs, &me, &host);

/* load all processors with node program (ALLNODES = -1) */
load0("nodeprogram",ALLNODES);

/* send data in params[6] and data[i][j] to processors */
params[1] = tracing;
params[2] = tracesize;
params[3] = verbose;
params[4] = picl;
params[5] = user;
params[6] = trace;
for (i=0; i<n; i++){
  params[0] = datasize[i];
  send0(&params,7*sizeof(int),0,i);
  send0(&data[i][0],datasize[i]*sizeof(float),1,i);
  };

/* receive results from processors in results[i][j] */
for (i=0; i<n; i++) recv0(&results[i][0],resultssize[i]*sizeof(float),i);

/* output results */
...

/* release node processors */
close0(1);
_______________________________________________________________________________

Generally, communication between the host process and the node processes
is relatively slow, and the role of the host process in the computation
should be minimized. 

NODE: The node processes generally do most of the work in parallel
applications. The node process begins by calling open0 to enable
interprocessor communication and to find out how many processors are
allocated, the node number of the processor this process is executing on and
the host identification number. The last executable instruction should be a
call to close0. If tracing is desired, then tracenode and tracelevel should be
called. A common structure for a node program is the following. (See
creference.shar and/or userguide.shar for a description of these commands.)

/* enable interprocessor communication and find out who I am */
open0(&numprocs, &me, &host);

/* receive data from the host in data[j] */
recv0(params, 7*sizeof(int), 0);
datasize = params[0];
tracing = params[1];
tracesize = params[2];
verbose = params[3];
picl = params[4];
user = params[5];
trace = params[6];
recv0(&data, datasize*sizeof(float), 1);

/* if tracing desired, start tracing */
if (tracing){
  tracenode(tracesize, flush, sync);
  tracelevel(picl, user, trace);
};

/* do computation, communicating with other nodes as need be */
...

/* if tracing, stop */
traceexit();

/* send results back to host */
send0(&results[0], resultssize*sizeof(float), me);

/* send back trace data, disable communication, and exit */
close0();

In this pseudocode, tracing is started after the data for the host is
received and is stopped before the the results are sent back to the host.
This is not necessary, but is commonly done when producing trace files for
use with ParaGraph. (The communication with the host will show up in the
animation as a long period during which nothing interesting is happening.)

GENERALIZATIONS: A host process is not required on many systems, and, even
when one exists, it need not be involved in the collection of trace data. The
following two commands support this generality. 

1) host0() returns a 1 if the process was spawned by a host, and returns a 0
otherwise. This is not equivalent with there being a host process or not, but
it is usually sufficient. (On some systems, the node processes can be spawned
"by hand" and still communicate with a host process.) A common use for the
routine is the following: 

if (host0()){ /* send results to host */
  send0(...);
  }
  else{ /* send results to standard output */
  printf(...);
  fflush(stdout);
  };

Thus, a single node program can be written that will run whether there is a
host process or not. Note that a multiprocessor system that supports the
hostless programming model must support the node processes writing directly
to standard out and to disk files. 

2) tracefiles(tempfile, permfile, verbose) is used to specify (or override)
where trace data is sent. For example, if traceenable is used to specify that
trace data is to be stored in a file called "trace", then the call 

tracefiles("temp","perm",verbose);

will instead cause trace data from this process to be sent to a file called
"perm". This file can be the same as that specified by the other nodes or
different. The data is added in append mode, so either is legal. (Because
append mode is used, trace files from previous runs of the parallel code will
be appended to, not overwritten, so it is important to delete or rename old
trace files.) How the data is saved (in verbose or compact form) is also
determined by the verbose parameter in the tracefiles call. The command only
works on systems that allow node processes to open files and write directly
to disk. On machines where this is not possible, i.e. machines that support
only the host/node programming model, this command will "fail" and the trace
data will be sent back via the host as if tracefiles had never been called.
Note that if permfile is the NULL string, then an alternative tracefile is
not created, and trace data is still sent back to the host. 

The tempfile parameter is used to specify a disk file where "flushed" trace
data is to be saved. For example, if the flush option is set to 2 in
tracenode, then the trace data is normally sent to the host when the local
trace buffer fills up. This can be very expensive, and is normally
discouraged. To make flushing less expensive, the flushed trace data can now be
sent to a disk file. (On the iPSC/860 and the Delta multiprocessors, disk
access can be relatively inexpensive.) If not NULL, the tempfile string is
the prefix of the name of the file used for this temporary storage. The rest
of the name is constructed from the node number, so that every process has
a unique temporary file. This file is unlinked before exiting, so it is
deleted when the node process terminates. Traceflushing, both automatic and
explicit, is always expensive. But, if it must be done, then this facility
makes it somewhat cheaper on systems where node processes can write directly
to disk. If "tempfile" is the null string, then a disk file is not created,
and trace data will be sent to "permfile" or the host during flushing. 

The tracefiles command can occur either before or after the call to tracenode.
If it is called afterwards, a trace record recording the call will be saved
in the trace file. The trace file can also be changed during the execution
of the node process by calling tracefiles multiple times. These calls will
significantly perturb the performance of the node process, and should not be
done cavalierly. NOTE: The host process should call traceenable whenever any
node processes are tracing. The host coordinates the saving of the trace data
even when tracefiles is called in the node process(es). If traceenable is not
called, then the host process does not send the required messages to the
nodes, and the trace data is never saved. (If there is no host process, then
the node processes use a different algorithm to decide when to save the trace
data.) 

HOSTLESS
--------

In the hostless programming model, there is no host process. The user
allocates processors and spawns processes using system calls, e.g. getcube
and load on the iPSC series, mexec on the Delta, and xnc on the nCUBE/2. For
most node programs, the only change required to go from a host/node to a
hostless programming model is to remove all interprocessor communication with
the host. Therefore, node processes must be able to read from standard input or
disk files to get data and write to standard output or disk files to save
results. While there are no disk i/o commands in PICL, there is also nothing
to prevent node processes from calling the native commands to do disk i/o.
Since most of these commands are standard C and FORTRAN calls, portability is
usually retained. In order to write truly portable node programs, it is
important to be able to identify whether there is a host process. To support
this, the function host0() has been included in PICL. See the description of
the command above, in the HOST/NODE section. 

One other complication of the hostless programming model is that without a
host, there is no longer a default process to collect, interpret, and save
trace data. In consequence, the command tracefiles has been added (see
HOST/NODE section) to indicate where trace data is to be stored.

While host0 and tracefiles are sufficient to "port" most node programs to a
hostless environment, the hostless programming model is still less flexible
than the host/node programming model, as indicated by the following
limitations. 

1) The host can no longer be used to define a logical number of allocated
processors that differs from the actual number of processors allocated.
Therefore, only a power-of-two number of processors can be allocated
on the nCUBE/2, iPSC/2, and iPSC/860 multiprocessors.

2) The setarc0 command can no longer be used to redefine the underlying
topology and the algorithms used by the port routines. Instead, a default
topology and default algorithms are used. To remove this restriction would
require that the port routines be redesigned to allow the node processess
to redefine the topology. This has not yet been done.

3) All allocated nodes must be loaded with PICL processes for tracing to work
(since the tracing logic uses a barrier to prevent traceflushing from
perturbing the performance of the parallel program). 

Note that these are not PICL limitations. Rather, they are limitations caused
by not having a host process, and they disappear if a host process is used.
While these limitations will not affect most PICL programs, they need to be
kept in mind when deciding to do without a host process.