By using the flocking and glide-in mechanisms provided by Condor, we were able to bring together a computational pool consisting of 2510 processors from various locations and of varying characteristics. Table 1 shows the number and type of processors at each participating site.

Interesting facts about the participating machines:

• The 1024 SGI/Irix processors at NCSA currently rank that supercomputer as the 52nd fastest in the world, according to Top500.org. However, this is a very heavily used machine, and we were able to acquire at most 41 processors at any one time.
• The Linux machines at Argonne are part of the new Chiba City cluster.
• The machines at Georgia Tech are part of the Interactive High Performance Computing Lab.
• The computers in Italy are part of the Italian "Computational Grid" and as such, were spread throughout the entire country : Rome, Bologna, Padova, Milan, and Naples.

• The total number of workers
• The total number of workers on June 12th
• The total number of workers from 4AM to 8AM on June 12th
• The number of workers participating at Argonne, Wisconsin, and Georgia Tech
• The number of workers participating at INFN, NCSA (Origin 2000), and New Mexico
• The number of workers participating at Columbia, Northwestern, and NCSA (Condor pool)
• KLAPS (Thousands of Linear Assignment Problems solved per second)
• NOPES (Nodes evaluated per second)
• The equivalent number of HP-C3000 workstations participating

On June 8, 2000 at 11:05:36 CDT, Jean-Pierre Goux and Jeff Linderoth started running the MW-QAP code on nug30, logging in remotely to Jeff's Personal Condor pool at the University of Wisconsin-Madison. The computation completed on June 15, at 21:20:07 CDT, after which there was much rejoicing.

The nug30 computation was stopped five times during the week for various reasons:

• 6/09 14:23:29 Job was manually stopped so that new master process, capable of dealing with more than 1000 workers could be used
• 6/13 03:36:40 Job terminated due to bug in Condor schedd code (now fixed in the newest version of Condor)
• 6/13 14:12:51 Job was manually stopped at the request of the master machine's system administrator. The machine needed to be rebooted to fix (unrelated) NFS problems that were causing problems for other users
• 6/14 19:23:46 Job terminated due to bug in Condor schedd code. (The same bug that caused the second program termination)
• 6/15 11:27:42 Job was manually stopped to quickly remove a number of worker submissions that had be incorrectly made. The incorrect submissions were the result of Jeff incorrectly editing a configuration file.

After each termination, the computation was restarted from a checkpoint that was taken every 15 minutes during the run. (Thus at most 15 minutes of computation time was lost). Although no one is happy when bugs are present or human error occurs, these things will happen. Adding program robustness (in the form of checkpointing) was critical to the success of solving nug30.

In order to prove the optimality of this solution, 11,892,208,412 nodes of a branch and bound tree were explored. Solving the associated node subproblems and computing the branching information required 574,254,156,532 Frank-Wolfe iterations.

On average, there were 653 machines participating in the computation, with a maximum of 1009. One of the most remarkable features of the run was that almost 1 million linear assignment problems (LAPs) were solved each second during the course of the run. (One LAP must be solved for each Frank-Wolfe iteration). Table 2 shows a number of other interesting statistics about the nug30 run and the computational pool. The machine speeds have been normalized to an HP-C3000 workstation by comparing the time required for each participating machine to compute the same portion of the branch and bound tree. (Thus the "average" machine used in the nug30 computation was 56% as fast as an HP-C3000).

Table 3 shows the percentage of the work done at each participating location.

Table 4 shows the percentage of the work done by machines of each operating system and architecture type.

• Jean-Pierre and Jeff starting the nug30 computation
• Jean-Pierre, Jeff, Steve Wright, and Jeff's wife Helen out for celebratory beers upon the succesful completion of the nug30 computation.