Buffered CoScheduled (BCS) MPI is a novel implementation of MPI based
on global synchronization of all system activities. BCS-MPI imposes a
model where all processes and their communication are tightly
scheduled at a very fine granularity. Thus, BCS-MPI behests a system
that is much more controllable and deterministic.  BCS-MPI leverages
this regular behavior to provide a simple yet powerful monitoring and
debugging subsystem that streamlines the analysis of parallel
software. This subsystem, called Monitoring and Debugging System
(MDS), provides exhaustive process and communication scheduling
statistics. This paper covers in detail the design and implementation
of the MDS subsystem, and demonstrates how the MDS can be used to
monitor and debug not only parallel MPI applications but also the
BCS-MPI runtime system itself.  Additionally, we show that this
functionality needs not come at a significant performance loss.