From bresnaha at mcs.anl.gov Mon Jul 3 08:44:33 2006 From: bresnaha at mcs.anl.gov (John Bresnahan) Date: Mon Jul 3 08:44:11 2006 Subject: [SC06xmon] internalized summarizer API In-Reply-To: <4499DA4D.10604@lbl.gov> References: <4499DA4D.10604@lbl.gov> Message-ID: <44A93B61.1030500@mcs.anl.gov> i am trying to get to this today. sorry about the delay i have been switched onto an emergency deadline project and getting up to speed on it has taken all of my time. a file seems to be missing from the distro: | configure: error: cannot find sources (src/nl_log.h) in . or .. Dan Gunter wrote: > Hi, > > This is mostly for John Bresnahan, but I figured it would be good to let > everyone know the progress. I finally got through adding code to > NetLogger so the "summarization" tool is integrated with the NetLogger C > API. Code examples can be found in c/src/perf/ioperf.c and its config > files perf-config.{server,client}, and also c/src/perf/eventrate.c . It > just occurred to me that I may not have put the right automake magic to > get the configuration files into the distro., so I'm attaching them to > this email. Oh, joy, now you can all read the XML fun-ness. > > Preliminary performance results, after some optimization, are good. > Summarizing events is about 50% faster than writing them to /dev/null, > and a lot more informative :-) But I am eager to integrate this code > with the XIO code to see if the performance results pan out in practice. > Based on what I've done so far, I am hopeful that NetLogger activity > overlaps enough with I/O that the only perturbation we see is from the > occasional disk I/O of writing out the summary records (which we may be > able to minimize by using a separate disk for logs). > > Like before, the basic idea is that a configuration file (in XML -- "we > drank the kool-aid" as Martin would say) sets up everything about the > summarization, and you just need to point at that configuration file > when you start up NetLogger. The XIO instrumentation needs to be > rewritten to use the higher-level C API, but this should be a relief, > since it's pretty close to printf(). > > Clearly, we still need to push the summary records through Martin's > CDF-o-matic. But that part of the problem hasn't changed (for better or > worse), unless Martin knows something he's not telling us. > > -Dan > > I placed a tarball of NetLogger that contains the new code at: > > http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz > > This is "beta" software, so please expect crashes and the like the first > few times. > Also, thanks to XML configuration, you need to have libxml2 on your > systems. > > > ------------------------------------------------------------------------ > > > > EVNT > " > 100000 > csumm > value > > > > > sock > read.start > read.end > > > > > sock > write.start > write.end > > > > > ------------------------------------------------------------------------ > > > > EVNT > " > 100000 > iosumm > nbytes > > > > > fd > read.start > read.end > > > > > fd > write.start > write.end > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > SC06xmon mailing list > SC06xmon@george.lbl.gov > http://dsd.lbl.gov/mailman/listinfo/sc06xmon From bresnaha at mcs.anl.gov Mon Jul 3 08:44:33 2006 From: bresnaha at mcs.anl.gov (John Bresnahan) Date: Mon, 03 Jul 2006 10:44:33 -0500 Subject: [SC06xmon] internalized summarizer API In-Reply-To: <4499DA4D.10604@lbl.gov> References: <4499DA4D.10604@lbl.gov> Message-ID: <44A93B61.1030500@mcs.anl.gov> i am trying to get to this today. sorry about the delay i have been switched onto an emergency deadline project and getting up to speed on it has taken all of my time. a file seems to be missing from the distro: | configure: error: cannot find sources (src/nl_log.h) in . or .. Dan Gunter wrote: > Hi, > > This is mostly for John Bresnahan, but I figured it would be good to let > everyone know the progress. I finally got through adding code to > NetLogger so the "summarization" tool is integrated with the NetLogger C > API. Code examples can be found in c/src/perf/ioperf.c and its config > files perf-config.{server,client}, and also c/src/perf/eventrate.c . It > just occurred to me that I may not have put the right automake magic to > get the configuration files into the distro., so I'm attaching them to > this email. Oh, joy, now you can all read the XML fun-ness. > > Preliminary performance results, after some optimization, are good. > Summarizing events is about 50% faster than writing them to /dev/null, > and a lot more informative :-) But I am eager to integrate this code > with the XIO code to see if the performance results pan out in practice. > Based on what I've done so far, I am hopeful that NetLogger activity > overlaps enough with I/O that the only perturbation we see is from the > occasional disk I/O of writing out the summary records (which we may be > able to minimize by using a separate disk for logs). > > Like before, the basic idea is that a configuration file (in XML -- "we > drank the kool-aid" as Martin would say) sets up everything about the > summarization, and you just need to point at that configuration file > when you start up NetLogger. The XIO instrumentation needs to be > rewritten to use the higher-level C API, but this should be a relief, > since it's pretty close to printf(). > > Clearly, we still need to push the summary records through Martin's > CDF-o-matic. But that part of the problem hasn't changed (for better or > worse), unless Martin knows something he's not telling us. > > -Dan > > I placed a tarball of NetLogger that contains the new code at: > > http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz > > This is "beta" software, so please expect crashes and the like the first > few times. > Also, thanks to XML configuration, you need to have libxml2 on your > systems. > > > ------------------------------------------------------------------------ > > > > EVNT > " > 100000 > csumm > value > > > > > sock > read.start > read.end > > > > > sock > write.start > write.end > > > > > ------------------------------------------------------------------------ > > > > EVNT > " > 100000 > iosumm > nbytes > > > > > fd > read.start > read.end > > > > > fd > write.start > write.end > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > SC06xmon mailing list > SC06xmon at george.lbl.gov > http://dsd.lbl.gov/mailman/listinfo/sc06xmon