From bresnaha at mcs.anl.gov Mon Jul 3 08:44:33 2006
From: bresnaha at mcs.anl.gov (John Bresnahan)
Date: Mon Jul 3 08:44:11 2006
Subject: [SC06xmon] internalized summarizer API
In-Reply-To: <4499DA4D.10604@lbl.gov>
References: <4499DA4D.10604@lbl.gov>
Message-ID: <44A93B61.1030500@mcs.anl.gov>
i am trying to get to this today. sorry about the delay i have been
switched onto an emergency deadline project and getting up to speed on
it has taken all of my time.
a file seems to be missing from the distro:
| configure: error: cannot find sources (src/nl_log.h) in . or ..
Dan Gunter wrote:
> Hi,
>
> This is mostly for John Bresnahan, but I figured it would be good to let
> everyone know the progress. I finally got through adding code to
> NetLogger so the "summarization" tool is integrated with the NetLogger C
> API. Code examples can be found in c/src/perf/ioperf.c and its config
> files perf-config.{server,client}, and also c/src/perf/eventrate.c . It
> just occurred to me that I may not have put the right automake magic to
> get the configuration files into the distro., so I'm attaching them to
> this email. Oh, joy, now you can all read the XML fun-ness.
>
> Preliminary performance results, after some optimization, are good.
> Summarizing events is about 50% faster than writing them to /dev/null,
> and a lot more informative :-) But I am eager to integrate this code
> with the XIO code to see if the performance results pan out in practice.
> Based on what I've done so far, I am hopeful that NetLogger activity
> overlaps enough with I/O that the only perturbation we see is from the
> occasional disk I/O of writing out the summary records (which we may be
> able to minimize by using a separate disk for logs).
>
> Like before, the basic idea is that a configuration file (in XML -- "we
> drank the kool-aid" as Martin would say) sets up everything about the
> summarization, and you just need to point at that configuration file
> when you start up NetLogger. The XIO instrumentation needs to be
> rewritten to use the higher-level C API, but this should be a relief,
> since it's pretty close to printf().
>
> Clearly, we still need to push the summary records through Martin's
> CDF-o-matic. But that part of the problem hasn't changed (for better or
> worse), unless Martin knows something he's not telling us.
>
> -Dan
>
> I placed a tarball of NetLogger that contains the new code at:
>
> http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz
>
> This is "beta" software, so please expect crashes and the like the first
> few times.
> Also, thanks to XML configuration, you need to have libxml2 on your
> systems.
>
>
> ------------------------------------------------------------------------
>
>
>
> EVNT
> "
> 100000
> csumm
> value
>
>
>
>
> sock
> read.start
> read.end
>
>
>
>
> sock
> write.start
> write.end
>
>
>
>
> ------------------------------------------------------------------------
>
>
>
> EVNT
> "
> 100000
> iosumm
> nbytes
>
>
>
>
> fd
> read.start
> read.end
>
>
>
>
> fd
> write.start
> write.end
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> SC06xmon mailing list
> SC06xmon@george.lbl.gov
> http://dsd.lbl.gov/mailman/listinfo/sc06xmon
From bresnaha at mcs.anl.gov Mon Jul 3 08:44:33 2006
From: bresnaha at mcs.anl.gov (John Bresnahan)
Date: Mon, 03 Jul 2006 10:44:33 -0500
Subject: [SC06xmon] internalized summarizer API
In-Reply-To: <4499DA4D.10604@lbl.gov>
References: <4499DA4D.10604@lbl.gov>
Message-ID: <44A93B61.1030500@mcs.anl.gov>
i am trying to get to this today. sorry about the delay i have been
switched onto an emergency deadline project and getting up to speed on
it has taken all of my time.
a file seems to be missing from the distro:
| configure: error: cannot find sources (src/nl_log.h) in . or ..
Dan Gunter wrote:
> Hi,
>
> This is mostly for John Bresnahan, but I figured it would be good to let
> everyone know the progress. I finally got through adding code to
> NetLogger so the "summarization" tool is integrated with the NetLogger C
> API. Code examples can be found in c/src/perf/ioperf.c and its config
> files perf-config.{server,client}, and also c/src/perf/eventrate.c . It
> just occurred to me that I may not have put the right automake magic to
> get the configuration files into the distro., so I'm attaching them to
> this email. Oh, joy, now you can all read the XML fun-ness.
>
> Preliminary performance results, after some optimization, are good.
> Summarizing events is about 50% faster than writing them to /dev/null,
> and a lot more informative :-) But I am eager to integrate this code
> with the XIO code to see if the performance results pan out in practice.
> Based on what I've done so far, I am hopeful that NetLogger activity
> overlaps enough with I/O that the only perturbation we see is from the
> occasional disk I/O of writing out the summary records (which we may be
> able to minimize by using a separate disk for logs).
>
> Like before, the basic idea is that a configuration file (in XML -- "we
> drank the kool-aid" as Martin would say) sets up everything about the
> summarization, and you just need to point at that configuration file
> when you start up NetLogger. The XIO instrumentation needs to be
> rewritten to use the higher-level C API, but this should be a relief,
> since it's pretty close to printf().
>
> Clearly, we still need to push the summary records through Martin's
> CDF-o-matic. But that part of the problem hasn't changed (for better or
> worse), unless Martin knows something he's not telling us.
>
> -Dan
>
> I placed a tarball of NetLogger that contains the new code at:
>
> http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz
>
> This is "beta" software, so please expect crashes and the like the first
> few times.
> Also, thanks to XML configuration, you need to have libxml2 on your
> systems.
>
>
> ------------------------------------------------------------------------
>
>
>
> EVNT
> "
> 100000
> csumm
> value
>
>
>
>
> sock
> read.start
> read.end
>
>
>
>
> sock
> write.start
> write.end
>
>
>
>
> ------------------------------------------------------------------------
>
>
>
> EVNT
> "
> 100000
> iosumm
> nbytes
>
>
>
>
> fd
> read.start
> read.end
>
>
>
>
> fd
> write.start
> write.end
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> SC06xmon mailing list
> SC06xmon at george.lbl.gov
> http://dsd.lbl.gov/mailman/listinfo/sc06xmon