Items brought up at May 2002 HUG Meeting which the HEASARC should address (compiled by Steve Drake 05/08/2002: please send updates/corrections to him) T= Them (HUG) comment or question U= HEASARC reply or action Notice: I have not listed interoperability/compatability concerns here that really fall under the purview of the HEADCC rather than the HUG (1) T: We should investigate off-site storage of copy of HEASARC Archive. U: Nick should address this. Phil comments: "Are we talking about offline or online storage offsite? There are a number of things that could be done, but there is no money to do them. Does the HUG consider the NSSDC MOU insufficient?? I'm not sure of the scope of this proposed investigation." Steve D. comments: "There was a concern that if someone nukes GSFC, there goes the HEASARC Archive and all its copies. I checked with Beth Brown of NSSDC and they keep the tapes and CD's that we send them onsite also. Maybe the cheapest thing to do is to see whether the NSSDC material can be moved offsite (Laura Whitlock's basement?)." (2) T: Why the discrepancy in stats for Imagine the Universe site? U: Steve's number of >4 million is based on number of requested files, Jim's number of 0.5 millions is likely based on # of pages; remember that to download a page, a number of files are typically needed (apparently an average of 8 per page for this site). Steve will discuss this with Steve Fantasia to confirm. (3) T: Plot applet for Browse sometimes fails and/or hangs web browser. U: Tom and Christina should check this. (4) T: The search radius for cross-correlations cannot be adjusted or defaults to previous value. U: A bit obscure; Tom and Christina should check this or try and get clarification. (5) T: Exposure time is a very important parameter, and should be a default display parameter in all database tables in which it occurs. U: Ed should make it so. (6) T: In the ascii table option for results in Browse, the user should have the option of selecting the delimiter character (| should be default), with a blank being a possible value. U: Tom & Christina should make it so. (6.1)T: It would be nice to have an option for results in Browse to be put into a spreadsheet, e.g., an Excel table. U: We agree Tom comments: "During the meeting we mostly talked about how we might play with the text format to make it easier to import into Excel. Christina has just spent a couple of days looking at a more direct approach, providing a direct Excel-format output from Browse. The development version of Browse at http://olegacy/cgi-bin/w3bdev/W3Browse/w3browse.pl now has an Excel compatible format as one of its output format selections. If you run Browse from a PC or Mac you can pop up an Excel window directly from the browser. Otherwise one can save the file and copy it to another machine. This could be quite a useful capability and we'd be interested in comments about the general idea and the implementation. There are a couple of glitches that need to be worked out. Most notably, we have a blank top worksheet -- each table retrieved is placed in it's own worksheet but in general things seem to be working quite nicely." Mike C.'s comments: "it looks great and I think this is extremely useful already. One thing you might try is to embed links in the spreadsheet - for example the column headers could be linked to the help files; rosat sequence id's could be linked to the data - could you include links to the services too - that would be really neat one minor thing - the dates seem to be given in (I'm guessing) MJD, for example public_date = 50841. That's fine (makes it nice and sortable and calculable) but maybe this should be documented?" Tom's response: "Putting the links for the help is easy enough. Christina also thought we might use the top sheet which is currently empty, to give help, which might include a note on the format of dates and position fields." Nick's feedback: "- people will not notice it, because it is buried in the output selection menu, maybe these should be check boxes - the first spreadsheet should have the details of the query on it... - one thing that was asked for was a means to download the entire catalog into excel, maybe we should provide that as an addition to the tdat files (i.e. put them already made in the same ftp area, and have a link to that from the browse page or the browse results page)." Tom's reply: "I agree it could be hard to find but checkboxes will use a lot of real estate... This would be the fifth distinct display format (HTML table, Text table, XML, FITS, Excel). I'm hoping that with this format we can avoid putting in more options like 'text separator' and such. These aren't hard to do, but just make the interface more and more complex. One random thought I've had that might help this is provide a little tip at the beginning of each Browse session. E.g., randomly select "To query an entire table ..." "To get data in Excel ..." "You can get a GIF of a plot by ..." "To cross correlate two complete tables ..." "To list all tables by mission use ..." "The Browse Batch interface ..." ... to advertise a number of the less well known features of Browse. The frequent users of Browse would then see all of these. We'd have all of the tips on some easy to read page so that a user who remembered there was a tip about reading Excel data but not what it was could refresh their memory. > > - the first spreadsheet should have the details of the query on it... ... plus some explanations of the formats of the data and warnings about Excel limits (256 columns, 65536 rows). > > - one thing that was asked for was a means to download the entire > catalog into excel, maybe we should provide that as an addition to the > tdat files (i.e. put them already made in the same ftp area, and have > a link to that from the browse page or the browse results page) I'd been thinking of putting the capability to download an entire table on the Table Index page -- allowing any format. It would be easy to do there, but that's a fairly obscure page -- not that the TDAT files are better known. It should be fairly easy to write a TDAT to Excel converter so that we could add that to the process that updates the TDAT files. Probably easy enough to do both. There are a fair number of tables that exceed the 64K row limit. My initial thought is just to give the first 64K rows for them. The alternative is to split the table into multiple spreadsheets." (7) T: Some mission database tables, e.g., XMM-Newton tables, have no class parameter field so that, for example, to find out how many AGN or CVs that have been observed is not simple, but requires finding a table of AGN or CVs, and doing a cross-correlation. U: This is a function of what info the mission has provided in their tables: XMM-Newton did not provide us classes (even though they did, I believe, ask proposers for them in their proposal submission process) in their tables, and that's why they aren't there. (8) T: It would be very useful if there were an explicit high-level option to "Download All Rows of Table" in Browse (it is possible right now but requires going down to the Parameter Search level). U: This seems like a reasonable thing to do, but (i) should maybe be restricted to HEASARC tables only, and (ii) caveats should be given that this might result in a large download: it would be useful to explicitly list how many rows there are in the table, for example, so that the user would know how much (s)he was going to get back. (9) T: It would be good to expand and better document the HEASARC's nh webtool, e.g., give the original reference, list caveats such as the relatively low spatial resolution, the possibility of underestimates of total H column density in regions towards the Galactic Plane and molecular clouds where a significant fraction of the H may be molecular rather than atomic, adding other sources for nh such as the Elvis et al. (1994, ApJS, 95, 413) NEP survey, the various Lockman Hole surveys, the Elvis et al. (1989, AJ, 97, 777) and Murphy et al.(1196, ApJS, 105, 369) values towards 173 and 220 AGN and quasars, respectively, the Stark et al. (1992, ApJS, 79, 77) Bell Labs Survey, the Hartman WSRT Survey, etc. U: The reference is actually given on the main Nh page and on the nh Help page; the latter discusses the 1 x 1 degree gridding used. Caveats can certainly be given more prominence. Adding other diverse sources for nh values is more problematical: it would be hard to combine pencil-beam surveys such as the AGN ones with all-sky or wide-angle ones. Possibilities include: (i) Making a Browse table or tables of Elvis et al & Murphy et al. and have a link on the nh page which would spawn a pre-formatted query to these tables. (ii) We could fairly quickly add the Hartman data to our tool [Steve Snowden has a copy of this dataset and the IRAS 100 micron data formatted in the same grid that he had the soft X-ray background data], and probably the Stark data. (iii) It's not clear to me what the best way to add the regional data such as the NEP and LH surveys would be. (10) T: It would be very useful to have an option when using the soft X-ray background tool to generate an output data of the model say for xspec. U: Steve Snowden says that he is writing an ftool to do something like this that should be ready in another few months. Maybe we could do this on the Web tool by having an option which would send a preformatted call to WebSpec? (11) U: Keith Arnaud presented an open question about whither XSPEC should go after version 12 is released (12) U: Steve Drake issued a call for input on the new HEASARC CookBook material on the Web. The HUG suggested that the X-Ray Summer School might be a good venue to test it, and Ilana Harrus has agreed to do this. The HEASARC booth at the Summer AAS Meeting might also be a good place and time. Rita Sambruna will have students working with her this summer who would also be good guinea pigs. (13) T: The Integral simulator is difficult to install. U: Chris should follow up on this. Chris comments: "I have little control over "OSIM", which is a collaborative effort between ISDC and the INstrument teams. I did serve as a beta tester, and gave them lots of feedback prior to its release (Ken did also). I believe they are aware of the problems, but with the early AO release, they simply could not fix it in time; it was of litte use. What I can do here, certainly in advance of the next AO is implement XSPEC/WebSPEC support for SPI (for which we are developing resonse and background models), and hopefully implement IBIS in some more approximate way as well. This does'nt address imaging though." (14) T: Are Integral calibration data publically available, and, if not, when will they? U: Chris should follow up on this. Chris comments: "Its a matter of project policy - I could easily put a large number of SPI calibration spectra, in PHA or other simple format, on my FTP site now. As far as the official ISDC archive, its a policy decision as to when it becomes public." (15) T: There were several questions about the large amount of RXTE data downloaded: (i) How many people download RXTE data, is it a lot or is it dominated by a small number of expert users? (ii) What type of RXTE files do users download? Are they unnecessarily downloading the kitchen sink, when all they really need is a much smaller subset of files? U: Steve will follow up on this in consultation with Alan. Steve will (again) try to organize a meeting this month to discuss these and other RXTE Archive and table issues (16) T: A general comment was made about improving the notification of bugs or other problems to users of Xspec (presumably this could apply to other HEAsoft software), e.g., by having an option that, if enabled, when a user started up am Xspec session would query our site for bug reports released since the last time this user started up, and relay them to the user. Specific comments/requests were made concerning APEC models such as how up-to-date are they, or can they be tuned by the user, e.g., to let him/her add missing lines or change specific atomic data parameters? U: Keith is considering the bug notification idea. He noted that the APEC models used in Xspec are created by a code that is publically available, although it might not be trivial for the typical user to make changes and/or additions to its input data. (17) T: There is an urgent need to get the HETE-2 data in documented FITS formats which are compatible with existing software tools. U: We agree. Mike Corcoran has been actively trying to get the binary file formats from MIT so he can create FITS file versions of them, and will keep trying. On May 16th Mike sent this comment: "Here is an update on the HETE2 archive. I've been trying to figure out the MIT processing pipeline and it's pretty slow going but I'm making a bit of progress. I've copied over the relevant solaris binaries & scripts to /FTP/hete2/ops, and (running on circe) can now split out the instrument data from the archived telemetry files (in /FTP/hete2/data/Archive). That's the good news. The bad news is that the split files are still in ipp format. I've identified scripts which (apparently) create postscript lightcurves from the split gamma-ray data but haven't yet figured out how to run them - the problem is in trying to (partially) reproduce the MIT processing environment, setting paths, etc, and most of the scripts are designed to deal with the daily telemetry data, not the data in the archive (which requires some modification to the paths, which need to be modified anyway to run here and not at MIT!). The idea (if MIT is comfortable with it) is to split out the instrument data from the telemetry files, then convert the split ipp data to either ascii (there are scripts to produce qdp lightcurves for example) then to FITS. I'm not sure how long this will take though (if I can get some response from MIT it will go quicker of course)." (18) T: It would be nice if the HEASARC could get a copy of the AGILE data. U: Nick discussed how the proposal to the 2000 Senior Review to do this was rated so low as to sink it. Lorella said that Agile's launch date is unlikely to be before 2004, in any event. Any further comments/suggestions? (19) T: Could there be fhelp-like help available for Xanadu tasks? U: Bill and/or Keith should address. (20) T: Could there be better help documentation for XSTAR? U: Steve relayed this to Tim Kallman. (21) T: There were concerns about the impact of NVO-related activities on the HEASARC budget, and whether they would adversely affect core HEASARC functions. U: Nick agreed that this was a worry, and that he was aware of this as a potential problem. (22) T: Could there be a new functionality in Browse so that one could request a specific data product such as a background map for, say, 500 observations, in a one-click shopping mode? [See attached note from Brandt amplifying this]. U: Tom and Christina should address. ------------------------------------------------------------------------ Hi, I wanted to write down the one final BROWSE comment that I only was able to mention briefly at the end of the HUG meeting yesterday. I can give this best as an example... Recently, we wanted to obtain the WGACAT FITS image files for several hundred ROSAT PSPC observations as cataloged by WGACAT. However, we could not find a "natural" way to do this in BROWSE, largely because the data are organized on a source-by-source or pointing-by-pointing basis. The only way to do it now seems to be to go through pointing-by-pointing, copying the one relevant file each time for each pointing. It would be helpful to have a "natural" and general way in BROWSE to access and copy the same _type_ of file for _many different_ data sets, so that problems like the above involving many different data sets have an easy solution. I wrote down my other BROWSE comments yesterday and gave them to you; feel free to contact me if there are any questions. Thank you, Niel ---------------------------------------------------------------------------