NTEGRAL Science Data Center (ISDC) =================================== http://isdc.unige.ch Release Note ============ Package: osa_sw Version: 7.0 Rel. Date: 26-Sep-2007 Contents ======== 1. Introduction 2. Portability 3. System Requirements 4. Acknowledgements 5. Changes since last Release 1. Introduction ============ This is the release note for the official ISDC 'Off-line Scientific Analysis' software version 7.0 This release contains instrument specific analysis software for the four INTEGRAL instruments (IBIS, SPI, JEM-X and OMC), and some generic tools. It runs on Linux, Mac OS X and Solaris platforms. This Release Note gives some portability information and describes system requirements. The software is available to the scientific community as downloadable binary tar files from the ISDC public release page at http://isdc.unige.ch/index.cgi?Soft+download If needed OSA can be compiled and installed from the source code as well. The source code tar file is available via the above URL. Information for user support is available at http://isdc.unige.ch/index.cgi?Support+info 2. Portability =========== Binary Packages =============== The software was checked to correctly run on the following platforms: - Linux - Fedora Core release 1 (Yarrow) (libc-2.3) and - Fedora Core release 7 both 32 and 64 bit x86. - Mac OS X Tiger (intel & ppc) - SUN Solaris 8 both 32 and 64 bit SPARC. The binary OSA software packages were built on the above platforms. They are ready to use and contain everything you need to run the OSA software. If needed you may compile and link your own software with the libraries included in the binary OSA packages. In general, the OSA software will also run on a variety of other Linux platforms. As we at the ISDC do not necessarily have access to those platforms, we only provide full support for the platforms mentioned above. Depending on the particular choice of your Linux distribution you may miss specific system libraries when running OSA from a binary package. Please let us know in case you require additional libraries. Source Code Package =================== Before you consider building OSA from source, please consider using the supplied binary packages. If you need to build the OSA software from the source code, the following platforms have been tested: - Linux Fedora Core release or newer GNU C/C++ compiler version 3.2.2 to 4.1.2 and G95 Fortran compiler version from 11-May-2007 - Mac OS X Mac OS 10.4 using GNU C/C++ compiler shipped with Tiger (ie, 4.0.2) G95 Fortran compiler version from 11-May-2007 - SUN Solaris SUN Solaris 8 using SUN Forte 7 compilers !!! Warning !!!: OSA is known NOT to compile with SUN's Forte 6.1 and 6.2 3. System Requirements =================== Binary Packages =============== - Linux - Solaris no special requirements - Mac OS X X11 -- if not yet available on yur system, you will have to install X11 from e.g. the Mac OS X installation disks. Source Code Package =================== Before compiling and installing the osa_sw version 7 from the source code, you need to make sure that the following packages are installed: - GNU make version 3.79.1 (or higher) available via the ISDC WWW-site at URL http://isdc.unige.ch/index.cgi?Soft+download in the section on 'Third Party Software' !!! Warning !!!: You really need GNU make for Linux and Solaris. The compilation will fail with SUN's make, for example. Most Linux distributions will already include the correct make. - ROOT version 5.08.00 Since OSA version 3.0 you can choose between an installation with and without ROOT. If you choose to install without ROOT, you will NOT benefit from all OSA functionalities, i.e. GUI support is not available. To learn more about how to install OSA without ROOT please see Appendix B section 'Setting up the Environment' of the 'Installation Guide for the INTEGRAL Offline Scientific Analysis' ROOT is available via the ISDC WWW-site at URL http://isdc.unige.ch/index.cgi?Soft+download in the section 'Download OSA 7.0 for Developers' !!! Warning !!!: For Mac OS X users it is required to install ROOT from the source code. For other operating systems it is recommended to install ROOT from the source code. If you want to download ROOT as a binary package, you must make sure that the compiler used to compile ROOT is identical to the one you are using for the installation of the OSA SW. Otherwise, please download the ROOT source code package and compile and install it yourself. Using different compilers may result in serious problems with your X-system. - X11 On Mac OS X you have to have X11 available. You may install it from the Mac OS X installation disks. - Disk space for installation The unpacked osa_sw binary package requires approximately 2 GB depending on the operating system. The unpacked osa_sw source code package requires some 87 MB of disk space. Once the software is built and installed a total of some 2 GB of disk space is needed dependending on the operating system and compiler used. Note: After the successful installation from the source code, you may reduce the disk space needed by osa_sw by executing 'make distclean' in the same directory where you executed 'make global_install'. This will reduce the amount of disk space needed to some 890 MB. You may additionally remove the source code directories (support-sw, analysis-sw, contrib-sw). This will reduce the disk space needed to some 790 MB. 4. Acknowledgements ================ The OSA math library (isdcmath) includes code from the following packages: - BLAS / LINPACK http://www.netlib.org/blas/index.html - CDFLIB90 http://odin.mdacc.tmc.edu/anonftp/ - LAPACK http://netlib.org/lapack/index.html - PORT from Bell Labs http://netlib.bell-labs.com/netlib/port/ - PDA from Starlink http://star-www.rl.ac.uk/static_www/soft_further_PDA.html 5. Changes since last Release ========================== ================== OSA 6.0 -> OSA 7.0 ================== ----- General ----- This is the first release that is built and tested on 64 bit linux and solaris systems. Please note though that the 64 bit code is less well tested than the 32 bit code. One should have no problem running the 32 bit versions on a 64 bit system. ----- JEM-X ----- This new release introduces important modifications to add robustness in the source-detection process of the imaging step. Source detection in j_ima_iros is now made in three optimized energy bands, irrespectively of the choice of the energy bands made by the user. This has the advantage of reducing the possibility of fake sources being detected because of statistical fluctuations in the different bands. The detection of hot pixels in the data correction step has been improved. The mosaic program, j_ima_mosaic, is now able to combine any mosaic created by itself, irrespective of the instrument. This allows in particular to combine JEM-X 1 and JEM-X 2 mosaics of the same field in order to benefit from the total exposure time on a particular region to increase the depth of JEM-X observations. A new tool has also been added to OSA in order to perform source detection on any JEM-X image (Science-Window images or mosaics). This tool, called j_ima_src_locator, has to be called outside of the JEM-X scripts, since the imaging step already includes source detection. Matching with a source catalogue can be made very easily with the same tool as that used in the JEM-X script, q_identify_srcs. ----- ISGRI ----- The energy correction has changed drastically: Event rise time and pulse height are corrected by 2 calibration laws above and below 50 keV. The resulting positions of the calibration lines are constant over the mission. The drift of the energy calibration gain and offset with activation and time are now much better calibrated at intermediate energies. This results in more constant Crab light curve. The dependence on switch on time is no more used. The correction gains and offsets are corrected for the effect of total accumulated dose. This correction is also applied on the low energy cutoff value (low threshold per pixel). A 3 dimensional look up table is used to compute energy from rise time and pulse height as well as a random value that take into account the non univocal nature of the pulse height/rise time to energy conversion. A default parameter concerning the detection of noisy pixels based on timing characteristics was modified. The default constrain on the attitude stability has been tightened. The dead-time correction has been improved to handle rare Telemetry defects. By default the background map normalisation is calculated using all ISGRI modules. This gives better results if the list of bright sources used in the background normalisation determination is not accurate, i.e. the background subtraction is more robust. A better modeling of the spider was achieved by a new attenuation ic file. Coordinates for the center of the mosaic are now by default computed automatically. The sumhist program can now be used to create background maps for ISGRI. A number of minor bugs have been corrected to improve robustness and portability on 64 bit operating systems. There are new calibration (ic) files for: Veto calibration (calibrated over a longer time scale); Attenuation coefficient; 3D look-up-table; Background maps, RMF, ARFs (updated for the new energy calibration). ------ PICsIT ------ The main novelty of this release is the possibility to perform a scientifically sound analysis of GRB detected in spectral timing mode. As known, to save telemetry, PICsIT is set to work by integrating onboard data into histograms. Data are simultaneously stored in the spectral imaging data set (64x64 pixels, 256 energy channels, 1 histogram per ScW) and in the spectral timing data set (no spatial information, only 8 energy bands, but high time resolution, from 1 to 500 ms). In every standard ScW, the user can find data in both modes. Spectral timing data have already given good results by detecting dozens of GRB ([1]; see also Fig. 1), but only with the OSA7 release will be possible to have available the response matrices (RMF/ARF) for this mode, prepared by P. Laurent (CEA/Saclay). We refer the reader to the discussion in [2] for more details. In addition, the executable for the lightcurve extraction from spectral timing data has been improved to include an automatic alert message in output when the rate is greater than a certain value, which in turn can be set by the user. Other improvements in OSA7 for IBIS/PICsIT concerns an update of the mosaic tool (correction of source positioning) and of the calculation of energy bands from HEPI-LUT for spectral imaging data. Although OSA for PICsIT has a predefined set of energy band (8 for single events and 8 for multiple events), it is possible for the user to select his/her own set of energy bands. However, the main problem is the preparation of a proper set of background maps. OSA for PICsIT includes in IC files a set of maps for the 8 standard energy bands. If the user wants to change the energy bands, he/she should also prepare a corresponding new set of background maps. References: [1] IBIS/PICsIT Source Catalog, http://www.iasfbo.inaf.it/Research/INTEGRAL/Catalogue/picsit_soucat.html [2] G. Di Cocco et al., Proceedings of the VI INTEGRAL Workshop, to be published (arxiv:0707.0573). --- SPI --- The SPI software is stable and therefore only smaller improvements and new features are added. A third background modeling method has been introduced: background templates. These templates which are part of the IC tree, are based, e.g., on the GEDSAT rate. The templates can be selected from the spi_science_analysis GUI (or the command-line) instead of the flatfield or the background tracers. The templates are provided by the SPI instrument team in Toulouse. The flatfield method, however, remains the default method. The second major change is the availability of a timing imaging mode in spiros. This mode allows to construct images even when strongly variable sources are in the field of view. Previous versions of spiros could only construct light curves or spectra when variable sources were present. With OSA7, also images can be obtained. The spi_science_analysis pipeline script has been updated and the GUI adapted to accommodate the new features described above. The third important addition is the inclusion of spimodfit. Spimodfit is an alternative to spiros. It allows to derive spectra for sources in the field of view and most importantly to study the diffuse emission. Spectra obtained by spiros and spimodfit are found to agree generally very well. Spimodift is not yet included in the spi_science_analysis script, but is a stand alone tool. Apart from these major changes, all software has been updated and improved. The IC files have also been updated, the bad pointing filter now covers all revolutions up to Rev. 539. The documentation has been updated and improved. The online user manual can be found by clicking "Support" and "SPI analysis" from the main ISDC page, or directly at http://isdc.unige.ch/index.cgi?Support+spi --- OMC --- The new method introduced in OSA 6.0 to calculate the fluxes by re-centering on the catalogued positions is now the default (IMA_wcsFlag=yes). As a consequence, the OMC analysis uses by default the OMC Input Catalogue. The location of the catalogue must be passed to the scripts using the "ISDC_OMC_CAT" environment variable. ================== OSA 5.1 -> OSA 6.0 ================== ------- General ------- You can trade disk space from memory in the case where running OSA is taking too much memory. Recommendation: e.g. uncompress the events files will help. After OSA 6, there will be no future support for old Linux and MacOS Panther. ----- JEM-X ----- * This new release introduces a much more robust gain calibration. Improvements in the software reduced considerably the (already small) number of revolutions which produced incorrect gain corrections. The automatic calibration performed inside OSA still fails for a few revolutions affected by numerous instrumental problems. Those revolutions have been identified by Jem-X team, and the gain calibration has been redetermined manually. These calibration files are now included in OSA's Instrument Characteristics tree. The analysis script determines by itself the best gain correction to be applied. For future revolutions affected by this problem which are not yet distributed in the IC files, separate gain correction files can be obtained from the following web page: http://isdc.unige.ch/index.cgi?Support+jemx * A table of bad time intervals is now included in OSA. For Jem-X, it includes all the periods that should not be used for standard scientific works, either because of hardware problems or because of intentional modifications of the instrument configurations for the purpose of testing and calibrations. By default, these periods are automatically excluded from any analysis with OSA. * Several bug fixes and improvements have been performed in the imaging tool. In particular, a significance map can now be produced for individual Science Windows. In addition, the images are now corrected in order to allow the extraction of fluxes, either from the Science Window images or from the mosaics. A generic tool to extract fluxes and spectra from INTEGRAL images is available in OSA. The mosaic tool implements now a correct projection, which removes the limitation on the angular size of the mosaic. ----- ISGRI ----- The main improvements of OSA 6 concerning ISGRI are as follow: * The calculation of good time intervals takes better into account pointings with several periods of stable attitude. More stringent limits on the attitude stability are used by default in OSA 6.0, which improves the cleaning of sources. In some rare cases of telemetry gaps, the calculated GTI were not fully correct, this is now fixed. * A "bad time interval" table has been added in the IC tree and is used by default by the analysis to disregard the corresponding data. It points to periods of time during the mission affected by peculiar instrument setting used during calibration, solar flares, bad settings of rise time thresholds, problem with the IBIS VETO configuration or passage through the radiation belts. * A non accurate calculation of the dead time in case of very strong background has been corrected. * Selection of events based on rise time is now performed on corrected values. * Background subtraction involves the normalization of background maps. This normalisation now disregards pixels illuminated by very bright sources, which improves significantly the results especially when bright sources are on the border of the FOV. The normalisation of the background maps is also performed independently for each module. All this improves the quality of the science products. In particular lightcurves of Crab are now much more stable. * The mask supporting structure (including the NOMEX honeycomb structure) is absorbing the photons before they hit the detector. This absorption depends on the incident angle of the photon and of its energy. A numerical model of this structure has been created and adjusted to flight and ground calibration data. The images/spectra/count rates are now corrected for the effect of this structure. The resulting corrected count rates are increased by a factor up to 3 at low energy. The instrument response included a correction for the average absorption of the mask supporting structure that has now been removed. This correction improves the accuracy of the count rate determination (at various off-axis angles), in particular the stability of the Crab lightcurve. ------ PICsIT ------ * The main difference in OSA 6.0 with respect to OSA 5.1 is the presence of an "off-axis" correction in ip_skyimage. This correction is calculated according to the off-axis PSF studies and therefore it takes into account several effects that can generate distortions in the PSF. This correction is something temporary, while detailed studies are on going in order to disentangle the different contributions and to correct them separately. * Another news is the release of new RMF/ARF for single and multiple events with a proper channel width, in agreement with onboard HEPI-LUT. --- SPI --- * A new background modelling method has been introduced: instead of using background tracers, SPI flatfields are now part of the SPI IC-tree which are used to model the background. In depth analysis by the SPI team showed that this method is superior to other background estimation methods and is therefore the new default method. * The input data are now screened and bad pointings are automatically removed from the analysis. This is done by an IC file that contains all bad pointings. This IC file is provided and maintained by the SPI team. * It is now possible to perform phase resolved spectroscopy with SPI, even down to milli seconds (i.e. the Crab) and including double star correction for binary systems. The number of bins and binsize can be defined by the user. The ephemeris is taken from a file the user has to provide that contain valid pulse periods (and derivatives) valid for the observation. * The SPI response binner has been improved and the slope of SPI spectra no longer depends on the number of bins or the binsize. It is now also possible to use very narrow and very broad bins together without problem. * The spi_science_analysis pipeline script has been updated to accommodate the new features described above and the GUI has been slightly redesigned to improve the usability. * Several small bugs in the image reconstruction program spiros (version 9.2.1) have been corrected. Otherwise, the functionality remains the same as in OSA 5.1. * The documentation has been updated and improved. The UM has been re-worked heavily, and the SPI data analysis WWW site includes a full user oriented documentation tree. This page can be found by clicking "Support" and "SPI analysis" from the main ISDC page, or directly at http://isdc.unige.ch/index.cgi?Support+spi Scientific validation of the SPI data analysis system is going on at the ISDC and in different instrument team sites. --- OMC --- * A new method to calculate the fluxes by re-centring on WCS positions has been implemented. The new method reads the coordinates from the OMC Input Catalogue, and is able to calculate the fluxes by re-centring on the catalogued position, after deriving the WCS solution. One advantage of this method is that it can process the mosaics of sub-windows successfully if the source coordinates are accurate enough. It also allows to re-process old data when more accurate positions are available for re-centring. * The WCS solution has been improved in those shots without reference stars. In such cases, the solution obtained for the last shot with reference stars available is used. * Now the user can specify the OMC_ID/s of the source/s he/she is interested in, to extract the fluxes only for the given source/s. In OSA 5.1, this was only possible in the imaging tool (o_ima_build). * The IC files have been substantially improved. Using data from the Earth observation, a new calibration method was developed, and it allowed to derive new flatfield matrices covering almost all the mission. The photometric calibration was updated as well. ================== OSA 5.0 -> OSA 5.1 ================== ------- General ------- The main windows of the Graphical User Interfaces have been improved for all the instruments to easily handle the parameter files of the analysis scripts. They now contain: - a "Save As" button to create a file containing a command-line with the parameters currently set in the GUI. - a "Load" button to read and load parameters from a file previously saved from the GUI - a "Reset" button to reset all the parameters to their default values (as defined in the $ISDC_ENV/pfiles) ----- ISGRI ----- The main changes concerned the ii_shadow_build module which creates shadowgrams of ISGRI events. * A problem in the spectral detection of noisy pixels performed in ii_shadow_build has been corrected. These problems had the effect of rejecting, from the analysis, good pixels in special conditions where very bright sources were present in the partially coded field of view. * ii_shadow_build has been also improved to handle the case of missing contexts, to correctly use the GTI per module and to employ the HK3 data when needed. * A problem in the handling of module switches has also been fixed. This problem was noticed in OSA 5.0 in the sense that module switches were not taken into account properly. This impacts approximately 30% of the Science Windows. Besides these ii_shadow_build changes, it is now possible to combine ISGRI spectra with spe_pick even if the spectra require different ARFs. For the moment, there is still only one ARF, but we expect to release soon three ISGRI ARFs, valid for different periods of time. The new version of spe_pick handles transparently any number of ISGRI ARFs, and produces spectra that can be read immediately with both xspec11 and xspec12. Other minor or cosmetic changes were made in several other s/w modules. ------ PICsIT ------ The OSA 5.1 release for PICsIT includes the fix of a bug in the WCS settings for skyimages, that caused a problem in source positions in mosaics (although there was not this problem in doing mosaics with ScW images from OSA 5 and HEASARC varmosaic tool). A new version of the PIF for spectra extraction has been implemented, although the algorithm is still unstable (see PICsIT known issues). ----- JEM-X ----- The gain fitting and correction module j_cor_gain has been upgraded to version 6.4 and marks a giant leap forward for gain determination in those few revolutions where gain smoothing has previously failed. These have mostly been confined to Crab calibration revolutions, but a handful of revolutions where there were multiple unscheduled switch-offs of the unit were also previously difficult to analyse. Along with the gain aging factors introduced in OSA 5.0, we can now be certain of gain corrections being correct to better than 3%. To see how well gain smoothing and correction has been applied to a particular revolution see the new JEM-X gain results: http://www.spacecenter.dk/~oxborrow/sdast/GAINresults.html Currently this page is under construction, but within a few weeks you should be able to see how gain smoothing of the calibration spectra has proceeded and where the resultant Xenon, Molybdenum and Copper lines from the instrument background are found in the processed science data. These results are made with the software contained in OSA 5.1, and represent the current state-of-the-art in JEM-X energy determination. Anyone with questions or problems pertaining to JEM-X energy corrections and spectra should contact Carol Anne Oxborrow at oxborrow@dnsc.dk No change in j_ima_iros. No change in source extraction (for spectra and lightcurves) except a bug fix. j_ima_mosaic-5.1.0 has been included. The previous version of j_ima_mosaic (5.0.4) needed that at least intensity maps of type RAW_RECT were always produced by j_ima_iros, even if the user actually wished to only combine intensity maps of type RES+SRC. It is now possible with the new version to only (though it is obvious that VARIANCE maps are a requisite to produce weighted mosaic maps) produce at the IMA level RES+SRC maps if one wants to make mosaics of RES+SRC maps. Because of a patch of gti_data_gaps, it is now possible to analyse Scws containing restricted imaging data. Nevertheless, the events gap GTI might be overestimated. ================== OSA 4.2 -> OSA 5.0 ================== Please make sure that the ftools Package (http://ftools.gsfc.nasa.gov) release you use is 5.3 or newer. At ISDC, we have used ftools 5.3.1 for our testing. Inital tests with HEADAS 6.0 show that all works except that Xspec 12 does not read beyond the first spectral extension. The Xspec team at Goddard will issue a patch soon to fix this. ----- ISGRI ----- The ISGRI analysis software has been improved in almost all areas. Besides bug fixes, the main improvements are: 1) ISGRI pixels regularly become noisy. Noisy pixels are detected on board and the transmission of affected events is stopped automatically until the pixel comes back to a normal state. A small fraction of affected events are however transmitted and they could have a dramatic effect on the image quality if not properly flagged. Two new algorithms have been implemented. The first and most powerful one flags event as soon as the statistic of the time difference between event time of an individual pixel is not as expected. On average few hundred pixels are killed every pointings which improve dramatically the image and count rate reconstruction. The second algorithm flags event that come from pixel with abnormal event spectral distribution. This kills few hundred pixels in addition and mainly improves the source spectra. Only few percent of the detector pixels are affected, which does not change significantly the instrument sensitivity. As a result, the count rate of a bright source is now stable within 3% (deviations up to 20% were not unusual with OSA 4.2). 2) The OSA 4.2 dead time correction was affected by a possible on-board counter overflow in case of very bright source or background conditions. This is now correctly taken into account. 3) The off-axis correction and background maps were not properly interpolated with energy, which could generate artificial spectral features. These interpolations are now performed by a new executable weighting and interpolating the maps taking the ARF and average source spectral shape into account. 4) The ISGRI energy calibration is a convolution of the rise-time correction (LUT1), of the electronic calibration (LUT2) and of the gain calibration. The gain calibration takes now into account temperature effects and the degradation of the detector with time and after solar flares. This has been calibrated within 1%. 5) The on-board pixel low energy thresholds have been decreased by about 2 keV. The analysis software has been modified to take the values of those low energy threshold better into account. The on-board modification and the improvement of the software together increases the low energy count rate of the instrument by a factor of 5 between 13 and 22 keV. 6) A new and better LUT2 has been derived using ground calibration data. A validated RMF taking the new LUT2 into account is however not yet available so the new LUT2 was not included in OSA5. However, the ARF was modified to provide a reasonable response down to 17 keV and to take into account all the OSA5 software improvements affecting the source count spectra. 1% systematic errors should be taken into account when fitting spectra. 7) In rare cases the pointing attitude determination was not properly taking into account the good time interval. This is fixed. 8) The image cleaning has been improved very significantly and several options have been added to fix known source positions to improve ghost cleaning, to force cleaning of faint sources even if found negative. 9) Spectral and lightcurve extraction benefits now from a new fitting procedure that works much better for faint sources and removes the significant hard tails that were found with previous versions of the software. 10)New background maps have been built from empty fields. As these are defined in narrow energy bands, they could be used for spectral extraction. Besides this, the analysis script parameters and graphical user interface were improved. Several improvements important for bright sources are still being worked out and could not be included in the current OSA release. Those should be made available later this summer in the form of new calibration files: a) The new LUT2 and associated RMF and time dependent ARF. b) An off-axis correction depending on azimutal angle derived from a ray tracing model. Preliminary version of those files are available at the ISDC. ------ PICsIT ------ PICsIT has been almost completely revised for the release of OSA 5.0. Some bugs have been fixed in order to have a more robust pipeline (e.g. overflow error in lightcurves), while some user requests have been met, like the use of a variance shadowgram not corrected for background. Indeed, in the previous releases of OSA, the shadowgram of the variance was updated for the background subtraction after its creation. This meant that the significance in output from OSA was already corrected for systematic effects (or, at least, for most of them) of background. With OSA 5.0, it is possible to select the corrected variance or the raw variance: in the latter case, the significance map is not corrected and therefore it is necessary to analyse the S/N distribution in the pixels in order to evaluate the background level. OSA 5.0 also provides 5 new sets of background maps prepared by Piotr Lubinski (ISDC) based on public data of different epochs. The default set of maps used in OSA 5.0 remains the set with 1.7 Ms of exposure built using revolutions 49 to 67, but the user will have more maps amongst the IC files with which to try different background subtractions. A prototype of the point sources spectra extraction module, based on PIF (Pixel Illumination Factor), is also available. It should be used with extreme care since PICsIT does not always have sufficient counts in a single Scw when compared to the background (we remind you that the Crab rate is about 10c/s in comparison with a background rate of ~2500 c/s over all the detector). It is worth mentioning that it is possible to extract the spectrum from the imaging results (a script by M. Chernyakova is available in the contributed scripts page). Last, but not least, the spectral timing mode settings have been updated with the change of the energy boundaries of the channels. The number of channels is still 4, with time resolution of 4 ms, but the boundaries have been changed to: 260-364, 364-676, 676-1196, 1196-2600 keV. These new settings are useful in high energy GRB studies since PICsIT is presently the only in-orbit instrument able to explore the MeV range. --- SPI --- A new version of the image reconstruction program spiros (version 9.2) has been developed. This key element of the ISDC SPI analysis system offers additional functionalities in "timing" mode, while the "imaging" and "spectral" modes are basically unchanged. The spi_science_analysis pipeline script has been fully redesigned. The "alternative" part was removed and the interface simplified. It is now much easier to launch the script without Graphic User Interface, either from command-line analyses, or from scripts. A new script spi_grb_analysis, including two new executables, has been developed to ease analyses of Gamma-ray bursts. This script calls many times the spi_science_analysis script. The main parameters are the start and the stop of the GRB, either in UT or in IJD, and they are entered as command-line arguments. A new set of Instrument Response Functions (IRFs) is part of the package. It include three independent responses for before, between, and after the detector 2 and 17 failures. The responses have been fully recomputed after the discovery that a part of the JEMX mask was not properly modeled in the GEANT simulations. This problem was affecting however only analysis of sources with very large offset angles towards the JEMX instruments. A large effort has been made to improve the documentation. The UM has been re-worked heavily, and the SPI data analysis WWW site has been redesigned. It now includes a full user oriented documentation tree. This page can be found by clicking "Support" and "SPI analysis" from the main ISDC page, or directly at http://isdc.unige.ch/index.cgi?Support+spi Scientific validation of the SPI data analysis system is going on at the ISDC and in different instrument team sites. A paper dedicated to the evaluation of the performance of point-source data analysis has been published in MNRAS and is available from our documentation tree. ----- JEM-X ----- The imaging software has been completely reworked. The set of three components for image making, source finding and intensity correction has been replaced by a single component: j_ima_iros. This produces better images and makes much better estimates of the source fluxes. As the name suggests, an IROS mechanism is used for better fluxes and improved sensitivity. Also a much more detailed description of the detector behavior, mask support structure and collimator geometry has been included. A completely new component is the mosaicking tool, j_ima_mosaic, especially adapted to the JEM-X images from j_ima_iros. It replaces the ISDC tool, image_mosaic, that did not make the best of the JEM-X images. The spectrum and light-curve extraction have also been updated with a more precise detector modelling and collimator description. It is now also possible to get a PIF (Pixel Illumination Function) by setting a parameter. The gain correction algorithm has been updated to cope with the higher frequency of the so-called 'glitches' - sudden drops in gain under the calibration sources that take up to half an hour to restabilize. It has been noticed that, due to the heavy irradiation of the small parts of the detector under the calibration source, the gain development is a little different from the rest of the detector which leads to an apparent 'drift' of the Xe fluorescence line in energy with time. This has been included in the corrections to a first order. A side-effect of the introduction of j_ima_iros is the requirement for two large (400 MB) IC files. The backprojection lists (BPL, one for each instrument: jmx1_bpl_grp_0001.fits and jmx2_bpl_grp_0001.fits) represent the "Aperture Response Function" i.e. the description of the transparency between each pair of detector pixel and sky pixel. --- OMC --- Celestial coordinates (RA_FIN and DEC_FIN) of all observed sources are now calculated in o_src_get_fluxes, by fitting the best astrometric solution to the faint photometric reference stars. A new solution is computed for each effective integration. This allows to correct the inaccuracy due to the thermoelastic deformations, which affect the alignment of the OMC optical axis with the S/C attitude reference. The imaging tool (o_ima_build) has been substantially updated to improve the WCS support as well as to make it more user friendly by adding new parameters. Now the user can specify the minimum and maximum values for the shot integration time, as well as the OMC_ID of the source he/she is interested in. If a given source is selected, small images are built containing only the OMC sub-window/s corresponding to the selected source. Thanks to the improvement achieved in the astrometric solution, a new OMC misalignment matrix has been calculated. By using it, the accuracy of the computed celestial coordinates and the WCS support is better than 2 arcsec in most cases. The scripts have been also updated to include new parameters and make them more user friendly. Optionally, now the user can run the scripts to build only the images. ================== OSA 4.1 -> OSA 4.2 ================== ----- ISGRI ----- * In ii_shadow_ubc 2.9.2, there is a change in the way the efficiency and background corrections are performed. * ii_skyimage 3.9 As the efficiency correction is done now in ii_shadow_ubc, it is not redone in ii_skyimage. Consequently, it is important not to mix versions of ii_skyimage and ii_shadow_ubc from old releases (before OSA 4.2) and this new one. You cannot use old binned shadowgrams with the new ii_skyimage. It is now possible to fix all source positions in the individual Scw fit. The covariance matrix is corrected by off-axis effect. The mean is no more subtracted in the SNR images. * ibis_science_analysis was improved as regards user-friendliness. It can use more than 10 energy bands for imaging (see the updated cookbook in the ISGRI User Manual) ------ PICsIT ------ * The main change in the PICsIT s/w is that the count rates were changed and are now higher (~30%) with respect to the OSA 4.1 or older versions. This was required to match the MC model used to build the RMF/ARF. * Mosaic should work now for long integration and there is the possibility to make mosaics for polar regions with two more projections (zenithal equidistant ARC and stereographic STG) in addition to the gnomonic (TAN) already present in older versions. --- SPI --- New versions for several components were produced: - a bug was removed in spi_clean, - spi_gain_corr and spi_scripts were modified to cope with new ISDC directory structure and data format. No new functionality is available in OSA 4.2. ----- JEM-X ----- * The Instrument Model data now contains calibration information to support regularized pixel positions. * New background models have been delivered, but their use is still tentative. * Deadtime calculations now also handle the non-nominal case of both 16MHz and 8MHz CPU speed settings mixed in the same science window. * By default, shadowgrams are now produced in normal coordinates with regularized pixel positions. * For spectra and lightcurve extraction, the modelling of the of flux distribution on the detector has been improved and the idealization of a flat background given up. * Because of the limitations of the mosaic tool, image mosaics are currently not produced by the analysis scripts. It is still possible to do this manually. * The analysis scripts handle both old and new style of the directory structure. --- OMC --- * Templates and executables have been updated to record more information in the source results tables. As specific examples, the derived magnitude is now given for the different possible extraction sizes (1 pixel, 3*3 or 5*5 pixels). * In case of (GRB) Trigger Mode data present in a Science Window, the scripts will automatically create images. The scripts now also include an optional step (IMA2) to create a summary results table. ================== OSA 4.0 -> OSA 4.1 ================== ---- IBIS ---- * An important bug in the GTI calculation is fixed. The cookbook example of the crab light curve now shows a constant curve. The big dip in the crab light curve generated with previous software was due to this bug. * A bug is fixed in the script that was preventing to run the CAT_S alone. * A bug is fixed in the energy band choice of ii_pif_build when attempting NOMEX correction with ii_light. The example in the cookbook now works correctly. * lc_pick now works for ISGRI. * lc_pick now understands standard source name besides SOURCEID. --- SPI --- A major bug in spi_science_analysis was fixed. Now the SPI analysis works properly on Linux platforms. --- OMC --- * A new algorithm to compute the background based on the 11x11 exterior rim has been included. * The fluxes are now computed by a sub-sampling method and aperture corrections are applied by integrating the PSF. * A method to compute the centroid and the PSF width has been developed. * Several of the "un-detected" saturations can be now detected by comparing with the expected PSF. * New flags indicate the presence of source contamination. ================== OSA 3.0 -> OSA 4.0 ================== Starting with OSA 4, you may choose between two different mathematical libraries: - NAG, the mathematical library from the Numerical Algrithms Group - isdcmath, a compilation of public domain routines ----- ISGRI ----- * The energy correction has been improved, taking into account the effect of the bias voltage and the effect of a global temperature. Temperature gradient correction will be implemented later. The shift in gain is now taken into account. The gain-corrected value of risetime is written in the column ISGRI_PI. * Binning and background correction have been improved. The problem linked to uniformity image is corrected. The time keywords have now the OGIP standard meaning: ONTIME is the sum of the GTI, EXPOSURE = ONTIME*Average(Effi). Note that the images and spectra are computed in counts/sec and ONTIME value is the reference value for flux and spectrum evaluation. * Several bugs and problems of image reconstruction and spectral extraction modules are now corrected. * The "NOMEX effect" is now described in a new instrument characteristic file and corrected for in ii_skyimage and ii_spectra_extract. The NOMEX is a honeycomb support structure behind the mask absorbing low energy photons. Its geometry makes the absorption very dependent on the source position in the FOV and on energy. * Spectral extraction can now be performed using least square fit, which is now the default method. Background correction maps for spectral extraction are now available and they are the default input. * A new executable to extract light curves has been added, but the NOMEX effect removal is not fully implemented, yet. * Spectra can also be extracted based on a PIF (Pixel Illumination Factor) method but without NOMEX effect correction. In general, the ability to have event lists with stand-alone PIF building now allows different kinds of analysis using just standards FTOOLS. * A new ARF file corrects for the "snake effect" seen in reconstructed spectra with the present RMF (i.e. the fact that powerlaw spectra appear as having wiggles instead of being straight). ------ PICsIT ------ * An early version of an improved set of background maps is now available. This gives cleaner images with higher signal-to-noise ratio. * A PICsIT specific mosaic executable is now available directly in the pipeline. * It is possible to obtain PICsIT lightcurves of the whole detector by means of the spectral timing data. This submode generates lightcurves in four energy bands with time resolution of 2 s and could be useful in the study of Gamma-Ray Bursts (GRB). " --- SPI --- * A completely new spi_science_analysis script has been developed including a new GUI. The user can select one of two "pipelines", a "default" and an "alternative" one. The "default" pipeline makes use of software developed by Jurgen Knodelseder (see the proceedings of the 5th INTEGRAL conference in Munich) for event binning and background modeling. These programmes run much faster than their counterparts from OSA3, which are still available in the "alternative" pipeline. This new script is more user friendly. It does a number of tasks automatically, such as saving the exact list of used parameters and the logging information in dedicated files. The number of input parameters has been reduced and they are well documented through the GUI. The given default values can always be easily overwritten when required. * A new program (spi_add_sim) has been added, allowing to add a simulated source to the actual observation. This tool is useful to evaluate the possible effects of crowding in relatively realistic conditions (see Dubath et al. 2004, A&A submitted for more details). * A new system of gain coefficient files handling is also available as part of the new script. The gain coefficient files derived now for each revolution in the standard ISDC processing are used by default. If they are not available the IC files are used instead. And as before, users can impose their own calibration index file through a normally hidden parameter. * A new response provided by the Goddard group is included, describing SPI after the failure of detector #2. The analysis system will select automatically the appropriate response (with or without detector 2), although the system cannot currently handle two different responses at the same time (neither a time variable response). When mixing data prior to and after detector 2 failure, users have to select one of the two responses. The differences in the derived results with the two responses are very small (and negligible) in most cases. Users can check their results by repeating their analysis twice, once with each of the responses, or analyze their data before and after the failure independently with the appropriate response. ----- JEM-X ----- * Gain corrections have been significantly improved. They now handle non-standard cases better, like, e.g., a shut-off and restart of the instrument during an Integral revolution. * More realistic background models have been generated but their use is still non-standard and only meant for evaluation purposes. * The imaging tools now also provide variance maps together with the sky images. * The source spectrum and lightcurve software has been significantly updated. * Binned detector lightucurves and spectra are now also normalized to a default detector area of 100 cm2. --- OMC --- * The flux extraction algorithm has been slightly refined and some more critical cases for the automatic extraction are now recognized and flagged in the results table. This includes the detection of extended sources generating several OMC sub-windows. * Two new parameters allow the user to include only shots falling within a given integration time range in order to avoid data without relevant signal or with saturated pixels. * The images produced by the offline tool o_ima_build are now tagged with WCS information and use the pixel coordinate system of the telemetry. * The table of results produced by o_src_collect includes now the barycentric time. ------------- Generic tools ------------- * The barycentric correction tool "barycent" was corrected and largely improved in functionality. * evts_pick can now merge SPI events. ================== OSA 2.0 -> OSA 3.0 ================== We recommend that you have HEAsoft 5.3 (ftools) installed. Earlier versions are also fine, except that the tool rbnrmf only works for ISGRI as of 5.3 ------------- Generic tools ------------- Barycentrization now gives sensible results. evts_extract was improved and now also creates the GTI extension required by several FTOOLs. The handling of missing data was improved in spe_pick and a bug in handling larger datasets removed. Note that spe_pick does not handle IBIS data yet. The performance of idx_find was largely improved. og_create now checks if the given input exists before undertaking any action. ----- ISGRI ----- Many components (ii_spectra_extract, ii_skyimage...) and the ibis_script were updated. You can now stop and resume before the mosaic. Dead time calculation was improved for ISGRI. Energy correction is now temperature dependent. Changes in dal3ibis now allow to find small telemetry gaps inside a packet and they are used all over the software. The new executable ii_pif_build enables to create a PIF from a spacecraft position and a source position. New IC files were delivered since OSA 2.0 and energy correction was significantly improved in OSA3.0. IC-category OSA2 OSA3 ----------------------------------------- ISGR_ARF_RSP 3 4 ARF is now much better, correspond also to change in the energy correction ISGR_RMF_GRP 5 9,10 9 is the new original full resolution matrix, 10 is our suggestion for smart rebinning and is the default in the script ISGR_ATTN_MOD 1 2 A bug was fixed ISGR_BACK_BKG 2 3 This is the first non dummy one ISGR_COVR_MOD 1 2 ii_skyimage has changed, so those responses have changed too ISGR_OFFS_MOD 1 3 new energy correction strategy ISGR_RISE_MOD 1 3 new energy correction strategy ISGR_TEMP_MOD - 1 Energy correction now is temperature dependent ------ PICsIT ------ An automatic source location is now performed. ip_st_lc_extract was updated to take into account some compatibility problems with XRONOS package of HEASARC. --- SPI --- The spi_science_analysis pipeline script, and the different programs called by the script, offer the same functionalities as in the last release 2.0 (although a number of software bugs have been corrected for). A new alternative script is in development and will be announced soon. The new Instrument Response Function (IRF) is part of the package, as well as the latest RMFs required to fit with XSPEC the spectra extracted with spiros in the spi_science_analysis processing. ----- JEM-X ----- Consistency of results across operating systems has been improved, mainly by using and OS (Operating System) independent randomizer in the gain correction step, when events are assigned to PI bins. Realistic background model data have been included in the IC files and background model information pertinent to the individual Science Windows is now routinely extracted. The use of these data in the further steps, though, is still in an exploratory stage and further fine tuning will be needed. Shadowgrams can now also be background subtracted, but at this stage this is not recommended for routine analysis and disabled as default. Source searching can be limited to a user defined angle to avoid spurious source detections in the noisy outer rim of the field. In addition, for forced source detections via a User Catalog, the software can be instructed to always use the catalog coordinates even if the source is independently found. A new, more refined algorithm is used by default in source spectrum and lightcurve extraction. Also, events flagged as problematic due to their position in the detector, e.g. from a dead anode zone, or their gain corrections are now by default excluded. The flux values for source lightcurves and spectra are now normalized by default to counts/second/100cm2 in each band. This nominal detector area corresponds approximately to the geometrical detector area of 490 cm2 multiplied by the mask open fraction (0.24) and the collimator open fraction (0.89). Other normalizations can be selected via parameters. The effective area response (ARF) is now corrected for radius selection and for dead detector zones. Compatibility with the OGIP FITS time standards has been improved for source lightcurves. Binned detector spectra and lightcurves now also allow for radius and event quality selection, where possible. The analysis scripts can now be installed and run without ROOT, foregoing the GUI support. If users define a start time and time step for the analysis, the start time will be set individually for each science window but always aligned with the original start time by multiples of the time step. --- OMC --- New, improved calibration files for flatfielding and flux to magnitude conversion have been derived and included in the IC data set. With these new files photometric accuracy is about 2-3% in the central 4x4 degrees of the OMC FOV, opposed to variations of up to 15% before, caused by contamination problems during the calibration. A redundant dependency on planning data in the analysis executables which gave rise to occasional problems has been removed. ================== OSA 1.1 -> OSA 2.0 ================== Besides various bugfixes for all the instruments, the OSA 2.0 release mainly intends to allow users to run ISGRI spectral extraction. While it has been technically tested, some scientific evaluation is still on-going and spectral extraction results have to be cautiously analyzed. ---- IBIS ---- Some bugs were fixed in ii_shadow_build, ii_shadow_ubc, ii_skyimage and in the overall script. Consequently, ii_shadow_build should not crash anymore, and bright source pixels should not be mistaken as noisy pixels anymore. Improvements were also performed in ii_skyimage. The Graphical User Interface now works. A new executable (ii_light) has been integrated in ibis_scripts. It allows to produce lightcurves. There are a new RMF and a new ARF for ISGRI. While spectral extraction can be run, it has not been scie