General comments that apply to multiple sections of the document ------------------------------------------------------------------------ [on the typographical convention to italize the words "must", "should", "recommend", etc.] Italicizing these words *may* be distracting. *Must* it be done? I *should* not *recommend* it. Shall it be *optional*? (Rob Seaman) ------------------------------------------------------------------------ Any chance of blessing checksums on this go-around? Then we could fret about using "one's-complement" versus "one's complement", too. (Rob Seaman) ------------------------------------------------------------------------ There is a very real possibility that UTC will be turned into something other than a flavor of Universal Time, i.e., other than an approximation to Greenwich Mean Time. Perhaps we should consider how best to deal with such a situation. A generic "UT"? Use raw UT1? Define a UTA (Astronomical) retaining IAU mediated leap seconds? This is a broader issue than FITS, but we would have to start somewhere. (Rob Seaman) ------------------------------------------------------------------------ I suggest introducing "keyvalue", "keyrecord", "keyfield" as contractions for "keyword value", etc. (Mark Calabretta) ------------------------------------------------------------------------ It would be useful for the FITS 3.0 document to list inconsistencies in syntax [between previous versions of the Standard] in an appendix, e.g. such as the requirement for a slash between value and comment introduced at version 2.0. (Mark Calabretta) ------------------------------------------------------------------------ We need a "Don't Make Me Think" policy with the FITS format. As a programmer, even after reading the documentation on the FITS format, I still struggle with the header portion of the file. How does one know how many 2880 byte chunks preceed the image data???? Like the ARF files we use currently at my work (defense) the format is overly complex and poorly thought out. (Kevin Thomas) That question is answered in the docs. If the file conforms to the standard then the 2880 byte chunk with the END card is the last one of the header. There weren't any other questions in the original post, so it's hard to provide any other answers other than to note that FITS was designed when 32 k was a lot of directly addressable memory, and everything else was found by running the tape back and forth until the item in question was either found or not. (Steve Allen) More standards might benefit from users and programmers having to think. [...] More standards could benefit from a similar dose of self-restraint in minimizing resource requirements. One might have cause, for instance, to be skeptical that many emerging XML-based standards will still be in wide use a quarter century hence. (Although one trusts VOEvent will be one of the shining exceptions :-) Personally, I'm quite confident that FITS will still be with us in 2032. (Rob Seaman) ------------------------------------------------------------------------ I assert that despite its high google score, "two's complement" has an inappropriate apostrophe. [anonymous] Response: This is consistent with Wikipedia which defines the term as "two's complement" (http://en.wikipedia.org/wiki/Two's_complement). Under the "discussion" tab on that page there is some justification for this usage, and in particular gives an explaination for the different position of the apostrophe in "ones' complement" and "two's complement". ------------------------------------------------------------------------ ------------------------------------------------------------------------ Specific comments 2.2 Definition of "deprecate" It is perhaps a little confusing to refer to "applications" in this context, when the standard governs data files and their interpretation, but only advises the functionality of applicable software. The following wording is an attempt to clarify the definition: "This term refers to obsolete features or structures in FITS that _should not_ be used in new FITS files, but _shall_ remain valid indefinitely. FITS users and data engineers should be aware that the use of deprecated structures is considered bad form and should be avoided." For reference, here a list of all the FITS features that have been deprecated: - BLOCKED keyword - EPOCH keyword - random groups - CROTAi keyword Additional structures or features that v3.0 would deprecate: - special records - implicit decimal points in ASCII table fields (Dick Shaw) This wording is a great improvement but still it doesn't quite capture the dichotomy between FITS writers and FITS readers, new and old. There are two types of deprecation: 1) Constructs that ought never be used by writers (but must still be understood by readers) e.g. omitting the "/" comment delimiter, or implicit decimal points in ASCII table fields. It is safe to outlaw such usage because doing so has no effect on old readers. 2) Constructs that shouldn't be used by writers but might still be in order to help old readers (and must still be understood by new readers). EXTEND, EPOCH and CROTAn are examples. (Mark Calabretta As I recall, the definition of "deprecate" was carefully crafted to satisfy the radio interferometry community. The reason for deprecating random groups was to prevent their use from spreading beyond their current use with radio interferometry data files; other types of data should not be distributed in the random groups format, and instead should use the more modern binary table structure. The decision to deprecate random groups was not intended to force the radio interferometry community to give up using random groups and rewrite all their software to use binary tables instead. I think a similar argument can be made for deprecating CROTA. There are a huge number of existing instruments that continue to produce data files containing this keyword, so it is not realistic to expect that they all be modified to use the newer rotation matrix keywords. I think we can "strongly discourage" entirely new intruments and software systems from relying on the old CROTA keyword, but it would go too far to disparage any continued use of CROTA as being "bad form". (William Pence) ------------------------------------------------------------------------ 3.5 [which deprecates special records] Extensions ARE special records from the point of view of classic FITS readers. Rather than deprecating or attempting to ban them, perhaps just explicitly state that new uses for special records require IAUFWG approval. (Rob Seaman) I think it's okay to deprecate 'special records' as the general extensions should be able to handle our future needs. In an age with computer virus, it may be safer to have some handle on what can be in a FITS file. (Preben Grosbol) The justification given is appropriate. The special records rule was the biggest single escape hatch in the original FITS Agreement of 1979; it was the basis for the Generalized Extensions Agreement of 1984, which led to TABLE, BINTABLE and IMAGE extensions. I agree that the Generalized Extensions Agreement "appears to be sufficient to meet most future needs". Indeed, I would go further and say that the burden of proof is on anybody who claims that the 1984 Agreement is not able to satisfy some astronomical data structure requirement. The variety of conventions that have been built on top of BINTABLE show that it alone has enough versatility and functionality to solve an enormous range of problems. Also, if we encounter something that BINTABLE cannot do, it is always possible for us to encapsulate *any* other data format within FITS, if the other format solves a problem that FITS can't solve, by creating extensions with XTENSION='' and the other file in an NAXIS=1 byte array. Therefore, although this item makes me nervous, I have decided to accept deprecation of special records. (Don Wells) ------------------------------------------------------------------------ 3.7 [which exempts existing FITS files from new FITS requirements] Unenforceable in the absence of some explicit mechanism tagged to the DATE keyword (for instance). On the other hand, such a mechanism would be equivalent to explicitly versioning each file. This would be a change to the basic philosophy of FITS. I think of opening old Word files with new editions of MS Office - or worse - trying to do the reverse. (Rob Seaman) The argument is understandable but the implications very wide. It would allow dramatic changes such as redefinition of keywords. Something like this is needed but we may need to consider the best wording. (Preben Grosbol) In principle the new Sec. 3.7 introduces a FITS version that is in FITS v3+ certain things are not valid any more. We previously were very reluctant to introduce the concept of versioning but maybe it cannot be avoided. It is a rather principle matter which should be discussed in some detail (maybe at the ADASS BoF). (Preben Grosbol) I agree with Preben that there are also problems with introducing versioning, but otherwise conformance is unenforceable. The suggested new wording of 3.7 can't serve as a substitute for an explicit version since A) not all files have DATE keywords, so how would you know, and B) it will take several years - perhaps never - for all prior software to adopt the new standards. Rather, projects wanting to assert compliance would include a recognizable FITSVER (for example) keyword. Files missing this tag would by default only claim compliance with the original standard. In a real sense, the standard as revised is really a brand new standard, subclassed out of FITS. (Rob Seaman) It might be that the text of 3.7 could be improved but what is the actual impact of all this ? Does it apply to specific pieces of software ? to existing one ? to new one ? Or just to (human) programmers instead of software ? The main point is that it does NOT force OLDER files to need to be rewritten to conform to new standard "additions". Just as e.g. the Y2K change to the DATE format did not force anybody to rewrite old file with the new ISO timestamp. So a generic reader might have to support older and newer variants (which gives a burden on us to clearly document how and when they changed, and from what into what). Or if it supports only the new variant it shall warn in case of non-compliance "maybe this is an older file" (or check if possible if this is the case), and be somewhat lenient. However it is always possible to add a COMMENT which claims conformance with the latest (e.g. 3.0) standard. (Lucio Chiappetti) The "once FITS always FITS" philosophy captures the spirit of FITS, but in practice each new version of the FITS Standard has imposed new requirements that in principle could invalidate existing FITS files. For example, version 2.0 of the FITS Standard introduced a new requirement that the value and comment fields in a keyword MUST be separated by a slash character. I think the FITS community in general has taken a somewhat pragmatic view that it is OK to add a new requirement to FITS that *might* invalidate older FITS files, as long as the benefit of the new requirement is perceived to out weigh the possible negative affects. [...] In the new draft of the FITS Standard, the technical panel listed 22 specific changes to the FITS requirements in the "Summary of Recommended Changes" document (at http://fits.gsfc.nasa.gov/fits_draft.html), but most of these either remove requirements, or only add a recommendation. There are only 3 proposed new absolute requirements in this list: 1. Keywords that have a value shall not be repeated in a header. 2. PCOUNT and GCOUNT must immediately follow the last NAXISn keyword in all conforming extensions (as is already required in IMAGE, TABLE, and BINTABLE extensions). 3. Embedded space characters are now forbidden within numeric values in an ASCII Table (e.g. "1 23 4.5" is no longer allowed to represent the decimal value 1234.5) (William Pence) The proposed new statement that existing FITS files are exempt from any new requirements is, I think, mainly intended as a political statement to reassure institutions that the FITS committees are not imposing new unfunded mandates that require modifications to existing FITS archives. I don't see this statement as having much relevance to the way software is implemented. (William Pence) A file either conforms to the FITS standard or it does not. A ban on duplicate keywords is unenforceable unless it is paired with versioning. The statement above would fail to impress a lawyer since it isn't paired with a way for either humans or computers to determine which files were grandfathered in. Further, there is a sense of legal entrapment in promulgating such a new requirement with no realistic way to encourage instrument teams and others to redesign their systems to avoid duplicates. For instance, the ICE/ccdacq software permits observers to enter their own file of keywords, perhaps including duplicates. Users can trivially use IRAF hedit to add duplicates, etc. Perhaps there is no way to duplicate a keyword with CFITSIO? Who would enforce the ban? In any event, the FITS standard should be kept free of political statements. (Rob Seaman) ------------------------------------------------------------------------ 4.1.1 [which recommends that the keyword order be preserved during data processing operations] Harmless, but also unenforceable. Not just an attempted constraint on software, but also on humans editing headers manually using IRAF hedit, for instance. (Rob Seaman) ------------------------------------------------------------------------ 4.1.2.2: "COMMENT" should be in tt font (Mark Calabretta and Preben Grosbol) ------------------------------------------------------------------------ 4.1.2.3 [which requires that keywords with a value shall not appear more than once in a header] No, but there are many headers that unintentionally repeat keywords. They should not therefore be rendered unconforming FITS. (Rob Seaman) It may be too strong to forbid such repeats of keywords. It should certainly be deprecated but it may place a significant load on many applications which just want to add some keywords to a header. They would be required to actually check if they would duplicate or not. A deprecation and definition that the last value takes precedence may be more appropriate. (Preben Grosbol) I have many examples (hundreds of thousands?) of files in which keywords are repeated. Rather than the wording in the current proposal, I would replace the attempt at a requirement with a strong recommendation and a clarification that the final copy of any such repeated keyword should take precedence. (Rob Seaman) I strongly support this. The proposed text makes such files invalid FITS, retrospectively. Taking the last instance of a keyword is a much more reasonable interpretation. But I note that a program that blindly drops all but the last instance may lose the information conveyed by: the number of instances, the list of values and their order, and any associated comments. Are there existing applications where a keyword can occur more than once with different values, in which more than just the last occurrence are intended to carry significant information? (Tim Pearson) In the instruments delivered to Keck by UCO/Lick and Caltech there are repeated occurrences of keywords which not only have values, but for which the data type of the value is different in the different occurrences. (Steve Allen) I similarly cannot see the value of this particular proposed change: FITS readers will need to support repeated keywords forever, given the very large numbers of existing files with them, so it's not even as if this would simplify reading FITS. I am also very much in favour of instead simply clarifying that the last occurence has precedence. (Thierry Forveille) I agree with Thierry that there are many files which have repeated keywords, but I agree with another poster that there are existing implementations which assume it's the first instance, not the last instance, which prevails. So I think we should just strongly deprecate (not ban, and not impose an interpretation). (Jonathan McDowell) I'll just second this. I have seen many cases where a FITS header gets multiple keyword instances after it has been modified by different programs. This is awful of course as the header is then ambiguous, but it happens. Software may or may not use the last instance as a runtime value; for example, if the software does a linear search through the header it is possible that it will take the first value encountered (for either reading or writing) as this is more efficient, and there are not supposed to be multiple keyword instances. If the software does this on a write, then it is the *first* value in the header which will have the correct value. We should strongly recommend that software which writes to a header not create multiple values. If there already *are* multiple values, frankly it is not often clear what to do, as one does not like to delete information from a header. (Doug Tody) One approach would be to say that headers _should not_ contain repeated keywords, and if a repeat does occur then the value is not defined (unless the values are identical). Ideally, this could have a few desired effects: it would encourage authors of FITS verifiers to flag instances of repeated keywords (though I suspect they do already), it would encourage FITS writers to pay attention to this problem, and it would encourage application developers to be refrain from silently adopting the first, last, or whatever instance of a keyword value without telling the user. (Dick Shaw) This looks to me like the best wording so far. I too have had many a weary battle with ``software engineers'' to get them to to conform to the existing standard; expecting a change which in so many cases just doesn't matter (the example of duplicated instrument temperatures) will just get zero priority. Would it be too complex to make the non-duplication of reserved words mandatory and of others just strongly recommended? (Peter Bunclark) We should encourage data providers (and users) to avoid duplicate keywords. We should understand why such keywords may be created in the first place. Our major software packages should reach agreement on a common strategy should duplicates be encountered - whether this is that the behavior remain indeterminate, or the first instance or the last instance take precedent. Applications should detect duplicates which affect their functionality as with any other header peculiarities. Libraries should provide routines and utility programs for validating HDUs against a wide variety of exceptions, including duplicate keywords. (Rob Seaman) All cases I have seen were instances of "updating" the information by appending a second copy of the keyword an the (then) end of the header. Some of these might trace back to real time writing on 9-track tapes, when rewinding to update would have been a significant cost and this was done as an optimisation/compromise (better have slightly ambiguous information that plainly incorrect one), while others certainly are just plain laziness/carelessness. It is easier to write at the end of the header, so I'd expect the latest version of the keyword to almost always be last in the header. Does anyone on the list have examples of FITS files where the prefered version (when that can be determined...) of a duplicated keypword is not the last occurence? (Thierry Forveille) Since the recent discussions have made it clear that it is not a rare occurrence for existing FITS files to contain the same keyword multiple times (the technical panel had hoped this was not the case), it appears not feasible to simply declare that this practice is prohibited in a conforming FITS file (again because of the "Once FITS always FITS" rule). It seems that the most we can do at this point is to strongly discourage this practice, because it is indeterminate which of the keywords will read by software systems when retrieving the value. (William Pence) ------------------------------------------------------------------------ 4.2.1: I don't see the point of distinguishing between "fixed-format" and "free-format" character strings. Fixed-format is just a special case of free-format and the term is never used elsewhere, not even to describe the special formatting required for the XTENSION keyword. The same comment applies for integer and floating-point values. (Mark Calabretta) ------------------------------------------------------------------------ 4.4.1.1 [which recommends that BITPIX data type should be appropriate to the range and accuracy of the data] Few users will choose 64 bits when 16 will do. Those that do may well have a good reason. (Rob Seaman) ------------------------------------------------------------------------ 4.4.1.1 the ordering of NAXISn is never explicitly restricted to increasing numerical order. The only statement for any of the mandatory keywords is presented in table 4.5, which suggests NAXISn be ordered, but never outright says it. (Rob Seaman) ------------------------------------------------------------------------ 4.4.1.2 [which requires XTENSION names be registered with the IAU FWG] This is a requirement placed by the IAU, not FITS per se. Might such policies better be concentrated in a common section? On the other hand, it is eminently practical to manipulate unknown but conforming extensions. In particular, software we write today can skip and list and otherwise manhandle conforming extensions from tomorrow. (Rob Seaman) I would prefer a strong urging accompanied by the usual justification rather than a 'must', precisely because XTENSION is our most important escape hatch, and I don't want to inhibit experimentation unnecessarily. However, I won't insist on this matter. (Don Wells) Registration seems not a very heavy task and should not hinder experimentation (although I did not see much experimentation in the area of "new extensions" but in "new conventions" or flavours of BINTABLEs. Maybe we should somehow formalize the registration procedure a bit more (de-registering if unused after an expiry time ?). [...] Should we define a convention by which "experimenters" are allowed to define XTENSION='X-something' until they are ready to register XTENSION='something' ? But is this really needed ? Until the unregistered extension is used only at one site and the file is not "widely" distributed, there is no harm. And even if it were distributed, I'd expect a generic reader to reject it with "extension 'something' unsupported" (after all I would not expect a *generic* reader to support FOREIGN or A3DTABLE !). (Lucio Chiappetti) ------------------------------------------------------------------------ 4.4.1.2 The EXTEND keyword is no longer mandatory. I am aware of several IDL FITS libraries which follow the existing standard and require the presence of the EXTEND keyword before extensions will be examined. These libraries would no longer function as-is if the EXTEND keyword became optional. (Craig Markwardt) The catfits task in the STSDAS package under IRAF prints a "table of contents" for a FITS file. If the EXTEND keyword is absent, catfits does not look for extensions. I won't defend the EXTEND keyword; I'm just answering your question by giving an example. If an application expects data in one or more extensions, there isn't much point in the application first checking the EXTEND keyword. (Phil Hodge) The SECCHI instrument on the STEREO mission produces some files with table extensions, and other files which don't. The value of the EXTEND keyword is set to either T or F to account for this. (William Thompson) Strictly speaking, this does not conform to the definition of the EXTEND keyword in the FITS Standard, which states that the value field *shall* contain a logical constant with the value T, and thus must never be set to F. All the more reason I think to demote this keyword by making it optional instead of required if the file contains extensions. (William Pence) To me, having EXTEND=F when one knows that no extensions are present is one of those "bedrock of logic" that Rob Seaman refers to. (William Thompson) The difference here is that I was quoting the revised definition in the new draft standard, which doesn't allow for the possibility that EXTEND = F. Given that there are existing FITS files with EXTEND = F, the definition should probably be revised to at least not prohibit this usage. One problem with relying on the veracity of EXTEND = F is that even if the FITS file doesn't currently have extensions, they might be added at a later time, and there is a good chance the EXTEND keyword value would not get updated to T in the process. So just as software should not assume that a file has extensions if EXTEND = T, it could be unwise to assume there are no extensions just because EXTEND = F. The purpose of this proposed change to the standard is to allow a FITS file to containing extensions even if the primary array neglects to contain the EXTEND = T keyword. Similarly, a FITS file should be allowed to have extensions if EXTEND = F, even though this is contradictory, simply because the actual existence of the extension should take precedence, regardless of what the EXTEND keyword implies. Software can easily determine for itself whether there are extensions by examining the file, so there is no need to depend on the EXTEND keyword, whose presence or value may be unreliable. (William Pence) Since the STEREO mission has started producing FITS files with EXTEND = F, it is too late to prohibit this usage in the FITS Standard (because of the "once FITS, always FITS" rule). The technical panel was not aware of this usage when it drafted the change to the EXTEND keyword definition (which was modeled after the definition of the BLOCKED keyword which can only have a T value, never F). The definition of the EXTEND keyword should probably be revised to explain what is meant by EXTEND = F. (William Pence) At the time when extensions were introduced, most readers would not even think of looking for them. Thus, it was important that users could look on their primary header and see (and then complain) that there were extension data available in the file which were not detected by the local reader. Now that extensions are common and most readers would check, I see little problem in making EXTEND optional. The ESO readers, I know of, all would scan the full file no matter what. (Preben Grosbol) The "fault" about EXTEND was in its original definition. The keyword EXTEND=T was present "if the FITS file MAY contain extensions" and it being equal to T did "not imply extensions be present". As such it resembles a bit the present versioning/no-versioning discussion. But it was/is pretty useless to tell whether the file really contains extensions. So the safest way taken by FITS readers was to scan anyhow after the PHDU for named XTENSION kwds. (Lucio Chiappetti) I don't particularly object to the EXTEND change (which doesn't invalidate any old FITS file, just gives a (very) small added flexibility to new ones, so doesn't break the "once FITS always FITS rule"), but don't see its point either. Of course it saves 80 bytes in every FITS file, but I suppose there must be some stronger reason. (Thierry Forveille) Whether or not a FITS file has conforming extensions following the primary array is a simple observable fact; the presence or value of the EXTEND keyword will not change that fact, so the EXTEND keyword seems to serve little useful purpose, and even worse, it can create confusion if EXTEND is not present, or has the value = F in a file that actually has extensions. So the purpose of the proposed change to the EXTEND keyword is to make it clear to software developers that they should not depend on the existence, or value, of the EXTEND keyword when deciding whether or not to read or write an extension in a FITS file. (William Pence) By the way, we get many files delivered for archiving within MAST that use both the EXTEND and the NEXTEND keyword. Although not a reserved keyword, NEXTEND is commonly used to describe the number of included extensions. I guess we are in the minority on this, but when these keywords disagree with the file contents we either correct them or ask the contributors to do it. I don't know if either is really useful for a FITS reader, but they can be useful for getting an idea of the file structure when just reading the primary header. (Randall Thompson) There is a danger in explicit references to either the number of extensions and individual ones. It is trivial to remove or add extensions to FITS files. With explicit references in the primary header, one would need to parse and correct it to maintain valid header information. (Preben Grosbol) This mention of the NEXTEND keyword caused me to rethink what is really meant by EXTEND = F: EXTEND = T is defined to mean that the FITS file *is permitted* to have extensions following the primary array (the actually wording in the Standard is "may contain extensions" ); it does not mean that the file actually has any extensions. If EXTEND = F, then this logically means the opposite, i.e., that the FITS file is *not permitted* to have any extensions. But this is not the meaning that the STEREO mission folks really intended by setting EXTEND = F since they surely don't care if users were to add extensions to the files at a later date. In fact, I can't think of any case where it would really be appropriate to set EXTEND = F. (William Pence) EXTEND does not appear in the index (Mark Calabretta) ------------------------------------------------------------------------ 4.4.1.2 PCOUNT and GCOUNT In principle okay but one should check if this would require existing software to be changed. There may still be legacy tasks which write e.g. Random groups format. The new section 3.7 would not work in that case. (Preben Grosbol) I am not sure what you mean here ? The fact that PCOUNT and GCOUNT are inthe 5th and 6th place in table 4.8 ? But table 6.1 shows (correctly) them in arbitrary position for random groups ! And random groups are NOT a "generalized conforming extension" but something which pre-date them (maybe this could be stated more clearly). (Lucio Chiappetti) It may be a matter of exact wording but the random groups header is the prime header. A FITS reader which does not know about random groups may flag such a header as an error if PCOUNT does not follow directly after the last NAXISn keyword. (Preben Grosbol) Rereading the FITS v3 proposal its wording should be okay with respect to random groups as it is actually a primary header. My main concern is the wording of Sec. 3.7 which may open the door for changes too wide. (Preben Grosbol) The new draft of the FITS Standard (available from http://fits.gsfc.nasa.gov/) does not propose any changes at all to the definition or use of random groups. One of the proposed changes, however, is to require that the PCOUNT and GCOUNT keywords must immediately follow the last NAXISn keyword in all conforming extensions, but this does not apply to random groups. The Standard tries to make a clear distinction between the random groups structure and conforming extensions, so hopefully there should be no confusion between the 2. (William Pence) I guess I'd like to know if there are any such extensions. If not, this is relatively safe. If so, make it a strong recommendation for an explicit list of grandfathered extension types and an absolute requirement for any newly defined extensions. (Rob Seaman) at least some of your FOREIGN extensions have the order of these 2 keywords reversed. (William Pence) We'll look into the behavior you describe. I would expect most extension types, including FOREIGN, to be conformable to this more strict keyword ordering whether it is required or merely preferred. (Rob Seaman) ------------------------------------------------------------------------ 4.4.2.3 the ADS ID is recommended to be used in the REFERENC keyword. Recently I more and more noticed the use of DOI numbers for stuff like this (see http://doi.org/). As this seems to be standardized across many more disciplines than just Physics and Astronomy, should this not be the recommended way to store a reference? Or perhaps recommend this in addition to the ADS reference? (Peter Weilbacher) ------------------------------------------------------------------------ 4.4.3.1 Reference to specific extensions As there are many recommendations in the document, it would be good to retain a deprecation of explicit reference to other extensions. The point is broken links if HDU's are moved. This opens the old issue on how to create a unique reference to any HDU. It would be an advantage for some applications, such as in interferometry, where cross-references between binary table sometimes are used. (Preben Grosbol) ------------------------------------------------------------------------ Table 4.9 and 7.7 It would look nicer to start the tables with BITPIX 8 (Preben Grosbol) ------------------------------------------------------------------------ Section 4: Since the description of WCS keywords like CDELTn was moved to section 8, it would be good to have a reference to that section in section 4. (Preben Grosbol) ------------------------------------------------------------------------ 7.2.2 and 7.3.2 TTYPEn The reason for avoiding '-' was the mapping of the TTYPEn names into variables is some languages. It is irritating to have '-' in keywords for the same reason and most applications would just substitute them with '_'. I would either retain the old wording or if '-' should be allow at the same point deprecate the usage of it. (Preben Grosbol) ------------------------------------------------------------------------ 7.2 It may be good to repeat a reference to fortran in the description of the TDISPn keyword (Preben Grosbol) ------------------------------------------------------------------------ 7.2.5 [which disallows embedded spaces in numeric fields of an ASCII table] are there any examples of such usage in the field? (Rob Seaman) No, as far as we know. If there are any, then it is very likely that most current software systems do not support embedded spaces in the value and will silently read an incorrect value, or will exit with an error. Thus, it seems better to me to outlaw this usage rather than just not recommend it or deprecate it. (William Pence) If there are no known instances, "outlawing" is equivalent to clarifying the standard. This is likely such a case. If there are many instances, I don't think we can escape from taking a position on what the software should do. (Rob Seaman) ------------------------------------------------------------------------ 7.2.5 Deprecate implicit decimal point We should check it with the data centers. The reason for including such cases was to accommodate direct encoding of legacy tables which could have used such formats to save space (at the time when a punch cards was real and only had 80 columns). (Preben Grosbol) ------------------------------------------------------------------------ 7.3.3.2 that would "remove the option to use the heap and the PCOUNT keyword in ways other than described in the variable length array section 7.3.5". This wording makes me nervous because tricks with the heap and the gap area are an obvious potential escape hatch that we left in the Binary Tables Agreement in 1991. I decided to examine the color-coded differences document (very nice!). I see changes marked that seem to be related to this item, but I am unsure which changes accomplish the goal of the item. Perhaps this item in the Recommended Changes document should have some text added to clarify this point, or maybe the differences document is missing some color-coding. (Don Wells) The recommendation is to make a subtle change the wording of section 7.3.3.2, which describes how the heap should be used, from this: "One use for this data area is described in section 7.3.5." to this: "The use of this data area is described in section 7.3.5." The current ambivalent wording (i.e. "One use" instead of "The use") is left over from when the variable length array convention was only described in an unofficial appendix. Now that this convention has been officially approved as part of the FITS Standard (in 2005), it seems appropriate to state more definitely how the heap is intended to be used. This is more consistent with all the other data structures (e.g. the primary array, or random groups) that are defined in the FITS Standard to be used in only a single specific way. The technical panel also recommends deleting the blanket statement at the end of this section ("This does not preclude other uses for these bytes.") for a similar reason. This boilerplate statement was routinely added at the end of all the conventions that were described in the unofficial appendices in the FITS standard. Now that this convention has been officially moved into the body of this standard, this disclaimer is not necessary nor strictly appropriate. As Don rightly points out, this change would eliminates a potential "escape hatch" in FITS that conceivably could serve a useful purpose in the future. However, I would argue that it is better to eliminate this ambiguity for now; if someone does come up with a clever alternate use for the heap in binary table, then the IAU FITS Working Group always has the power to modify the standard to allow this new use in the future, as long as it does not conflict with the use of the heap in existing FITS (William Pence) ------------------------------------------------------------------------ 7.3.5 I encountered two sentences that seem to me to be inconsistent. The first is "The meaning of a negative value for either of these integers [in the array descriptor] is not defned by this standard." The second is "The storage referenced by an array descriptor must lie entirely within the heap area; negative offsets are not permitted." The first sentence leaves open the option of a negative offset (and that is an obvious escape hatch, because the integers are definitely *signed*), but the second sentence closes off the option. (The text of the second sentence is present in v2.1, so this isn't the change made for item 21; I don't remember when this prohibition was introduced; certainly the IAU-FWG could remove it if they wish.) Apparently the second sentence overules the first sentence, but the situation makes me nervous. Negative offsets would address the gap area, and this might be the basis for some clever design that I cannot imagine at this moment. However, I see nothing to stop a designer from using simple integer fields in the binary table to contain pointers that her software would construe as data descriptors for the gap area. I have decided not to request removal of the negative offset prohibition, but I do suggest that the apparent conflict of the two sentences be reviewed. (Don Wells) Back in 2005 when the IAU FITS Working Group approved the variable length array convention as part of the FITS Standard, it was decided to specifically define that the 2 integers in the "P" array descriptor are "signed" 32-bit integers (and not "unsigned" integers) and to add the statement that "The meaning of a negative value for either of these integers is not defned by this standard." Allowing negative array descriptor values is not necessarily a contradiction to the later statement that "negative offsets [in the heap] are not permitted", because it is possible to define a convention that maps (via some mathematical function) a negative descriptor value into a positive heap offset value. Note that the same thing was done long ago with the BITPIX keyword; originally the BITPIX value had to be positive, but later the definition was extended so that if the value is negative then the actual number of bits per pixel is derived by multiplying the BITPIX value by -1. Arguably, the most useful function for mapping a negative array descriptor value into a positive heap offset is this one: heap_offset = 2**32 + descriptor_value = 4294967296 + descriptor_value (where the descriptor_value is a negative signed 32 bit integer in the range -1 to -2147483648). This function effectively doubles the useful size of the heap: positive descriptor values can refer to heap offsets in the range 0 to 2147483647, and the negative descriptor values map into heap offsets in the range 2147483648 to 4294967295. Incidently, it is no coincidence that this mapping function produces exactly the same positive heap offset value as would be given if one were to interpret the 32-bit descriptor value as an unsigned 32-bit integer. (William Pence) I'm really also concerned by this inconsistency raised by Don and I did ask the same question before. I don't think it's wise to try to allow heap offset to be larger than 2**31 -- if the offset is a signed integer, then the limitation of the heap offset should be 2**31. If more is wished, then move to 64-bit integers. I'm sure the gain in space is minimal, just the rare cases when the heap size is between 2**31 and 2**32 would diminish the size of the pointers. And 32-bit machines won't anyway be able to use heaps larger than 2**31 anyway. (Francois Ochsenbein) And now that we have P and Q pointers, if one needs, wants and can go beyond 2**31, one has just to use Q. I thought this issue was settled long ago. (Lucio Chiappetti) ------------------------------------------------------------------------ 8.1 where CROTA2 is mentioned it should be stated in this context that, while it may occur with CDi_ja, it must not occur with PCi_ja. (Mark Calabretta) ------------------------------------------------------------------------ Table 8.1 and elsewhere in the related text: The 'a' symbol used for the alternate code should be in math font. (Mark Calabretta) ------------------------------------------------------------------------ Table 8.2: it would be better if RESTFREQ was presented in the same way as RADECSYS. A footnote describing these forms would be useful. (Mark Calabretta) ------------------------------------------------------------------------ Table 8.2 The note at the bottom reverses the meaning of i and j. In that note, k also refers to column number in a pixel list. (Mark Calabretta) ------------------------------------------------------------------------ 8.2.1 There are a number of keywords, ijPCna, ijCDna, iVn_ma, and so on, where 'a' could be pushed right out of the 8-char keyword name for plausible values of i, j, k, n, and m. In such cases 'a' is still said to be "blank" although it is not the blank character. (This isn't really made clear in Paper I.) (Mark Calabretta) ------------------------------------------------------------------------ 8.2 [in reponse to an extended discussion about whether the CDELTn keyword may = 0] The proposed new FITS Standard document does mention at one point in the text that CDELTn must be non-zero, but this restriction should probably also be stated in the actual definition of the CDELTn keyword. (William Pence) ------------------------------------------------------------------------ 8.2 it is noteworthy that we made a subtle, though important mistake in the definition of WCSAXESa by allowing it to take a default value. The effect of this is to nullify both NAXIS and WCSAXESa and force parsers to be implemented in two passes - not a catastrophe but probably not what we had in mind! My notes for wcspih() say this: * Use of the WCSAXESa keyword is not mandatory and with its default value * of "the larger of NAXIS and the largest index of these keywords [CRPIXj, * PCi_j or CDi_j, CDELTi, CTYPEi, CRVALi, and CUNITi] found in the FITS * header" it effectively invalidates the use of NAXIS for determining the * number of coordinate axes and forces a preliminary pass through the header * to determine the "largest index" in headers where WCSAXESa was omitted. * Also since the use of WCSAXESa is optional there is no way to determine * the number of coordinate representations (the "a" value) other than by * parsing all of the WCS keywords in the header; even if WCSAXESa is * specified for some representations it cannot be known in advance whether * it was specified for all of those present in the header. Hence the * scanner must be implemented in two passes. Note how WCSAXESa nullifies itself and takes out NAXIS for good measure! It actually becomes a liability if its value is smaller than the largest index. (Mark Calabretta) ------------------------------------------------------------------------ 8.3 Is keyword "RADECSYS" an allowed alias for "RADESYS" for the primary WCS? I didn't see that option in WCS Paper II. (Phil Hodge) RADECSYS appeared in earlier drafts of Paper II but was subsequently replaced with RADESYSa. RESTFREQ in early drafts of Paper III was replaced by RESTFRQa, but RESTFREQ is still mentioned explicitly because no doubt everyone still uses it. It would be appropriate for RADECSYS and maybe RESTFREQ to be marked in Table 8.2 as deprecated. The broader question is what a FITS writer should be allowed to write, and what a FITS reader should be prepared to read. Obviously the latter must be prepared to exorcise greater evils than the former is allowed to produce! RADECSYS falls into this category and much more besides, as indicated by the usage notes appended for the WCSLIB header parsers wcspih() and wcsbth() (the latter being a work in progress). (Mark Calabretta) ------------------------------------------------------------------------ 8.3 probably should indicate that no time system (UTC, TAI, TCB, etc.) is currently defined for MJD-OBS or any of the other time- related keywords. (This is not important for anything in the existing WCS papers.) (Mark Calabretta) ------------------------------------------------------------------------ 8.4, definition of RESTWAV: Misspelling of "Vaccuum" (Mark Calabretta) ------------------------------------------------------------------------ Table 8.6 the footnote mark should be given in the caption as in table 8.5 (Mark Calabretta and Peter Weilbacher) ------------------------------------------------------------------------ 8.4.1 the reference to Table 8.6 should be 8.7. (Mark Calabretta) ------------------------------------------------------------------------ 8.4.1 the discussion at the bottom should indicate that the diurnal Doppler correction is only weakly dependent on position and therefore great accuracy is not required. (Mark Calabretta) This depends on the context, and on what's meant by "no great accuracy": it's 1 m/s for 7 arcsec (worst case position on the sky), so for planet searches we need arcsecond precision or so (our best precision is under 1 m/s, and we want some safety margin to keep that particular contribution to the error budget negligible). Perhaps just give the 7" --> 1 m/s equivalence here and let readers make their own decision? (Thierry Forveille) And for pulsar timing it's even more relevant to get the position right. But even at the level of seven arcseconds on the surface of the earth there are issues because some (pre-1980) geodetic datums can produce topocentric positions which are off by many hundreds of meters. In Paper III we did not want to get into the specifics of geodetic datums. For that reason we stopped at recommending post-1980 coordinates. Whether that means WGS-84 or ITRF or ETRF or GTRF or any other satellite-centered, VLBI-aligned reference frame is irrelevant at the level of 1 m (and below that level everything is so heavily model dependent that FITS WCS III didn't dare to tread). For the quality and interpretability of any data with such precision nothing can substitute for having read Lindegren and Dravins and being familiar with the work of IAU Comm 52 (and all the efforts of its predecessors RCMAM and others). I'm not much more sure how all that would fit into the standard than I was at the time we were considering Paper III. (Steve Allen) In the context of the discussion on the bottom of p92, by position I meant location on Earth, not direction towards the source. By "great accuracy is not required" I meant accuracy better than a few metres! On the Earth's surface 1m = 32mas, which ought to be good enough even for planet hunters. As Steve says, the subtext is that Paper III wasn't about to tackle keywords for terrestrial reference frames - they typically agree to within 0.1m. (Mark Calabretta) ------------------------------------------------------------------------ Table 8.7? footnote: how could CUNITia (not CUNITka) substitute for VELOSYSa anyway? Or should the footnote really refer to the units of VELOSYSa specified later? (Mark Calabretta) ------------------------------------------------------------------------ 8.4 in the description of ZSOURCEa, the reference to ZSOURCEa should be SSYSSRCa. (Mark Calabretta) ------------------------------------------------------------------------ Section 8: there are a few anomalies in the set of WCS keywords as published. The main one is that no bintable or pixel list equivalent of DATE-OBS is defined. Also, the pixel list form for WCSNAMEa was mistakenly specified as TWCSna instead of WCSNna. I can do no better than to append an extract of the usage notes for the WCSLIB 4.3 function wcshdo() (as yet unreleased) which writes out a WCS data structure as a FITS header. This might be a good opportunity to legitimise these extended usages - I note that Table 8.2 has already captured WCSHDO_TPC and WCSHDO_PV from the errata. (Mark Calabretta) ------------------------------------------------------------------------ Appendix A, It would be better to name Appendix A 'Formal Syntax of Keyword Records' to be in line with section 4.1 (Preben Grosbol) ------------------------------------------------------------------------ Appendix A, Formal Syntax of Keywords Why not use BNF notation (c.f. wikipedia)? If X... means *one* or more, then the definition of FITS_value_keyword_record incorrectly forces a space in byte 11. Likewise, keyword_field should be keyword_field := [keyword_char...] | [keyword char...] [space...] There are other anomalies, e.g. what does the grammar make of COMMENT = '???' / Is this a comment card or what? HISTORY = 'more or less bunk!' / Says Henry Ford (Mark Calabretta)