Memorandum: January 4, 1999 ----------- from J.-F. Grivaz to P. Dornan, W.-D. Schlatter, A. Blondel cc P. Gay, N. Konstantinidis The purpose of this memorandum is to express and substantiate my disagreement with the decisions regarding the Hnn analysis, taken by the spokesman on the day following the HTF meeting of December 17. I understand that the advice of the chairman of the steering committee and of the Thursday meeting organizer has been taken by the spokesman, which is the reason why this memorandum is also addressed to them. In addition, a copy is sent to the conveners of the Higgs task force. This memorandum is intended to be made available to the ALEPH collaboration soon after being delivered to its addressees. 1. Synopsis: ------------ In a recent ALEPH-note, Marumi Kado and myself gave a detailed account of a new analysis designed for the search for the standard model Higgs boson in the Hnn channel. The development of this analysis had been reported by Marumi in a series of talks to the Higgs task force in October and November. The essential features of this analysis, its history, and a comparison with the so-called reference analysis are presented in Sections 2 to 4 of this memorandum. During the HTF meeting of December 17, a long discussion took place regarding the merits and the opportunity of the new analysis. In contrast to what had been the case for the previous presentations by Marumi, the spokesman and the chairman of the steering committee were present, but obviously unaware of the matter under discussion. The Thursday meeting organizer was not more present that time than in the previous instances. A summary of the arguments exchanged at the December 17 meeting, and of its conclusions, is given in Section 5. In the late afternoon of December 18, and after consulting the chairman of the steering committee and the Thursday meeting organizer, the spokesman sent a mail to inform the HTF of his decisions regarding the Hnn analysis. Those decisions have little to do with what seemed to be the conclusions of the HTF meeting held on the preceeding day. In addition to being unacceptable from a purely deontological point of view, it is my belief that the decisions taken by the spokesman, and the way they are taken, are bad for the Higgs task force and bad for ALEPH. The spokesman's mail, with my comments to it interleaved, can be found in Section 6, followed in Section 7 by my conclusions. 2. Features of the revisited Hnn analysis: ------------------------------------------ Although all the necessary material can be found in the note entitled ``The Hnn analysis revisited'' submitted by Marumi and myself on December 9 (ALEPH/98-096 PHYSIC/98-045), a brief account of the specific features of this analysis is given here. The initial goal was to perform a full reassessment of the reference analysis (ALEPH/98-009 PHYSIC/98-009), rather than to try to marginally improve its performance by added details here and there. The first result was a new preselection (ALEPH/98-087 PHYSIC/98-037), proposed by Cal and Marumi and adopted for the LEPC and other future analyses. The next developments were performed in view of an optimal result for a 95 GeV signal mass, close to the expected sensitivity for the standard model Higgs boson at 189 GeV. The issue of the dependence of the analysis performance on the signal mass was to be addressed in a later stage. For neural network based analyses, this problem can be solved either through the commonly accepted method of a signal training sample composed of various signal masses (the cocktail approach), or through some new technique allowing several neural networks, each optimal at a given mass, to be interpolated (the sliding neural network approach). Such techniques have already been proposed by Hongbo (October 8 HTF) and by Marumi (November 19 HTF). In the revisited Hnn analysis, the two main backgrounds are first addressed separately, using kinematic features only. To discriminate the signal against the qqbar background: After an essentially harmless additional preselection, a seven variable neural network (NN) was trained. The seven variables are identical to or can be reconstructed from variables used in the reference analysis. They were chosen for their obvious physical interpretation. The E_12 variable was however not considered at this point because it is too sensitive to the beam related background. To discriminate the signal against the WW background: Background events with an obvious (W-->l/tau nu) decay are first removed by simple cuts on the lepton isolation (if there is such a lepton) and on the tau minijet energy. The latter variable was used in the reference analysis while the lepton isolation is a new variable, inherited from SUSY searches. A three variable NN was then trained. Again, the three variables are identical to or can be reconstructed from variables used in the reference analysis. At this point, a four variable NN was trained, using as inputs the outputs of the above anti-qqbar and anti-WW NN's, and of the b-tagging NN for the two most energetic jets. For the training, all backgrounds were taken into account (except for the irreducible ZZ-->bbnn process), in amounts in accordance with their contributions after all preselections. The final touch is a cut on E_12, totally harmless in the absence of beam related background. The basic philosophy sustaining the revisited analysis is that each problem should be addressed in turn and understood on its own, unless correlations prevent from doing so efficiently. The performance can be summarized by an expected background level of 5.4 events for a 40% efficiency on a 95 GeV mass Higgs signal. 3. History of the revisited Hnn analysis: ----------------------------------------- http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf081098.ps http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf211098.ps http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf051198.ps http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf191198.ps The above references direct to transparencies presented by Marumi in HTF meetings on October 8 and 21, and on November 5 and 19. It is therefore difficult to argue that the collaboration was faced, at the last minute, with a completely new analysis, especially in view of the fact that the essential features were already present in the October 21 presentation. Indeed, the following mail was sent to the HTF by the conveners on October 22: Dear all, At the end of yesterday's meeting, a quick review of the analyses status (hZ and hA) was done. Some of you had to leave before the end, thus we will give a summary here. After a review per channel, it was concluded that the extra time needed to obtained the finalised versions will be small (not more than four weeks). Thus people would have this extra time to finalise the *current* developments. Then, It will be nice to have the different persons involved in each analysis in touch all together in order to obtain the convergence as soon as possible. The main issues after that are the systematics, the background subtraction, the problem of the overlap between the analyses,...etc. Cheers, Pascal + Nikos The deadline of four weeks to finalize the *current* developments was met with Marumi's presentation on November 19. Following the conveners' recommendations, people involved should have met at that point to converge as soon as possible. A meeting was organized for that purpose on December 3 right after the regular HTF. The Wisconsin group decided to boycott that meeting. A second attempt was made on December 16, which was also cancelled because Xidong had left for vacation... In the mean time, a comprehensive note on the revisited Hnn analysis was issued and advertized by a mail to the HTF, thus allowing any motivated member to ask questions or to raise objections. 4. Comparison of the LEPC and revisited analyses: ------------------------------------------------- Except for the new preselection proposed by Cal and Marumi last September (ALEPH/98-087), the Hnn LEPC analysis is a straightforward extrapolation of the 183 GeV analysis (ALEPH/98-009). The only modifications to the eleven variable NN inputs were the removal of the track isolation angle, shown to be useless, and the introduction of the missing mass, necessitated by the softer requirement on that variable in the new preselection (Xidong at the Mainz HTF). The number of input variables happened that way to remain unchanged. In the absence of any solid documentation, the only uncontroversial anchor for the Hnn LEPC analysis is the information which had been provided to Patrick in view of the combination of the various channels. It features a background of 8.2 events for a 40% signal efficiency on a 95 GeV mass Higgs boson. The reduction from 8.2 to 5.4 background events, for the same 40% signal efficiency, is quite substantial, and it is interesting to investigate to which specific features it can be attributed. The most obvious one is that the revisited analysis was designed to provide optimal performance at 95 GeV. A LEPC-like analysis, but with a NN trained on a single signal mass of 95 GeV, rather than on a mass cocktail, provides a more adequate reference for a comparison, with a background level of 7.2 events. Training the NN on all relevant backgrounds (rather than on qqbar and WW only as in the LEPC analysis) brings a clear improvement: the background is reduced to 5.7 events. Taking the anti-qq and anti-WW preselections into account improves the performance only slightly, and for efficiencies in excess of 50%. This is not too surprising since most of the variables used in the anti-qq and anti-WW preselections are also input to the eleven variable NN. One exception is the lepton isolation from which a twelve variable NN could benefit (Jennifer at the November 19 HTF). Other combinations were tried: removing E_12 from the NN and adding a post-cut; using the smaller and larger b-tagging NN outputs rather than two symmetric combinations; replacing kinematic variables of the reference analysis by similar ones from the revisited analysis. No further gain was achieved. The remaining improvement, from 5.7 to 5.4 background events, is therefore likely to be due to the ``hierarchical'' structure of the revisited analysis (although this cannot be proved easily). 5. December 17 HTF meeting: --------------------------- Most of the December 17 HTF meeting was devoted to the fate of the revisited Hnn analysis. The main objections raised against it were that this analysis was too fresh to be accepted without further scrutiny, and that it was unchecked. But it was also felt that the benefits brought in terms of performance could not be simply ignored. Finally, more emotional aspects entered the game. Was this analysis really fresh ? As detailed above, all steps in the development of the analysis had been regularly reported to the HTF since October 8. The transparencies presented were detailed and clear enough to allow a thorough understanding of the options taken and of the progress made. The essential features were all available on October 21. In accordance with the conveners' request, the end of all developments in view of Moriond and of the publication was explicitly signified by Marumi on November 19. Finally, a comprehensive ALEPH-note was made available on December 9. Except for the newcomers, among which the addressees of this memorandum are unfortunately to be counted, it is difficult to pretend discovering this new analysis on December 17. Was this analysis unchecked ? In addition to the ALEPH-note describing the new analysis, all of the code pertaining to it had been made accessible on the web through the HTF page, and advertized in the mail calling attention to the ALEPH-note. From this point of view, the situation of the revisited analysis is clearer than that of the LEPC one. The answer to the question ``Was the analysis unchecked ?'' is nevertheless, and unfortunately, yes. This is due to the unavailability of anyone willing to perform the necessary checks. With Cal absent from this meeting, Xidong away until mid-January, Jennifer kindly accepted to make the checks requested in early January. Whether these checks will actually take place or not is difficult to predict at the time of writing. How can the benefits be preserved ? For reasons which I am unable to understand, it seems that the anti-qq and anti-WW preselections or, more importantly, the improved composition of the background training sample, are modifications easy to accept, whereas changes to the neural network structure are viewed as exceedingly involved. This may have to do with the fact that the addressees of this memorandum have little experience with neural networks, which may appear to them more mysterious than they really are. As detailed above, it is not very difficult to improve the performance of the LEPC analysis (from 7.2 to 5.7 background events at 40% efficiency), close to that of the revisited analysis (5.4 events). Should the difference between 5.7 and 5.4 background events be considered as marginal ? In terms of efficiency, it corresponds to a relative improvement of 2% for a 40% absolute efficiency. Around 30 and 50% efficiencies, the corresponding relative improvements are 6 and 3%. In many instances, choices have been made in the HTF based on similar or smaller differences. Unless there are compelling, and therefore explainable, reasons not to do so, this is the only way to preserve, or rather to restore, ALEPH's leadership. Asking questions such as ``By how much does it change the expected limit ?'' is to be put at the same level as ``Why should we care for a 3% degradation of the data taking efficiency ?''. Emotional aspects: Another issue regarding which unbiased arguments are less easy to find is the possible ``unfairness'' which would result from a decision to go ahead with the revisited analysis. Irrespective of the merits of that analysis, such a decision would be unfair to those who had devoted all their strength first to the LEPC analysis, then to the assessment of systematic errors in view of the December 3 Thursday meeting. In fact, in a normal collaboration, it would be a most beneficial sharing of efforts that some secure a result while others prepare the following term, and all would then be proud of the common achievements. Unfortunately, no result was secured at all for the LEPC, and no specific work was reported at the December 3 meeting regarding the background subtraction systematics in the Hnn channel. In practice, the LEPC analysis did not really require any work other than keeping up to date with the data collection. The few modifications, already mentioned in this memorandum, with respect to the 183 GeV analysis had already been made by the time of the Mainz ALEPH week, and the background subtraction, in the absence of systematic studies, is a fairly simple technical issue. That only little effort was required by the Hnn LEPC analysis is demonstrated by the fact that Cal on one side, Jennifer and Xidong on the other, found the necessary time before the LEPC to develop competing analyses for the ZZ-->qqnn channel. At the end of the December 17 meeting, no consensus was reached. My recollection is that one of the conveners wrote a few lines on a transparency stating that the LEPC analysis (defined as what was provided to Patrick for the combination) remains as a back-up analysis until January 14, that the revisited analysis will be cross-checked by Jennifer in early January, that the mass dependence will be handled according to the cocktail method, and that, of the two analyses, the one with the best performance in terms of confidence level will be retained. The spokesman seemingly agreed to this proposal. At least, he did not express any disagreement. 6. The day after the December 17 HTF: ------------------------------------- In the late afternoon of December 18, the following letter was sent by the spokesman to the HTF conveners, to the head of the Wisconsin group, and to me. For an easier understanding of the arguments, my comments are interleaved in the body of the mail, highlighted by *** marks. ******************************************************************************* Dear Colleagues, Yesterday's Higgs meeting was one of the most unpleasant ones I have experienced in ALEPH; it was a bad example for our young physicists. The contrast with the previous day's WW meeting at Annecy could not have been greater. When we have differences we should solve them in a manner which respects all involved. This was not so yesterday. *** On this, I could not agree more, although I have the feeling that the *** main responsibility of what happened does not rest on my shoulders. *** I may elaborate on this later in a private mail. Following the meeting I checked the minutes of the Dec 3rd Thursday meeting. It makes a clear statement that the analyses for Moriond should be based on the ones presented by Nikos once the background systematics are understood. The Higgs group was thus encouraged to concentrate for the immediate future on understanding the background systematics for these. At that time there was no mention of any other Hnunu analysis being ready for the winter conferences. *** Some may find it rather surprising that the spokesman has to go back to *** minutes of meetings to determine what he did or did not approve, and what *** is good or bad for ALEPH. Anyway, it is hard to draw any conclusions from *** those minutes (unavailable on the Thursday meeting web page, by the way) *** other than: 1) the background studies in the Hqq channel give us confidence *** that we are facing a statistical fluctuation, and we should try to proceed *** with full background subtraction for Moriond and for the paper (first *** presentation); 2) the backup analysis ``a la Vancouver'' will be used for *** the early Winter conferences, including La Thuile (second presentation). *** And the overall conclusion was: *** ``The "backup" analyses are approved for the incoming conferences. *** The finalisation of the "full background subtraction" analyses before *** Moriond is recommended (supported by the 189 GeV Higgs search paper).'' *** It is very true that the ongoing developments in the Hnn channel were not *** mentioned at that meeting, the reason being that they had nothing to do *** with the matters discussed. The new developments had never been intended to *** be used for the LEPC. The issue of full, partial or no background *** subtraction is common to all Hnn analyses. The analysis which was shown to us yesterday has some very attractive features and is supported by a very good ALEPH note, however the overall performance is very similar to the one which can be obtained with the standard 11V neural net. *** The revisited analysis was not ``shown to us yesterday''. It has been final *** since November 19. It is too bad that the addressees of this memorandum *** were not cautious enough to be aware of that point. I appreciate that the *** quality of the supporting ALEPH-note is acknowledged. *** The second half of the sentence goes a bit fast. As of today, the only *** officially approved analysis is the LEPC one, and it is that analysis *** which provides the reference performance for the 11V-NN. It is the *** analysis ALEPH would anyway have to live with for Moriond if Marumi and *** I had not been working on the revisited analysis for more than three *** months. With respect to the LEPC analysis, the improvement brought by *** the revisited analysis cannot be questioned. *** The least we could expect is that our agreement be requested for the *** use in the 11V NN of some of the improvements which we made. Unless, of *** course, it is decided to open a new phase of development for Moriond and *** for the paper, in total contradiction with the policy set by the HTF *** conveners, as stated in their October 22 mail quoted above. This morning I heard that we have the Higgs talk at La Thuile which makes it most important that we concentrate on ensuring we have final results with systematics by February 18. Pursuing a new analysis at this time for the winter conferences for what is likely to be a minimal gain in the final limit would not seem wise. In the past we have missed important dates because we have been too ambitious. *** Here too, I find it surprising that the spokesman discovered only on *** December 18 that we have the Higgs talk in La Thuile, while I was already *** aware of the fact (maybe only of its very high probability, which does not *** change a thing). *** The question of systematic studies in the Hnn channel is essentially *** independent of whether it is the LEPC or revisited analysis which is used. *** Nothing prevents these studies from being done, except maybe for the *** unavailability until mid-January of some of those in charge of the *** reference analysis. *** As already explained in this memorandum, pretending that the revisited *** analysis is too new to be used for Moriond is a statement which cannot be *** defended in view of its history. *** How a gain is decided to be minimal is clearly a matter of appreciation. *** I made the point earlier that such statements clear the way towards *** decadence. I also do not think we can ever be too ambitious for ALEPH, *** unless ambition is mistaken for presumption. I rather fear that we now *** become too shy after being too complacent. *** At any rate, we had proposed to keep the LEPC analysis as a back up until *** January 14, thus avoiding any risk that we miss the La Thuile deadline. *** Had we made a similar decision for the LEPC, we would not have had to face *** the situation we know. The error at the time was not to try to do our best *** for the LEPC, it was not to have secured a fall-back solution. Alain, Dieter and I discussed the situation at length this morning. We decided that the safest option was to stay with the analyses directly based on those presented by Nikos at the Dec 3 meeting and to request the Higgs group to push ahead with all speed to perform these analyses on the reprocessed data and understand the systematics. In this way we are confident that, with cooperation from all, we can have final numbers and a first draft of the paper/conf note by February 18. This is to be the priority of the group, followed by the preparation of the 1999 analysis which should run online when data taking resumes. *** Nothing new here. Any option with a fall-back solution is a safe option. *** My personnal point of view (hard to prove, I agree), is that progress *** will be slower in the Hnn channel than it would have been otherwise, if *** only because, as already stated, some key advocates of the reference *** analysis are not currently available. Of course, Marumi and I will do the *** work in the context of the revisited analysis, and a good fraction of it *** should be useful irrespective of whatever choice is made in the end. There are a number of features of the Grivaz-Kado analysis which could be simply implemented in the standard one and we encourage everyone to work together to make such additions. Additionally, if a sound result from this analysis is available for the paper, it would be natural to include it as a supporting analysis as we have so often done in the past. It is our view that the merits, or otherwise, of the multi-neural net structure require time to be appreciated but that it is a strong candidate for the 1999 analysis. *** Making additions now in January to the reference analysis goes well beyond *** the boundary conditions which had been set within the HTF. At the time of *** writing, the LEPC analysis is, apart from the revisited analysis, what *** we have, and I expect that additions to the reference analysis based on *** Marumi's work would not be made without our consent. *** It is very true that supporting analyses have often been included in *** ALEPH papers in the past. As far as I remember, these supporting analyses *** had weaker performances than the ones used to derive the official result. *** Here the reverse would happen, by far if the LEPC analysis is to be used, *** by less, but still by a noticeable amount, if some upgraded reference *** analysis were to be used instead. I do not see how the choice made could be *** explained to the outside world. Probably, some kind of understatement *** such as ``similar performance'' would have to be utilized. The argument *** which would convince me to accept that is still to be found. *** ``Multi-neural net structure'' is a phrase, the purpose of which, I fear, *** is to frighten the reader. In fact, it boils down, in front of a given *** event, to asking three questions: *** 1) Does this event look like a qqbar or like a Higgs event ? *** 2) Does this event look like a WW or like a Higgs event ? *** 3) Is this event well b-tagged ? *** Each of these questions is answered by the value of a NN output. *** This is what we have already done for years with the way the b-tagging *** information is handled. Finally, these NN ouputs are combined into yet *** another neural network, which has the virtue of being trained against *** all relevant backgrounds (i.e., Wen, Zee and ZZ in addition to qq and WW). *** If anybody can explain to me why it is simpler to put all the information *** into a single box, mixing up the tau-minijet energy which addresses *** only the WW background and the acoplanarity (modified or not) which *** addresses only the qqbar background, I will be most grateful. *** It is my belief that the addressees of this memorandum did not pay any *** attention to these questions, and are motivated by considerations which *** have nothing to do with the supposed complexity of the revisited analysis. *** Regarding the time needed to appreciate the merits (or otherwise ?) of *** the approach taken in the revisited analysis, let me remind that it already *** amounts to more than two months and a half. And as to the vague promise for *** the 1999 analysis, I take it about as seriously as those which had been *** made to us in October, and torn in December. We trust that you will all appreciate that this decision has not been taken lightly and it is taken, in our view, in the best overall interest of ALEPH. Peter *** In the end, it is very difficult to understand which decision has really *** been taken. *** 1) Should we stay with the analyses reported by Nikos on December 3, and *** only work on systematics ? From the second paragraph of this mail, the *** answer seems to be yes. This is repeated at length in the fifth paragraph. *** 2) It is also stated in the fourth paragraph that pursuing a ``new'' *** analysis for the winter conferences would be unwise. *** 3) However, mention is made in the third paragraph of a possibly improved *** performance for the reference analysis. And in the sixth paragraph, people *** are encouraged to implement features of the revisited analysis into the *** standard one. This looks in contradiction with the first of the above *** decisions. *** 4) Finally, obtaining a ``sound result'' from the revisited analysis is not *** discouraged, with the possibility of including it somehow in the paper. *** This also looks contradictory to the second decision. *** I am therefore sorry to conclude that, in spite of the spokesman's mail, *** we remain in the dark. P.S. Could the convenors please circulate this to the Higgs group. *** It is of course because of this post-scriptum that I took the liberty to *** reproduce this mail in the present memorandum. ******************************************************************************* 7. Conclusion: -------------- As detailed in my reply to the spokesman's mail, my impression is that the situation has not been clarified at all. The same arguments as those used in that mail can be used to draw different, if not totally opposite, conclusions. For instance: 1) We should by all means have an essentially final result for La Thuile. For that purpose, the safest is to consider the LEPC analysis, as it was delivered at the time of the LEPC, as the uncontroversial back-up solution. 2) We should not embark now into new developments. Therefore, the only contender with respect to the LEPC analysis is the revisited analysis, the development of which was declared finished (as far as the 189 GeV data analysis is concerned) on November 19. This analysis is fully described in an ALEPH-note. 3) We should work on signal and background systematics with the available manpower without any further waste of time. Most of the work should be usable irrespective of the analysis which is used in the end. 4) Once the necessary checks of the revisited analysis are made, and if its improved performance with respect to the back-up solution is confirmed, it will be chosen for the publication. If not, no time will have been wasted and the LEPC analysis can be used. The decision should be taken no later than January 14 (first HTF meeting in 1999). Indeed, these conclusions turn out to be identical to those which a number of us had understood to be drawn at the end of the December 17 HTF meeting.