In "Software Engineering and Epistemology" in the April 1995 SEN, C. Michael Holloway has offered not only the software engineering community, but also the software quality community, a challenge I believe they cannot ignore: justify why we do what we do. Defining epistemology as "'How do I know that what I know is true,'" Holloway describes three typical epistemic approaches: argument from authority, proof by reason, and verification by experience. He calls for the creation of a "valid epistemological foundation" for software engineering to replace our current attraction for "'methodologies' and 'processes'" which, he appears to suggest, are based on neither "logical or experimental evidence"[1].
As a phenomenologist, I am inclined to suggest Holloway's epistemological preference conflates truth with agreement; the correspondence of abstract terms with 'real' referents, or the formal adequacy of logical deductions or inductions is one, but not the only, way to define truth. On the other hand, a Heideggerian definition of truth as "uncoveredness"[2] is not likely to spur a 'scientific revolution' in engineering. And I am certainly not suggesting we ignore Holloway's challenge merely because of differences in method or because of apparent disparity between concepts of "truth" or "knowledge." In fact, such differences may well be prerequisite to meeting his goals. Let me explain.
Paul K. Feyerabend is one of several philosophers of science who have examined 'scientific revolutions.' Feyerabend's position is that in the history of science 'progress' is characterized by the emergence of incommensurable theories that challenge the veracity of their predecessors. What had passed as "proper" empirical method becomes, on Feyerabend's historical retrospective, merely methodological dogmatism [3]. That position has not gone unchallenged, of course, and the literature is much too extensive to survey quickly here. I cite Feyerabend only to caution that we not take "scientific method," "logic," "experimental," "evidence" and other such terms at face value.
Philosophers of science have for decades haggled over what constitutes proper
induction from data or deduction from theoretical principles. And it is
problematic how one would construct strictly controlled experiments involving
the creative, and often temperamental, subjects we call software developers.
Certainly one can conduct investigations and reference popular statistical or
information theory maxims to support one's method. But methodological
conformity is not equivalent to adequacy, truth, knowledge,
etc.
Take for example a recent survey described in the Quality Management Journal [4]. The authors undertook what they call an "empirical study" of the differences in meaning between "total quality management" and "quality management." They scanned articles for these terms and then compared the incidence of those terms in a variety of contexts (e.g., management styles, etc.). The article goes to great length to justify its methodology and prove the statistical significance of its findings. Their statistically justified margin of error does not change the fact that some of the data they used was gained through semiotic interpretation - a decidedly non-empirical activity whose precise explanation has eluded philosophers, linguists, and psychologists for centuries.
It's details like this we tend to forget when we are in the grip of a methodology. This is not to deny statistical analysis its place, merely to suggest that the software engineering and quality community must be aware that the "principle of total evidence" may not be restricted to class-theory [5], but may instead have to deal with significantly less cooperative entities such as language, perception, and cognition. Even if one agrees with Holloway's view (if I read him correctly) that "anecdotal" arguments are epistemically specious, that does not justify naively embracing methodology that passes for analytical science.
I hazard the opinion that there is room in "science" for the subjective (experiential, anecdotal) as well as the abstractly theoretical, and much in between. I see no epistemological hierarchy between statistically elegant conclusions and a discussion between friends about how they like today's weather; I see different language constructs, different objectives to their use, different types of "agreement." Nor am I convinced that "empirical studies" are ipso facto more "valuable" than studies informed by other disciplines simply because they conform to metalinguistic formalism. After all, formally valid syllogisms can have patently false conclusions, and even the best programmer amongst us has fallen prey to garbage in, garbage out.
If the history of ideas has anything to teach us other than humility, it is that we improve both discipline (method) and knowledge incrementally, and not necessarily synchronously: sometimes progress requires Galilean obstinacy, refusing to 'see the evidence' in light of the prevailing 'scientific' mores [3]; sometimes it takes the patience of an Aristarchus waiting centuries until Newtonian theoretical constructs "sanctify" one's observations [6]. Along these lines, Burian and Martin have suggested that Feyerabend's conclusions from the historical evidence are too extreme: logical differences between theoretical concepts do not make statements derived from those concepts irreconcilable [7,8]. As I suggested above, it may be a matter of how one defines "agreement." Conflict between theories and methods helps us to recognize the unique advantages and inadequacies of each, showing us, as it were, the 'tolerances' of the tools we use to organize and increase our knowledge.
I took my title from Paul Feyerabend's paper, "How To Be a Good Empiricist: A Plea for Tolerance in Matters Epistemological" to suggest that the software engineering and quality community could well benefit from a debate over methods and concepts similar to that which Feyerabend helped start [9]. I have in mind here a thorough analysis and comparison of the methodologies available to us. I do not pretend to know what Holloway means by a "valid epistemology" (italics mine), but certainly the software engineering and quality community could start its analysis by defining "scientific" in light of several decades of debate in the philosophy of science. And we could all benefit from a close look at how linguistics, cognitive psychology, and sociological factors affect our choice of "methods," the evidence we chose to work with, and the conclusions we draw from that evidence.
Holloway's challenge has already been answered, if implicitly, by those editors and conference referees who accept only material boasting "empirical" or statistical foundations. Whether in the pages of SEN or in some other forum, those with expertise in appropriate disciplines (linguistics, cognitive psychology, philosophy, etc.) should take up the challenge as well. In "Philosophy of Science 2001," Feyerabend warns: "One of the most important and least discussed tasks of a democracy is to protect its members from the ideologies it contains" [10]. Reticence may not be Feyerabend's salient characteristic, but his point is worth considering: at stake is progress, as opposed to conformity, in the disciplines of software engineering and quality assurance.