Most of what I dislike about the open source movement can be summed up in one word: Hype. Unfortunately, and perhaps surprisingly, the advocates of open source are no better in this regard than their proprietary colleagues.

Best People
The claim is frequently made that open source programmers are the best programmers around. One author, apparently acting on input from open source zealots, said things like "Linux is the darling of talented programmers," and opined that the open source movement was "a fast-forward environment in which programming's best and brightest ... contribute the most innovative solutions" (Sanders 1998). Is there any truth to those claims? My answer is "No", for several reasons.

Attempts to define Programmer Aptitude Tests, which evaluate the capabilities of subjects to become good programmers, have historically been failures. In an early study, the correlation between computer science grades and practitioner achievement, was found to be negative. Although some programmers are better than others, nothing in the fields' research suggests that we have found an objective way of determining who those best people are.

Since we can't identify who the best people are, there is no way to study the likelihood of them being open source programmers. Thus, those who claim that open source people are software's "best and brightest" cannot possibly support those claims with any factual evidence. It is an interesting characteristic of programmers that most of them tend to believe that they are the best in the field. Certainly, I know that few programmers are better than me! It used to be a standard joke in the software field that, if a roomful of programmers were asked to self-rate themselves, none of them would end up in the second tier. Therefore, I suspect that if you took any group of programmers, including open source programmers, and asked them if they were the best and brightest, they would answer in the affirmative. To focus on open source quality claims, let's take a look at reliability and security.

Most Reliable
The claim is also frequently made that open source software is the most reliable software available. In this case, there are some studies containing interesting data. The first thing that should be said about open source reliability is that its advocates claim that a study identified as the "Fuzz Papers" (The Fuzz Papers) produced results that showed that their software was more reliable than proprietary alternatives.

I obtained the papers, read and analyzed them, and contacted their author to investigate the matter even further. The bottom line is, the Fuzz Papers have virtually nothing to say about open source software, one way or the other, and their author agrees with that assessment. He does say, however, that he personally believes that open source may well be more reliable. It is truly bizarre that anyone would claim that these peculiar studies of software reliability actually support the notion that open source code is reliable. To understand why I say "peculiar" you should read them yourself!

Most Secure
Analogous to the reliability claims, there are many claims that open source is more secure. The more that public and industry concern for security increases, the louder those claims become.

There is very little evidence on either side of the ledger regarding open source software and security. Certainly security holes have been found in proprietary software. Certainly also, holes have been found in open source code (for example see SP 2002). And both sides have made strong claims that their software is either the most secure, or that they are making it so.

Probably the most accurate statement one can make about software security is this: It is all too easy for programmers to leave holes, independent of how the code is being written (for a list of the top five security-related software defects, see SP 2003). The perversity of "crackers" is that wherever they seek security holes, they are likely to find them. Furthermore, they tend to hunt wherever the loudest claims are that the software is secure! For example, in the book Know Your Enemy (Honeypot Project 2002), there is a study of cracker techniques using "honeypot" systems to trap the crackers. One "black hat" was specifically going after Linux-based .EDU systems because of their claims of invulnerability, a chilling thought for both open source advocates and academics who use their wares.

With respect to the open source claims, there is plenty of anecdotal evidence (e.g., SP 2003b) to back the security claims of the open source advocates, as well as their proprietary counterparts. However, there is really no definitive evidence to cause either side to be seen as victorious.

So where do I stand on open source? I see nothing in particular, wrong with its fundamental ideas and ideals. But I see plenty wrong with the hype surrounding it. Not that it's any worse than its proprietary brethren in this respect. It's just that I expected more from this particular group! Yes, I do expect more from the open source advocates.

Author's Note
This column is derived, in open source fashion, from the upcoming book Making Sense of the Bazaar: Perspectives on Open Source and Free Software, O'Reilly and Associates, 2003 (available in early fall).

References

• The Fuzz Papers. A series of studies of utility/operating system software reliability, beginning with one published in 1990, another performed in 1995 and published in early 2000, and the most recent in the USENIX Windows Systems Symposium, Aug. 2000. Contact the author, Prof. Barton P. Miller of the University of Wisconsin Computer Science department, for further details.
• Glass 1999. "The Realities of Software Technology Payoffs," Communications of the ACM, Feb. 1999; Robert L. Glass.
• Glass 2002. Facts and Fallacies of Software Engineering, Addison-Wesley, 2002, Robert L. Glass.
• Honeypot Project 2002. Know Your Enemy: Revealing the Security Tools, Tactics, and Motives of the Blackhat Community, Addison-Wesley, 2002, by The Honeypot Project members.
• Sanders 1998. "Linux, Open Source, and Software's Future," IEEE Software, Sept. 1998, J. Sanders.
• SP 2002. "Holes Found in Open Source Code," the Software Practitioner, Sept. 2002.
• SP 2003. "Security-Related Software Defects: a Top-Five List," The Software Practitioner, Jan. 2003.
• SP 2003b. "Software Security: Which is Better, Open Source or Proprietary?" The Software Practitioner, Jan. 2003.
• Zhao 2000. "A Survey of Quality Related Activities in Open Source," Software Engineering Notes, May 2000, Luyin Zhao and Sebastian Elbaum.