The Costs/Quality Correlation - BrainDump

If the correlation between costs and results is measured, one might think that, since so much money is invested in quality assurance in big companies, this must either lead to bug-free software or count as a bad investment. Well, everybody knows that there is no bug-free software, and it is a small consolation that with so much money and effort, proprietary code companies still have so many critical bugs that often remain unfixed for too long.

If it sounds more optimistic, you can think what could have happened to software quality without companies spending so much money on it. If quality cannot be achieved when money is invested in it, can you imagine what would have happened if nobody cared about quality? So, it could be worse than now. Well, if money is not the problem what could be the reason that proprietary code companies still operate so inefficiently in comparison to open source software as far as quality is concerned?

A cornerstone concept in open source is that “Given enough eyeballs, all bugs are shallow.” It best describes the approach to bugs –- when many (knowledgeable) people review and use the software, there will always be somebody who will know the fix to even the most mysterious problem.  Sure, with 1.4 testers (not including external reviewers) per developer, you would think there must be enough eyeballs. Is the quality of the eyeballs a factor, or is it something else?

Our Bugs Are Our Trade Secrets

Most proprietary code companies are aware of their bugs (or at the least the most severe ones). But it is the attitude of the company and its management -- “our bugs are our trade secrets” -- that prevents disclosing and/or fixing of bugs. There are many examples of such an attitude, the latest one being the recent Ciscogate.

Sure, not all cases when somebody spots a bug are investigated by FBI and go to court, but this case clearly demonstrates that companies are more interested in hiding the truth at any cost than in correcting their product. Cisco's situation is a typical example of the arrogance with which big corporations deal with their opponents –- and not only in respect to disclosing “top-secret” information that their products have defects that could have a very strong negative impact on the business of many companies which have purchased and use their equipment.

If it were only Cisco and if it were only this case, it would have been fine. (No matter what the outcome of the case is, one point is certain: Cisco’s reputation is seriously damaged, more by their aggressive defense than by the fact that a severe bug has been discovered in their product). But a similar approach to bugs is typical for many proprietary code companies -- when a bug is found, they behave as if this bug is not their fault and try their best not to fix it, but to hide it. 

As a result of the approach proprietary code companies have towards their bugs, it is rare to see a proprietary code company have a publicly accessible bug database, while for most open source companies this is more the rule than an exception. Bug databases are very useful, because information about potential problems is gathered there.

Browsing through a bug database is not done because of curiosity to peep into other people’s mistakes.  Bug databases are not places to show the world how many defects a piece of software has; rather, by giving information about known bugs, they help all users of the software avoid getting into more trouble. When one knows that a particular module has some unrefined issues, he or she will use this module with caution in order not to get trapped. In a way, bug databases are a means of being more honest with your customers and partners.