Michael "Monty" Widenius, the lead MySQL developer, contests the benchmarks compiled by Great Bridge, and reported on by several prominent websites like ApacheToday and Slashdot.
Note: The following been edited for English. Edited copy posted 10:10am MST August 16, 2000
Our apologies to Monty for posting the undedited copy. - DevShed
I would really like to comment against the last PostgreSQL benchmark test, if there would be anything to comment. As they haven't released any information about the test or even the results one can just assume that they have done a test that doesn't have anything to do with the real world :(
All databases have some good and weak areas and it's very simple to design a test that tests the things others are weak at. MySQL 3.22 is for example not very good for mixed inserts + selects on the same table and that is also the only thing we ever have seen PostgreSQL users test in their previous tests when PostgreSQL comes outs as better. PostgreSQL has of course also a lot of weak areas, like slow connection, slow insert, slow create/drop tables, running long multiple transactions where you get a conflict at the end (in this page/row locking is better) and the need to run vacuum() from time to time (especially when you do a lot of deletes / updates). We will update our benchmark comparison web page with PostgreSQL 7.0.2 shortly, to give another picture of this story. The only way to know which database is best for your application is to write a simulation of the applications and test both databases.
We here at MySQL we have always tried to design very fair tests that no one can misinterpret or misrepresent. Anyone can download our benchmarks and run them themselves to verify our test results. One could argue that we don't measure everything, but we do add new tests all the time and we also accept tests from database users that want to test specific things against a lot of databases. This will, in the long run, make our tests very fair and easier to use as a decision tool when choosing a database.
It's a shame that Great Bridge funds a test that is solely done to confuse users instead of telling the truth; PostgreSQL is good in some areas, and bad in others, just like all other databases. No database can be used to solve all problems (at least not without providing a lot of hints for the database and a lot of dedicated code to solve the problem) and you can't win all benchmarks. If one starts claiming this they are on very thin ice!
I hope that Great Bridge doesn't end up in a lawsuit from the TPC organization by misusing their test. This may backfire on all open source databases as I have never seen any commercial database publish such a bad press release and the commercial companies may think that this is the way things are done in the open source world :(
The way to set up the databases in a test is also very crucial for the performance of the database. The article doesn't mention anything about this or even with which ODBC driver they used the different databases. The defaults for MySQL are for a database with moderate load that should take very little resources. MySQL also have two ODBC drivers, one slow with debugging and one fast. It would be very nice to know how they actually did use MySQL. To get any performance from Oracle, one has also to tune this a lot; The ODBC driver for Oracle also has very bad performance; This is a common known fact; No one runs a critical system with Oracle and ODBC. (I use Oracle here as an example, as they press release implicate that they are using Oracle for testing; If not, they are not testing against the proprietary database leaders).
One thing that also is interesting is that they don't mention which PostgreSQL version they are using. It's very unlikely that they did actually test PostgreSQL 7.0 as this has at least one very fatal bug in the index handling which made it useless for benchmarks (at least when we did a test run on it). If they have tested another version, a tuned PostgreSQL with non standard patches and non standard setup, they have broken even more rules. According to the given benchmarks numbers, it looks as if they have run PostgreSQL without disk syncing, which would be fine if they would have told it in the article and if they would also have done this with the other databases.
I agree that MySQL 3.22 is weak in the case where you do a lot of inserts + updates + selects on one table, but it's, on the other hand, equally strong if you don't do that; In MySQL 3.23 you can intermix insert + select but not updates. This will be fixed with BDB tables, but a release with this is still a few weeks into the future. Threads also gives MySQL better scalability than processes, that PostgreSQL uses, so we are very confident about the future.
I also don't agree with the argument that they are not testing MySQL 3.23 as this is still marked 'beta'. We have here at MySQL a completely different release schedule on our versions. We don't mark anything as release if there has been any significant bug report for the gamma version for a month. Compared to our release schedule, PostgreSQL 7.0 would be called alpha (I don't mean anything bad with this; Many users use our alpha version in a production environment with good results)
The net result is that the posted benchmark is about as dishonest as a benchmark can be, the important thing is that the people who read the press release understand that.