Michael "Monty" Widenius, the lead MySQL developer, contests the benchmarks compiled by Great Bridge, and reported on by several prominent websites like
ApacheToday and
Slashdot.
Note: The following been edited for English. Edited copy posted 10:10am MST August 16, 2000
Our apologies to Monty for posting the undedited copy. - DevShed
I would really like to comment against the last PostgreSQL benchmark test, if there would be anything to comment. As they haven't released any information about the test or even the results one can just assume that they have done a test that doesn't have anything to do with the real world :(
All databases have some good and weak areas and it's very simple to design a test that tests the things others are weak at. MySQL 3.22 is for example not very good for mixed inserts + selects on the same table and that is also the only thing we ever have seen PostgreSQL users test in their previous tests when PostgreSQL comes outs as better. PostgreSQL has of course also a lot of weak areas, like slow connection, slow insert, slow create/drop tables, running long multiple transactions where you get a conflict at the end (in this page/row locking is better) and the need to run vacuum() from time to time (especially when you do a lot of deletes / updates). We will update our benchmark comparison web page with PostgreSQL 7.0.2 shortly, to give another picture of this story. The only way to know which database is best for your application is to write a simulation of the applications and test both databases.
We here at MySQL we have always tried to design very fair tests that no one can misinterpret or misrepresent. Anyone can download our benchmarks and run them themselves to verify our test results. One could argue that we don't measure everything, but we do add new tests all the time and we also accept tests from database users that want to test specific things against a lot of databases. This will, in the long run, make our tests very fair and easier to use as a decision tool when choosing a database.
It's a shame that Great Bridge funds a test that is solely done to confuse users instead of telling the truth; PostgreSQL is good in some areas, and bad in others, just like all other databases. No database can be used to solve all problems (at least not without providing a lot of hints for the database and a lot of dedicated code to solve the problem) and you can't win all benchmarks. If one starts claiming this they are on very thin ice!
I hope that Great Bridge doesn't end up in a lawsuit from the TPC organization by misusing their test. This may backfire on all open source databases as I have never seen any commercial database publish such a bad press release and the commercial companies may think that this is the way things are done in the open source world :(
The way to set up the databases in a test is also very crucial for the performance of the database. The article doesn't mention anything about this or even with which ODBC driver they used the different databases. The defaults for MySQL are for a database with moderate load that should take very little resources. MySQL also have two ODBC drivers, one slow with debugging and one fast. It would be very nice to know how they actually did use MySQL. To get any performance from Oracle, one has also to tune this a lot; The ODBC driver for Oracle also has very bad performance; This is a common known fact; No one runs a critical system with Oracle and ODBC. (I use Oracle here as an example, as they press release implicate that they are using Oracle for testing; If not, they are not testing against the proprietary database leaders).
One thing that also is interesting is that they don't mention which PostgreSQL version they are using. It's very unlikely that they did actually test PostgreSQL 7.0 as this has at least one very fatal bug in the index handling which made it useless for benchmarks (at least when we did a test run on it). If they have tested another version, a tuned PostgreSQL with non standard patches and non standard setup, they have broken even more rules. According to the given benchmarks numbers, it looks as if they have run PostgreSQL without disk syncing, which would be fine if they would have told it in the article and if they would also have done this with the other databases.
I agree that MySQL 3.22 is weak in the case where you do a lot of inserts + updates + selects on one table, but it's, on the other hand, equally strong if you don't do that; In MySQL 3.23 you can intermix insert + select but not updates. This will be fixed with BDB tables, but a release with this is still a few weeks into the future. Threads also gives MySQL better scalability than processes, that PostgreSQL uses, so we are very confident about the future.
I also don't agree with the argument that they are not testing MySQL 3.23 as this is still marked 'beta'. We have here at MySQL a completely different release schedule on our versions. We don't mark anything as release if there has been any significant bug report for the gamma version for a month. Compared to our release schedule, PostgreSQL 7.0 would be called alpha (I don't mean anything bad with this; Many users use our alpha version in a production environment with good results)
The net result is that the posted benchmark is about as dishonest as a benchmark can be, the important thing is that the people who read the press release understand that.
Regards, Monty
| DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware. |
More BrainDump Articles
More By Michael "Monty" Widenius
 developerWorks - FREE Tools!
|
This demonstration gives you an overview of IBM® Rational® Build Forge Express Edition, a global offering that provides a framework to automate and execute software processes. Rational Build Forge provides a software assembly line that can support all of your tools, technologies, and platforms so you can achieve a repeatable, reliable, and traceable build and release process.
FREE! Go There Now!
|
|
|
|
CakePHP is a stable production-ready, rapid-development aid for building Web sites in PHP. This "Cook up Web sites fast with CakePHP" series shows you how to build an online product catalog using CakePHP.
FREE! Go There Now!
|
|
|
|
Set up a PHP Web interface for the Java(TM) business application using a database created in earlier in this series. The PHP Web interface collects information from users and sends the session data to the Java business application for processing and for a response.
FREE! Go There Now!
|
|
|
|
Visit IBM developerWorks to download a free trial version of WebSphere Extended Deployment Compute Grid, which lets you schedule, execute, and monitor batch jobs. Because online transaction processing and batch jobs execute simultaneously on the same server resources, you can avoid costly duplication of resources. Compute Grid supports job types of Java transactional batch, compute-intensive and a new type called "native execution", which enables non-Java workloads to run on distributed end points.
FREE! Go There Now!
|
|
|
|
Regression testing -- in which code is thoroughly tested to ensure that changes have not produced unexpected results -- is an important part of any development process. But many testing environments neglect the terminal-based applications that still form the backbone of many industries. In this tutorial, you'll learn how the Rational Functional Tester Extension for Terminal-Based Applications works with other Rational Functional Tester to help test terminal-based applications quickly and easily.
FREE! Go There Now!
|
|
|
|
Investigate the effects of field-level caching in dynamic array access, in part 5 of the UniVerse performance series.
FREE! Go There Now!
|
|
|
|
Get a free trial download of the latest version of IBM Rational Tester for SOA Quality V7.0.1, a functional and regression testing tool that enables the creation, comprehension, modification and execution of testing GUI-less Web services.
FREE! Go There Now!
|
|
|
|
Learn how IBM Rational Tester for SOA Quality addresses IBM WebSphere MQ with Web services. You get hands-on experience in creating a test, handling the WebSphere MQ series protocol, configuring the test, and then replaying it.
FREE! Go There Now!
|
|
|
|
It's a good time to be a Web developer. You've never had more choices in terms of technologies. There are so many great open source Web servers, databases, programming languages, and development frameworks. No matter what combination of technologies you prefer to work with, there is a single integrated development environment (IDE) that can increase your productivity: Eclipse. Here in Part 3, we introduce the RDT and RadRails Eclipse plug-ins and show you how to get these plug-ins and start using them. You will learn how to use RadRails to do many common Ruby on Rails development tasks.
FREE! Go There Now!
|
|
|
|
IBM Lotus Notes 8 provides a wide range of developers the ability to provide customized, integrated user interfaces via composite applications and via custom sidebar and toolbar plug-ins. This webcast provides you with tips and techniques to use with out-of-the-box capabilities of Lotus Notes 8, and survey how you can share useful components within your own company and within a larger community.
FREE! Go There Now!
|
|
|
|
All FREE IBM® developerWorks Tools!
|
/ 8 
