Benchmarking Tomcat Performance

Apache Jakarta JMeter

JMeter can be run in either graphical mode or in text-only mode. You may run JMeter test plans in either mode, but you must create the test plans in graphical mode. The test plans are stored as XML configuration documents. If you need to change only a single numeric or string value in the configuration of a test plan, you can probably change it with a text editor, but it’s a good idea to edit them inside the graphical JMeter application for validity’s sake.

Before trying to run JMeter to run a benchmark test against Tomcat, make sure that you start JMeter’s JVM with enough heap memory so that it doesn’t slow down while it does its own garbage collection in the middle of trying to benchmark. This is especially important if you are doing benchmark testing in graphical mode. In the bin/jmeter startup script, there is a configuration setting for the heap memory size that looks like this:

  # This is the base heap size -- you may increase or decrease it to fit your
  # system's memory availablity:
  HEAP="-Xms256m -Xmx256m"

It will make use of as much heap memory as you can give it; the more it has, the less often it may need to perform garbage collection. If you have enough memory in the machine on which you’re running JMeter, you should change both of the256numbers to something higher, such as512. It is important to do this first because this setting’s default could skew your benchmark test results.

To create a test plan for the benchmark, first run JMeter in graphical mode, like this:

  $ bin/jmeter

JMeter’s screen is laid out as a tree view on the left and a selection details panel on the right. Select something in the tree view and you can see the details of that item in the details panel on the right. To run any tests, you must assemble and configure the proper objects in the tree, and then JMeter can run the test and report the results.

To set up a benchmark test like the one we did above with both ab and siege, do this:

1. In the tree view, right click on theTest Plantree node and selectAdd -> Thread Group. 
   
2. In theThread Groupdetails panel, change theNumber of Threads (users)to149, change theRamp-Up Period (in seconds)to0, and theLoop Countto671.
   
3. Right click on theThread Grouptree node and selectAdd -> Sampler -> HTTP Request.
   
4. In the HTTP request details panel, change theWeb Serversettings to point to your Tomcat server and its port number, and change thePathunder theHTTP Requestsettings to the URI in your Tomcat installation that you would like to benchmark. For instance/.
   
5. Right click on theThread Grouptree node again and selectAdd -> Post Processors -> Generate Summary Results.
   
6. In the top pull-down menu, selectFile -> Save Test Plan as and type in the name of the test plan you wish to save. JMeter’s test plan file extension is .jmx, which has an unfortunate similarity to the unrelated Java Management eXtension (JMX).

Figure 4-1 shows the JMeter GUI with the test plan, assembled and ready to run. The tree view is on the left, and the detail panel is on the right.

Once you are done building and saving your test plan, you are ready to run the benchmark. ChooseFile -> Exitfrom the top pull-down menu to exit from the graphical JMeter application. Then, run JMeter in text-only mode on the command line to perform the benchmark, like this:

  $ bin/jmeter -n -t tc-home-page-benchmark.jmx
  Created the tree successfully
  Starting the test
  Generate Summary Results = 99979 in 71.0s = 1408.8/s Avg:   38 Min:    0 Max:
  25445 Err:     0 (0.00%)
  Tidying up ...
  ... end of run

Notice that the requests per second reported by JMeter (an average of1408.8requests per second) is significantly lower than that reported by both ab and siege, for the same hardware, the same version of Tomcat, and the same benchmark. This demonstrates that JMeter’s HTTP client is slower than that of ab and siege. You can use JMeter to find out if a change to your webapp, your Tomcat installation, or your JVM, accelerates or slows the response times of web pages; however, you cannot use

Figure 4-1.  Apache JMeter GUI showing the fully assembled test plan

JMeter to determine the server’s maximum number of requests per second that it can successfully serve because JMeter’s HTTP client appears to be slower than Tomcat’s server code.

You may also graph the test results in JMeter. To do this, run JMeter in graphical mode again, then:

1. Open the test plan you created earlier.
   
2. In the tree view, select theGenerate Summary Resultstree node and delete it (one easy way to do this is to hit the delete key on your keyboard once).
   

3. Select theThread Grouptree node, then right click on it and selectAdd -> Listener -> Graph Results.

4. Save your test plan under a new name; this time for graphical viewing of test results.

5. Select theGraph Resultstree node.

Now, you’re ready to rerun your test and watch as JMeter graphs the results in real time.

Again, make sure that you give the JMeter JVM enough heap memory so that it does not run its own garbage collector often during the test. Also, keep in mind that the Java VM must spend time graphing while the test is running, which will decrease the accuracy of the test results. How much the accuracy will decrease depends on how fast the computer you’re running JMeter on is (the faster the better). But, if you’re just graphing to watch results in real time as a test is being run, this is a great way to observe.

When you’re ready to run the test, you can either selectRun -> Start from the top pull-down menu, or you can hit Ctrl-R. The benchmark test will start again, but you will see the results graph being drawn as the responses are collected by JMeter. Figure 4-2 shows the JMeter GUI graphing the test results.

Figure 4-2.  Apache JMeter graphing test results

You can either let the test run to completion or you can stop the test by hitting Ctrl-. (hold down the Control key and hit the period key). If you stop the test early, it will likely take JMeter some seconds to stop and reap all of the threads in the requestThread Group. To erase the graph before restarting the test, hit Ctrl-E. You can also erase the graph in the middle of a running test, and the test will continue on, plotting the graph from that sample onward.

Using JMeter to graph the results gives you a window into the running test so you can watch it and fix any problems with the test and tailor it to your needs before running it on the command line. Once you think you have the test set up just right, save a test plan that does notGraph Results, but has a
Generate Summary Resultstree node so that you can run it on the command line, and then save the test plan again under a new name that conveys the kind of test it is and that it is configured to be run from the command line. Use the results you obtain on the command line as the authoritative results. Again, the ab benchmark tool gives you more accurate benchmark results but does not offer as many features as JMeter.

JMeter also has many more features that may help you test your webapps in numerous ways. See the online documentation for more information about this great test tool at http://jakarta.apache.org/jmeter.