How to Benchmark PHP Scripts for Speed

Suppose a test is done comparing the speed between two possible approaches in coding PHP that yields the following average results:

Test 1 average execution time = 630 Microseconds
Test 2 average execution time =602 Microseconds

You cannot simply conclude and recommend that the Test 2 approach is faster and better simply because it has a lower average or mean (which means faster). Numerically, 602 microseconds < 630 microseconds and the difference between the results is 630 – 602 = 28 microseconds. The main question, however, that is of so much importance to PHP developers/coders/programmers is this: “Does the difference of 28 microseconds really matter in terms of performance?” or stated another way: “Does the difference of 28 microseconds really significantly improve the script in terms of speed?”.

What you are facing every time you code are a lot of suggestions on how to optimize speed, but these may not be really significant at all. Instead of making your program run faster and more efficient, it actually contributes too little to be considered worthwhile. In some cases it can even make your program more complex and harder to maintain.

According to Donald Knuth: http://en.wikipedia.org/wiki/Donald_Knuth “Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%”

The Scientific Method: Comparing Performance

The most reliable methods of comparing performance are those that are used by scientists and engineers in their studies. For small samples (let's say 10 observations), you can use a Student's t-test method to objectively compare performance. This is a form of statistical testing:

References:
http://www.physics.csbsju.edu/stats/t-test.html

The following are steps on how to conduct the test:

Step 1: Set up the test scripts in a localhost environment. Since most web hosts use PHP 5 nowadays (as of July 2011), PHP 5 is used throughout this test. Specifically, below is the system used:

Operating system: Ubuntu 10.04
PHP version: 5.3.1
Server: Apache 2.0 in XAMPP
Browser used: Firefox 5.0

To make these tests relate to other previously done tests, we will be using similar benchmarking scripts such as thoughs used by http://www.phpbench.com.

Step 2: Clear cache before running any test in the browser. Echo the execution time in the browser and gather the data.

Step 3: Repeat the test 10 times to get 10 observations.

Step 4: Analyze the data using t-test. To speed up the analysis, an online tool will be used, such as the one found here: http://www.physics.csbsju.edu/stats/t-test_bulk_form.html

Establish the following hypothesis':

• Null hypothesis: There is NO significant difference between Group A and Group B.

• Alternative hypothesis: There is a “significant” difference between Group A and Group B. And if Group A < Group B in terms of execution time then Group A is significantly faster. Or if Group B < Group A then Group B is significantly faster.

Rules for rejecting Null hypothesis: Reject null hypothesis if the probability of the result is less than 0.05 (using a 95% confidence level). This level of confidence has only around a 5% chance of error.

Illustration #1: PHP Echo vs PHP Print

One of the most commonly asked questions pertaining to PHP speed practices is: “Which is faster: PHP echo or PHP print? Is the speed differences significant?"

Sample Test code:
Credits: http://www.phpbench.com

<?php
function Test1() {
global $result;
ob_start();
$x = microtime(true);
while($i < 1000) {
echo 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx';
++$i;
}
$temp = microtime(true) - $x;
ob_end_clean();
return $temp;
}
echo Test1();
?>

For print, simply replace:

echo 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx';

With:

print'xxxxxxxxxxxxxxxxxxxxxxxxxxxx';

Group A Execution time (in seconds) for echo (10 observations):

0.00019288063049316
0.00019717216491699
0.00020122528076172
0.00020694732666016
0.00019693374633789
0.00020098686218262
0.00020909309387207
0.00019598007202148
0.0002131462097168
0.00020503997802734

Group B Execution time (in seconds) for print (10 observations):

0.00020098686218262
0.00022482872009277
0.00020098686218262
0.00020194053649902
0.00020313262939453
0.00022697448730469
0.00022697448730469
0.00021600723266602
0.00023007392883301
0.00020217895507812

T-test Analysis Results:

t= -2.54
sdev= 0.101E-04
The mean of Group A(echo) is: 201.9405
The mean of Group B(print) is: 213.4085

Degrees of freedom = 18. The probability of this result, assuming the null hypothesis, is 0.020

Box plot of mean (average):

Conclusion and Recommendation

Based on the analysis results, you can conclude that php echo is indeed significantly faster than print at a 95% confidence level. Based on the box plot, the execution time of echo is significantly lower and more stable than print (which has wider spread).

As a recommendation, you should use PHP echo when outputting HTML to the browser to get speed benefits.

Illustration #2: Single vs Double Quotes When Outputting

Suppose you want to find out which one is faster:

Group A: Single quote and concatenated variable

$concat='Hello World';
echo 'The quick brown fox jumps over the lazy dog'.$concat;

Group B: Or simply this (double quotes and enclosing the variable):

$concat='Hello World';
echo "The quick brown fox jumps over the lazy dog$concat";

Sample Test Code:

<?php
function Test2() {
global $result;
ob_start();
$x = microtime(true);
while($i < 1000) {
$concat='Hello World';
echo 'The quick brown fox jumps over the lazy dog'.$concat;
++$i;
}
$temp = microtime(true) - $x;
ob_end_clean();
return $temp;
}
echo Test2();
?>

Group A Execution time in seconds:

0.0005650520324707
0.0006711483001709
0.00056099891662598
0.0005648136138916
0.00061702728271484
0.00056099891662598
0.0005640983581543
0.00056099891662598
0.00056099891662598
0.00056600570678711

Group B Execution time in seconds:

0.00069093704223633
0.0006868839263916
0.00075387954711914
0.00067687034606934
0.00067901611328125
0.00067496299743652
0.00069189071655273
0.00075507164001465
0.00074100494384766
0.00067996978759766

T-test Analysis Results:

t= -7.94
sdev= 0.349E-04

The mean of Group A is: 579.214 microsecond
The mean of Group B is: 703.049 microsecond

Degrees of freedom = 18. The probability of this result, assuming the null hypothesis, is less than .0001. The probability of this result, assuming the null hypothesis, is less than .0001

Box plot of mean (average):

Conclusion and Recommendation

Using single quotes is faster than double quotes; indeed, when you look at their box plot, the mean and variation of execution time using single quotes is small compared to double quotes.

As a recommendation to output something to the browser, it is faster to use single quotes and the concatenation method at a 95% confidence level.

Illustration #3: For Loop vs While Loop in PHP

Both for and while do loops are commonly used looping methods in PHP programming. But which one is really faster?

Test code used:

<?php
function Test3() {
$t = microtime(true);
$i = 0; while($i < 1000000) ++$i;
return (microtime(true) - $t);
}
echo Test3();
?>

Simply replace:

$i = 0; while($i < 1000000) ++$i;

With:

for($i = 0; $i < 1000000; ++$i);

When evaluating For loops.

Group A (While loop) data:

0.12238907814026
0.11577606201172
0.10313296318054
0.11513304710388
0.15706586837769
0.14488697052002
0.10994005203247
0.14042115211487
0.11743116378784
0.2314338684082

Group B (For Loop) data:

0.10314702987671
0.14091300964355
0.13284397125244
0.13269805908203
0.13518095016479
0.12527084350586
0.14642190933228
0.17303395271301
0.1328661441803
0.18761301040649

T-test analysis result:

t=-0.371
sdev= 0.316E-01

The mean of Group A is: 0.136 Second
The mean of Group B is: 0.141 Second

Degrees of freedom = 18. The probability of this result, assuming the null hypothesis, is 0.71

Box plot of mean (average):

Conclusion and Recommendation

This is a classic example of why you cannot judge just by looking at the averages or mean. Using the “while loop” is 0.136 second while the “For loop” is 0.141 second. The “While loop” is only faster by around 0.005 seconds. It is evident by their box plots being so similar. According to t-test analysis results, the probability is 0.71 - which is greater than 0.05. Thus, you cannot reject the null hypothesis and can conclude that there no significant difference in terms of speed.

As a recommendation, you can use either “while” and “for loops” in your code without worrying about its efficiency in terms of speed performance. If a specific section in your code requires a “while loop”, then use it, otherwise use a “for loop”. It won't slow anything down.

Summary

There are still a lot of remaining issues in PHP regarding speed benchmarks. By following the methods outlined in this article, a PHP developer or researcher can reliably determine whether a certain change in code can significantly contribute to speed or not.