KEEPING AN EYE ON WEB SERVERS so that they run smoothly is an essential part of the administrator’s job, for both performance and reliability purposes. Sensible logging helps to detect performance problems well before they become apparent to users, and provides evidence of potential security problems.

In this chapter I will first explain how to configure Apache for logging purposes, highlighting the most common problems. I will then introduce remote logging using syslogd, the standard logging server that comes with Unix. Finally, I will propose a remote logging solution, which will allow you to encrypt logging information and store it on a remote database using MySQL.

Log files show you what your daemons are doing. From a security perspective, Apache’s log files are used for:

• Logging requests made and pages served in order to identify “suspicious” requests.

• Logging Apache’s extra information, such as errors and warnings. This information is very interesting, because an attack generally creates some abnormal entries.

The importance of log files is often underestimated. Sometimes, even in important production servers, they are left there to grow and grow, until one day they make themselves noticed because they have filled up the file system.

Configuring Logging in Apache

I will give an overview of how to configure log files in Apache. Remember that this is not a comprehensive explanation, and for more information you should look at Apache’s official documentation: http://httpd.apache.org/docs/logs.html.

There are two types of log information in Apache: the access log (handled by the module mod_log_config) and the error log.

The access log records every request sent to the web server. A typical configuration is:

LogFormat "%h %l %u %t \"%r\" %<s %b" common
CustomLog logs/access_log common

Here, LogFormat sets a log format and assigns a name to it, common in this case. Then, Apache is instructed to log access information in the file logs/access_log, using the format defined in the previous line (common). To find out the exact meaning of each parameter, check Apache’s documentation. You will find out that Apache can log almost anything pertaining to a request, including the client’s address and the type of request itself. The log file format just described is the most common for HTTP requests (for example, IIS is capable of generating the same result), hence its name.

Apache server’s error messages are logged separately, using a different file. In this case, there is no definite format for the messages, and these directives are defined:

ErrorLog logs/error_log
LogLevel warn

The first directive, ErrorLog, instructs Apache to log all the errors in logs/ error_log. The second directive sets the minimum importance for a message to be logged (the “level” of the message). These levels are defined in Table 3-1.

Table 3-1. Apache Error Levels

Remember that if you decide to set the log level to crit, the messages for more important levels will be logged as well (in this case, alert and emerg).

Sometimes, it is advisable to delegate all the logging to specifically developed parsing engines or archiving utilities. When Apache is started, it runs the logging program and sends all the logging messages to it. This solution is valid in many situations. For example:

• When you don’t want to stop and restart your Apache server to compress your logs.

• When you have many virtual hosts. If you use a different log file for each virtual host, Apache will need to open two file descriptors for every virtual domain, wasting some of the kernel’s and processes’ resources.

• When you want to centralize your logging into one single host. The program specified in the configuration could send the log lines elsewhere instead of storing them locally, or for increased reliability, it could do both.
  
• When you want to create a special log filter that watches every log request looking for possible security problems.

There are some disadvantages to using an external program. For example, if the program is too complex, it might consume too much CPU time and memory. In addition, if the external program has a small memory leak, it might eventually chew up all the system’s memory. Finally, if the logging program blocks, there is a chance of causing a denial of service on the server.

To delegate logging to an external program ( piped logging), you can use the following syntax:

CustomLog "|/usr/local/apache2/bin/rotatelogs
/var/log/access_log 86400" common

The command /usr/local/apache2/bin/rotatelogs /var/log/access_log 86400 is run by apache at startup time.

In this case, the program rotatelogs will be fed the log lines by Apache, and will write them on /var/log/access_log. Remember that you can use the same syntax for Apache’s error log using the directive ErrorLog. For more information about how CustomLog and ErrorLog work, refer to Apache’s official documentation.