Reading the Log Files - Apache

There is one issue that seems to be overlooked by many system administrators: it is important to read and analyze log files. As I mentioned earlier, Apache has two types of logs: the error log and the access log.

An ideal error log on a running server is an empty one (apart from information about the server starting and stopping) , when the error level is set to notice. For example, a “File not Found” error probably means that there is a broken link somewhere on the Internet pointing to your web site. In this case, you would see a log entry like this:

[Sat Oct 05 20:05:28 2003] [error] [client 127.0.0.1] File
does not exist: /home/merc/public_html/b.html, referer:    
http://localhost/~merc/a.html

The webmaster of the referrer site should be advised that there is a broken link on their site. If there is no answer, you might want to configure your Apache server so that the broken link is redirected to the right page (or, if in doubt, to your home page).

If crackers are looking for possible exploits, they will generate “File not Found” entries in the error log, so keeping the error log as clean as possible will help to locate malicious requests more easily. Some exploit attempts are logged in the error_log. For instance, you could find:

[merc@localhost httpd]$ grep -i formmail access_log
[Sun Sep 29 06:16:00 2003] [error] [client 66.50.34.7]
script not found or unable to stat: /extra/httpd/cgi-bin/formmail.pl
[merc@localhost httpd]$

The formmail script is widely used, but it generates a number of security issues.

A segmentation fault problem needs attention as well. Apache should never die, unless there is a problem in one of the modules or an attack has been performed against the server. Here is an example:

[Sun Sep 29 06:16:00 2002] [error] [notice] child pid 1772
exit signal Segmentation fault (11)

If you see such a line in the log file, you will have to see what was going on at the time in the server’s activity (possibly reading the access_log file as well) and consider upgrading Apache and its modules as soon as possible. Because of Apache’s extensive use and deployment, most such problems in the core Apache package have been eliminated. Therefore, a segmentation fault message usually implicates an after-market or third-party module failure, or a successful DOS attack.

The Access Log

The access log includes information about what the user requested. If the error log reports a segmentation fault, you can use the access log to find out what caused Apache to die. Remember that if the cause of death is really sudden, because of buffering issues, the latest log information might not be in the log file.

You can also use the access log to check whether someone is trying to break into your system. Some attacks are easy to identify by checking for the right string in the log. You can find the entries for many Windows-aimed attacks just by looking for the exe string in the access log. For example:

[root@localhost logs]# grep -i exe access_log
200.216.141.59 - - [29/Sep/2003:06:25:22 +0200] "GET /_vti_bin/shtml.exe HTTP/1.0" 404 288
200.216.141.59 - - [29/Sep/2003:06:31:33 +0200] "GET /_vti_bin/shtml.exe HTTP/1.0" 404 288
193.253.252.93 - - [02/Oct/2003:02:17:53 +0200] "GET /scripts/..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ HTTP/1.1" 404 319
151.4.241.194 - -  [02/Oct/2002:02:34:46 +0200] "GET /scripts/..%255c%255c../winnt/system32/cmd.exe?/c+dir" 404 -
[root@localhost logs]#

The main problem with using grep to look for attacks: URLs can be URL-encoded (see Appendix B for more information). This means that the last entry you saw in the access_log shown above could be written as:

151.4.241.194 - - [02/Oct/2003:02:34:46 +0200] "GET
/scripts/..%255c%255c../winnt/system32/cmd.%65x%65?/c+dir" 404 -

When encoded in this way, this URL would escape the grep filter. To perform an effective search, you will need to URL-decode all the URLs from your log files (you can use Perl for this), and then compare them with the suspicious ones. This simple script (Listing 3-1) should come in handy.

Listing 3-1. A Simple Script to Use As a Filter

#!/usr/bin/perl
use URI::Escape;
use strict;

# Declare some variables
#
my($space)="%20";
my($str,$result);

# The cycle that reads
# the standard input
while(<>){

   # The URL is split, so that you have the
   # actual PATH and the query string in two
   # different variables. If you have
   # http://www.site.com/cgi-bin/go.pl?query=this,
   # $path = http://www.site.com/cgi-bin/go.pl
   # $qstring = "query=this"
   my ($path, $qstring) = split(/\?/, $_, 2);

   # If there is no query string, the result string
   # will be the path...
   $result = $path;

   # ...BUT! If the query string is not empty, it needs
   # some processing so that the "+" becomes "%20"!
   if($qstring ne ""){
           $qstring =~ s/\+/$space/ego;
           $result .= "?$qstring";
   }

   # The string is finally unescaped...
   $str = uri_unescape($result);

   # ...and printed!
   print($str);
}

Note that the script is slightly complicated by the fact that a + (plus) in the query string (and only in the query string) must be converted into %20 ($qstring =~ s/\+/$space/ego;), which is then translated into a space once the string is URL-decoded ($str = uri_unescape($result);).

You should call this script urldecode, place it in /usr/local/bin, and give it executable permission (chmod 755 /usr/local/bin/urldecode). To test it, just run it:

[root@localhost logs]# urldecode
hello
hello
this is a test: .%65x%65
this is a test: .exe
[root@localhost logs]#

The script acts as a filter as it echoes information to the standard output. The command to test your logs should now be:

[root@merc root]# cat access_log | urldecode | grep exe

You can change exe into anything you want to look for in your log.