Although sending log messages to a remote machine can sound advantageous, there are disadvantages that you need to be aware of. Here are the advantages of logging on a remote machine:

• A cracker won’t be able to delete or modify the logs after breaking into the system that runs Apache. The cracker could still violate the log server, however. Therefore, the log server should have much heavier security and accept only log requests, to minimize security risks.

• A cracker won’t be able to perform a log-oriented DOS attack against the server that runs Apache (filling all the disk space with long requests, for instance), because the logs are written elsewhere. A cracker can try to fill up the partition that contains the log files, however, but the direct target server of such an attack wouldn’t be the web server.

• There are no file permission problems on the log files, because they don’t reside on the local machine. You could argue that the logging machine would have the same issues, but such a machine shouldn’t have any users or external access, and it should be solely dedicated to logging.

• If you have several servers, you can make sure that all the logs are stored in one spot in the network. This would make organization of backups, log merging, and log analysis much easier.

There are several disadvantages to remote logging using syslog:

• It is unreliable. A log line could simply get lost on the way to the log server. There is no acknowledgment of any sort by the remote logging server regarding the information being written. An attacker could cause a denial of service on the remote logger by flooding it, or feeding it bad data.

• Centralizing logging to one server means that the log server represents a single point of failure.

• It is simple to create fake log entries. syslog’s protocol is based on UDP, and it is extremely simple to send a forged or spoofed UDP packet, because UDP is connectionless. You need to firewall the network carefully, and even then, a cracker might be able to create misleading log entries.

• It’s based on clear text. This means that the information can be forged on its way to the log server, because there is no reliable mechanism for checking that the packet that arrived is the same as the packet that was sent.

Some of these problems are structural and cannot be easily solved.

Secure Alternatives

The syslog option has pervasive structural vulnerabilities, but studying its problems and vulnerabilities will give you a good start to designing a remote logging architecture, without making the same mistakes again.

Here are alternatives that help to solve some of the shortcomings:

• syslog-ng (http://www.balabit.hu/en/downloads/syslog-ng/) This is a replacement for the standard syslog daemon. syslog-ng (the ng means next generation) can be configured to support digital signatures and encryption, to make sure that log messages weren’t modified. It can also filter log messages according to their content, and log messages using TCP rather than UDP (TCP is a connection-oriented protocol and is much harder to spoof ). Additionally, it can run in a jailed environment (see Chapter 6 for detailed information on how to run Apache in a “jail” using chroot()).

• nsyslogd (http://coombs.anu.edu.au/~avalon/nsyslog.html) This is another replacement of the syslog daemon, with some interesting features, including the use of TCP instead of UDP, and the ability to encrypt connections to prevent data tampering using OpenSSL. Unfortunately, while it is a feasible solution for several Unix systems, it doesn’t work well on Linux at the time of this writing.

• socklog (http://smarden.org/socklog/) This is yet another replacement of the syslog daemon. The main strength of socklog is that it’s based on daemontools (http://cr.yp.to/daemontools.html). daemontools’ architecture makes it possible to have encryption, authentication, compression, and log rotation quite easily. It’s definitely worth looking into.

• The e-mail available at http://cert.uni-stuttgart.de/archive
  /honeypots/2002/07/msg00100.html includes tips on how to hide a remote log server.

Syslog has several limitations. For example, as I mentioned, it is not feasible to store the access log information on syslog on a production server. In this section, I will explain how to configure your server so that it logs encrypted information over the network.

Two Possible Designs

To write logs over the net, you need to use a custom written program to log Apache’s messages on a remote server. You can configure Apache this way:

CustomLog "|/usr/bin/custom_logging_program" common
ErrorLog "|/usr/bin/custom_logging_program"

The program custom_logging_program could be a program that reads the log messages from its standard input, and sends them to a remote server via a TCP connection. The remote log server would be a program written specifically to talk to custom_logging_program. This way, all the problems connected with syslogd could be solved, and you would be absolutely sure that the system works the way you want it to work. There is only one problem: designing a functional client/server application like this can be very complicated. At the beginning of this chapter, I mentioned that logging programs should be kept as simple as possible. A program like this would be anything but simple; it would need to use cryptography, and would need to communicate with the server following all the rules set by the (newly designed) protocol. Writing the logging server program would be even harder.

To simplify the solution’s design, you could use an Apache module that would deal with SQL logging directly. A very good choice, for example, is http://www.grubbybaby.com/mod_log_sql/. The main advantage of this is that it is very easy to configure. You pay for this simplicity in terms of lack of flexibility, because:

• You cannot encrypt the log information.

• You can only deal with the access log (not the error log).

• You can only log using MySQL as a SQL back-end.

Basically, the first solution is powerful, but often too complex to be implemented (reinventing the wheel often means rediscovering the same bugs and vulnerabilities). The second solution is much easier to configure, but has very strong limitations.