HACK#20: Deal with Disk Hogs - Administration

Fortunately, you no longer have to be a script guru or a find wizard just to keep up with what is happening on your disks.

Think for a moment. What types of files are you always chasing after so they don't waste resources? Your list probably includes temp files, core files, and old logs that have already been archived. Did you know that your system already contains scripts capable of cleaning out those files? Yes, Im talking about yourperiodicscripts.

Periodic Scripts

Youll find these scripts in the following directory on a FreeBSD system:

  % ls /etc/periodic/daily | grep clean
  100.clean-disks
  110.clean-tmps
  120.clean-preserve
  130.clean-msgs
  140.clean-rwho
  150.clean-hoststat

Are you using these scripts? To find out, look at your /etc/periodic.conf file. What, you don't have one? That means you've never tweaked your default configurations. If that's the case, copy over the sample file and take a look at what's available:

  # cp  /etc/defaults/periodic.conf
  /etc/periodic.conf
  # more /etc/periodic.conf

daily_clean_disks. Let's start with daily_clean_disks. This script is ideal for finding and deleting files with certain file extensions. You'll find it about two pages into periodic.conf, in the Daily options section, where you may note that it's not enabled by default. Fortunately, configuring it is a heck of a lot easier than using cron to schedule a complex find statement.

Before you enable any script, test it first, especially if it'll delete files based on pattern-matching rules. Back up your system first!

For example, suppose you want to delete old logs with the .bz2 extension. If you're not careful when you craft yourdaily_ clean_disks_filesline, you may end up inadvertently deleting all files with that extension. Any user who has just compressed some important data will be very miffed when she finds that her data has mysteriously disappeared.

Let's test this scenario. I'd like to prune all .core files and any logs older than .0.bz2. I'll edit that section of /etc/periodic.conf like so:

Notice my pattern-matching expression for the .bz2 files. My expression matches any filename (*) followed by a dot and a number from one to nine (.[1-9]), followed by another dot and the .bz2 extension.

Now I'll verify that my system has been backed up, and then manually run that script. As this script is fairly resource-intensive, I'll do this test when the system is under a light load:

  # /etc/periodic/daily/100.clean-disks 
  Cleaning disks:   /usr/ports/distfiles/MPlayer-0.92.tar.bz2
  /usr/ports/distfiles/gnome2/libxml2-2.6.2.tar.bz2
  /usr/ports/distfiles/gnome2/libxslt-1.1.0.tar.bz2

Darn. Looks like I inadvertently nuked some of my distfiles. I'd better be a bit more explicit in my matching pattern. I'll try this instead:

  # delete old logs, cores  
  daily_clean_disks_files="messages.[1-9].bz2 *.core"
  # /etc/periodic/daily/100.clean-disks
  Cleaning disks: 
  /var/log/messages.1.bz2
  /var/log/messages.2.bz2   /var/log/messages.3.bz2
  /var/log/messages.4.bz2

That's a bit better. It didnt delete /var/log/messages or /var/log/messages.1.bz2, which I like to keep on disk. Remember, always test your pattern matching before scheduling a deletion script. If you keep theverboseline atYES, the script will report the names of files it deletes.

daily_clean_tmps.The other cleaning scripts are quite straightforward to configure. Takedaily_clean_tmps, for example:

  # 110.clean-tmps 
  daily_clean_tmps_enable="NO"     # Delete stuff daily
  daily_clean_tmps_dirs="/tmp"     # Delete under here
  daily_clean_tmps_days="3"        # If not accessed for
  daily_clean_tmps_ignore=".X*-lock quota.user quota.group" # Don't delete
                        # these
  daily_clean_tmps_verbose="YES"  # Mention files deleted

This is a quick way to clean out any temporary directories. Again, you get to choose the locations of those directories. Here is a quick way to find out which directories named tmp are on your system:

  # find / -type d -name tmp
  / tmp   /usr/tmp
  /var/spool/cups/tmp
  /var/tmp

That command asksfindto start at root (/) and look for any directories (-type d) named tmp (-name tmp). If I wanted to clean those daily, I'd configure that section like so:

  # 110.clean-tmps
  # Delete stuff daily 
  daily_clean_tmps_enable="YES" 
  daily_clean_tmps_dirs="/tmp /usr/tmp / var/spool/cups/tmp /var/tmp"
  # If not accessed for 
  daily_clean_tmps_days="1"
  # Don't delete these
  daily_clean_tmps_ignore=".X*-lock quota.user quota.group"
  # Mention files deleted
  daily_clean_tmps_verbose="YES"

Again, I immediately test that script after saving my changes:

  # /etc/periodic/daily/110.
clean-tmps 
  Removing old temporary files:
   /var/tmp/gconfd-root

This script will not delete any locked files or temporary files currently in use. This is an excellent feature and yet another reason to run this script on a daily basis, preferably at a time when few users are on the system.

daily_clean_preserve.Moving on, the next script isdaily_clean_preserve:

  # 120.clean-preserve
  daily_clean_preserve_enable="YES"   #  
Delete files daily
  daily_clean_preserve_days=7         # If not modified for
  daily_clean_preserve_verbose="YES"  # Mention files deleted

What exactly ispreserve? The answer is inman hier. Use the manpage search function (the/key) to search for the wordpreserve:

  # man hier
  /preserve
         preserve/ temporary home of files preserved after an accidental
                    death of an editor; see (ex)1

Now that you know what the script does, see if the default settings are suited for your environment. This script is run daily, but keeps preserved files until they are seven days old.

The last three clean scripts deal with cleaning out old files frommsgs,rwho andsendmail's hoststat cache. Seeman periodic.conffor more details.

Incidentally, you don't have to wait until it is time forperiodicto do its thing; you can manually run any periodic script at any time. You'll find them all in subdirectories of /etc/periodic/.

Limiting Files

Instead of waiting for a daily process to clean up any spills, you can tweak several knobs to prevent these files from being created in the first place. For example, the C shell itself provides limits, any of which are excellent candidates for a customized dot.cshrc file [Hack #9].

To see the possible limits and their current values:

  % limit
  cputime        unlimited
  filesize       unlimited
  datasize       524288 kbytes 
  stacksize      65536 kbytes
  coredumpsize   unlimited
  memoryuse      unlimited
  vmemoryuse     unlimited 
  descriptors    4557
  memorylocked   unlimited
  maxproc        2278
  sbsize         unlimited

You can test a limit by typing it at the command line; it will remain for the duration of your current shell. If you like the limit, make it permanent by adding it to .cshrc. For example:

  % limit filesize 2k
  % limit | grep filesize
  filesize      2 kbytes

will set the maximum file size that can be created to 2 KB. Thelimitcommand supports bothkfor kilobytes andmfor megabytes. Do note that this limit does not affect the total size of the area available to store files, just the size of a newly created file. See the Quotas section of the FreeBSD Handbook if you intend to limit disk space usage.

Having created a file limit, you'll occasionally want to exceed it. For example, consider decompressing a file:

  % uncompress largefile.Z
  Filesize limit exceeded
  % unlimit filesize
  % uncompress largefile.Z
  % 

The unlimit command will allow me to override the file-size limit temporarily (for the duration of this shell). If you really do want to force your users to stick to limits, read man limits.

Now back to shell limits. If you don't know what a core file is, you probably don't need to collect them. Sure,periodiccan clean those files out for you, but why make them in the first place? Core files are large. You can limit their size with:

  limit coredumpsize 1m

That command will limit a core file to 1 MB, or 1024 KB. To prevent core files completely, set the size to0:

  limit coredumpsize 0

If you're interested in the rest of the built-in limits, you'll find them inman tcsh. Searching forcoredumpsizewill take you to the right spot.

The Other BSDs

The preceding discussion is based on FreeBSD. Other BSD systems ship with similar scripts that do identical tasks, but they are kept in a single file instead of in a separate directory.

NetBSD.For daily, weekly, and monthly tasks, NetBSD uses the /etc/daily,  /etc/weekly, and /etc/monthly scripts, whose behavior is controlled with the /etc/daily.conf,  /etc/weekly.conf, and /etc/monthly.conf configuration files. For more information about them, readman daily.conf,man weekly.conf, andman monthly.conf.

OpenBSD.OpenBSD uses three scripts,  /etc/daily,  /etc/weekly, and /etc/ monthly. You can learn more about them by readingman daily.

See Also

• man periodic.conf
  
• man limits
  
• man tcsh
  
• The Quotas section of the FreeBSD Handbook (http://www.freebsd.org/doc/ en_US.ISO8859-1/books/handbook/quotas.html)