Administration
  Home arrow Administration arrow Page 3 - Advanced Concepts on Dealing with Files and Filesystems in BSD
Dev Shed Forums  
Administration  
AJAX  
Apache  
BrainDump  
DHTML  
Flash  
Java  
JavaScript  
Multimedia  
MySQL  
Oracle  
Perl  
PHP  
Practices  
Python  
Reviews  
Security  
Smartphone Development  
Style-Sheets  
Web Services  
XML  
Zend  
Zope  
Mobile Linux  
App Generation ROI  
IBM® developerWorks  
Forums Sitemap  
E-Commerce Hosting  
Linux Web Hosting  
Managed Hosting  
Small Business Hosting  
VPS Hosting  
Weekly Newsletter

 
Developer Updates  
Free Website Content 
 RSS  Articles
 RSS  Forums
 RSS  All Feeds
Write For Us Get Paid  
Request Media Kit
Contact Us  
Site Map  
Privacy Policy  
Support  
 USERNAME
 
 PASSWORD
 
 
  >>> SIGN UP!  
  Lost Password? 
Google.com  
ADMINISTRATION

Advanced Concepts on Dealing with Files and Filesystems in BSD
By: O'Reilly Media
  • Search For More Articles!
  • Disclaimer
  • Author Terms
  • Rating: starstarstarstarstar / 7
    2007-01-04


    Table of Contents:
  • Advanced Concepts on Dealing with Files and Filesystems in BSD
  • HACK#19: Access Windows Shares Without a Server
  • HACK#20: Deal with Disk Hogs
  • HACK#21: Manage Temporary Files and Swap Space
  • HACK#22: Recreate a Directory Structure Using mtree
  • HACK#23: Ghosting Systems

  • Rate this Article: Poor Best 
      ADD THIS ARTICLE TO:
      error-file:tidyout.log Del.ici.ous error-file:tidyout.log Digg
      error-file:tidyout.log Blink error-file:tidyout.log Simpy
      error-file:tidyout.log Google error-file:tidyout.log Spurl
      error-file:tidyout.log Y! MyWeb error-file:tidyout.log Furl
    Email Me Similar Content When Posted
    Add Developer Shed Article Feed To Your Site
    Email Article To Friend
    Print Version Of Article
    PDF Version Of Article

     
     
    ADVERTISEMENT


    Advanced Concepts on Dealing with Files and Filesystems in BSD - HACK#20: Deal with Disk Hogs
    ( Page 3 of 6 )

     

    Fortunately, you no longer have to be a script guru or a find wizard just to keep up with what is happening on your disks.

    Think for a moment. What types of files are you always chasing after so they don't waste resources? Your list probably includes temp files, core files, and old logs that have already been archived. Did you know that your system already contains scripts capable of cleaning out those files? Yes, Im talking about your periodic scripts.

    Periodic Scripts

    Youll find these scripts in the following directory on a FreeBSD system:

      % ls /etc/periodic/daily | grep clean
      100.clean-disks
      110.clean-tmps
      120.clean-preserve
      130.clean-msgs
      140.clean-rwho
      150.clean-hoststat

    Are you using these scripts? To find out, look at your /etc/periodic.con f file. What, you don't have one? That means you've never tweaked your default configurations. If that's the case, copy over the sample file and take a look at what's available:

      # cp  /etc/defaults/periodic.conf
      /etc/periodic.conf
      # 
    more /etc/periodic.conf

    daily_clean_disks. Let's start with daily_clean_disks. This script is ideal for finding and deleting files with certain file extensions. You'll find it about two pages into periodic.conf, in the Daily options section, where you may note that it's not enabled by default. Fortunately, configuring it is a heck of a lot easier than using cron to schedule a complex find statement.

    Before you enable any script, test it first, especiall y if it'll delete files based on pattern-matching rules. Back up your system first!

    For example, suppose you want to delete old logs with the .bz2 extension. If you're not careful when you craft your daily_ clean_disks_files line, you may end up inadvertently deleting all files with that extension. Any user who has just compressed some important data will be very miffed when she finds that her data has mysteriously disappeared.

    Let's test this scenario. I'd like to prune all .core files and any logs older than .0.bz2. I'll edit that section of /etc/periodic.conf like so:

      # 100.clean-disks

    daily_clean_disks_enable="YES"

    # Delete files daily

    daily_clean_disks_files="*.[1-9].bz2 *.core"

    # delete old logs, cores

    daily_clean_disks_days=1

    # on a daily basis

    daily_clean_disks_verbose="YES"

    # Mention files deleted

    Notice my pattern-matching expression for the .bz2 files. My expression matches any filename ( *) followed by a dot and a number from one to nine ( .[1-9] ), followed by another dot and the .bz2 extension.

    Now I'll verify that my system has been backed up, and then manually run that script. As this script is fairly resource-intensive, I'll do this test when the system is under a light load:

      # /etc/periodic/daily/100.clean-disks 
      Cleaning disks:   /usr/ports/distfiles/MPlayer-0.92.tar.bz2
      /usr/ports/distfiles/gnome2/libxml2-2.6.2.tar.bz2
      /usr/ports/distfiles/gnome2/libxslt-1.1.0.tar.bz2

    Darn. Looks like I inadvertently nuked some of my distfiles. I'd better be a bit more explicit in my matching pattern. I'll try this instead:

      # delete old logs, cores  
      daily_clean_disks_files="messages.[1-9].bz2 *.core"

      #
    /etc/periodic/daily/100.clean-disks
      Cleaning disks: 
      /var/log/messages.1.bz2
      /var/log/messages.2.bz2   /var/log/messages.3.bz2
      /var/log/messages.4.bz2

    That's a bit better. It didnt delete /var/log/message s or /var/log/messages.1.bz2, which I like to keep on disk. Remember, always test your pattern matching before scheduling a deletion script. If you keep the verbose line at YES , the script will report the names of files it deletes.

    daily_clean_tmps. The other cleaning scripts are quite straightforward to configure. Take daily_clean_tmps , for example:

      # 110.clean-tmps 
      daily_clean_tmps_enable="NO"     # Delete stuff daily
      daily_clean_tmps_dirs="/tmp"     # Delete under here
      daily_clean_tmps_days="3"        # If not accessed for
      daily_clean_tmps_ignore=".X*-lock quota.user quota.group" # Don't delete
                            
    # these
      daily_clean_tmps_verbose="YES"  # Mention files deleted

    This is a quick way to clean out any temporary directories. Again, you get to choose the locations of those directories. Here is a quick way to find out which directories named tmp are on your system:

      # find / -type d -name tmp
      / tmp   /usr/tmp
      /var/spool/cups/tmp
      /var/tmp

    That command asks find to start at root ( / ) and look for any directories ( -type d ) named tm p ( -name tmp ). If I wanted to clean those daily, I'd configure that section like so:

      # 110.clean-tmps
     
    # Delete stuff dail y 
      daily_clean_tmps_enable="YES" 
      daily_clean_tmps_dirs="/tmp /usr/tmp / var/spool/cups/tmp /var/tmp"
     
    # If not accessed for 
      daily_clean_tmps_days="1"
     
    # Don't delete these
      daily_clean_tmps_ignore=".X*-lock quota.user quota.group"
     
    # Mention files deleted
      daily_clean_tmps_verbose="YES"

    Again, I immediately test that script after saving my changes:

      # /etc/periodic/daily/110.
    clean-tmps
     
      Removing old temporary files:
     
     /var/tmp/gconfd-root

    This script will not delete any locked files or temporary files currently in use. This is an excellent feature and yet another reason to run this script on a daily basis, preferably at a time when few users are on the system.

    daily_clean_preserve. Moving on, the next script is daily_clean_preserve :

       # 120.clean-preserve
      daily_clean_preserve_enable="YES"   #  
    Delete files daily
     
    daily_clean_preserve_days=7         # If not modified for
     
    daily_clean_preserve_verbose="YES"  # Mention files deleted

    What exactly is preserve ? The answer is in man hier . Use the manpage search function (the / key) to search for the word preserve :

      # man hier
      /preserve
            
    preserve/ temporary home of files preserved after an accidental
                        death of an editor; see (ex)1

    Now that you know what the script does, see if the default settings are suited for your environment. This script is run daily, but keeps preserved files until they are seven days old.

    The last three clean scripts deal with cleaning out old files from msgs , rwho and sendmail' s hoststat cache. See man periodic.conf for more details.

    Incidentally, you don't have to wait until it is time for periodic to do its thing; you can manually run any periodic script at any time. You'll find them all in subdirectories of /etc/periodic/.

    Limiting Files

    Instead of waiting for a daily process to clean up any spills, you can tweak several knobs to prevent these files from being created in the first place. For example, the C shell itself provides limits, any of which are excellent candi dates for a customized dot.cshrc file [Hack #9].

    To see the possible limits and their current values:

      % limit
     
    cputime        unlimited
      filesize       unlimited
      datasize       524288 kbytes 
      stacksize      65536 kbytes
      coredumpsize   unlimited
      memoryuse      unlimited
      vmemoryuse     unlimited 
      descriptors    4557
      memorylocked   unlimited
      maxproc        2278
      sbsize         unlimited

    You can test a limit by typing it at the command line; it will remain for the duration of your current shell. If you like the limit, make it permanent by adding it to .cshrc. For example:

      % limit filesize 2k
      % limit | grep filesize
     
    filesize      2 kbytes

    will set the maximum file size that can be created to 2 KB. The limit com mand supports both k for kilobytes and m for megabytes. Do note that this limit does not affect the total size of the area available to store files, just the size of a newly created file. See the Quotas section of the FreeBSD Handbook if you intend to limit disk space usage.

    Having created a file limit, you'll occasionally want to exceed it. For example, consider decompressing a file:

      % uncompress largefile.Z
      Filesize limit exceeded
     
    % unlimit filesize
      % uncompress largefile.Z
      %
     

    The unlimit command will allow me to override the file-size limit temporarily (for the duration of this shell). If you really do want to force your users to stick to limits, read man limits.

    Now back to shell limits. If you don't know what a core file is, you probably don't need to collect them. Sure, periodic can clean those files out for you, but why make them in the first place? Core files are large. You can limit their size with:

      limit coredumpsize 1m

    That command will limit a core file to 1 MB, or 1024 KB. To prevent core files completely, set the size to 0 :

      limit coredumpsize 0

    If you're interested in the rest of the built-in limits, you'll find them in man tcsh . Searching for coredumpsize will take you to the right spot.

    The Other BSDs

    The preceding discussion is based on FreeBSD. Other BSD systems ship with similar scripts that do identical tasks, but they are kept in a single file instead of in a separate directory.

    NetBSD. For daily, weekly, and monthly tasks, NetBSD uses the /etc/daily,  /etc/weekly, and /etc/monthly scripts, whose behavior is controlled with the /etc/daily.conf,  /etc/weekly.conf, and /etc/monthly.conf configuration files. For more information about them, read man daily.conf , man weekly.conf , and man monthly.conf .

    OpenBSD. OpenBSD uses three scripts,  /etc/daily,  /etc/weekly, and /etc/ monthly. You can learn more about them by reading man daily .

    See Also

    • man periodic.conf
    • man limits
    • man tcsh
    • The Quotas section of the FreeBSD Handbook (http://www.freebsd.org/doc/ en_US.ISO8859-1/books/handbook/quotas.html)



     
     
    >>> More Administration Articles          >>> More By O'Reilly Media
     

       

    ADMINISTRATION ARTICLES

    - Network Booting via PXE: the Basics
    - Scalix: Linux Administrator`s Guide
    - Network Administration with FreeBSD 7
    - Components of an Information Architecture
    - The Anatomy of an Information Architecture
    - Configuring Load-Balanced Clusters
    - Load-Balanced Clusters
    - UNIX Time Format Demystified
    - Making Changes in the CVS
    - Building Your First CVS Repository
    - CVS Quickstart Guide
    - Authorizing Users in Samba
    - Handling User Accounts in Samba
    - Authentication in Samba
    - Accounts, Authentication, and Authorization





    © 2003-2009 by Developer Shed. All rights reserved. DS Cluster 2 Hosted by Hostway
    For more Enterprise Application Development news, visit eWeek