Advanced Concepts on Dealing with Files and Filesystems in BSD

In this conclusion to a two-part article, we continue our discussion of BSD commands and filesystems. It is excerpted from chapter two of the book BSD Hacks, written by Dru Lavigne (O’Reilly, 2005; ISBN: 0596006799). Copyright © 2005 O’Reilly Media, Inc. All rights reserved. Used with permission from the publisher. Available from booksellers or direct from O’Reilly Media.

HACK#18: DOS Floppy Manipulation

Bring simplicity back to using floppies.

If youre like many Unix users, you originally came from a Windows background. Remember your initial shock the first time you tried to use a floppy on a Unix system? Didn’t Windows seem so much simpler? Forever gone seemed the days when you could simply insert a floppy, copy some files over, and remove the disk from the drive. Instead, you were expected to plunge into the intricacies of the mount command, only to discover that you didn’t even have the right to use the floppy drive in the first place!

There are several ways to make using floppies much, much easier on your FreeBSD system. Let’s start by taking stock of the default mechanisms for managing floppies.

Mounting a Floppy

Suppose I have formatted a floppy on a Windows system, copied some files over, and now want to transfer those files to my FreeBSD system. In reality, that floppy is a storage media. Since it is storing files, it needs a filesystem in order to keep track of the locations of those files. Because that floppy was formatted on a Windows system, it uses a filesystem called FAT12.

In Unix, a filesystem can’t be accessed until it has been mounted. This means you have to use the mount command before you can access the contents of that floppy. While this may seem strange at first, it actually gives Unix more flexibility. An administrator can mount and unmount filesystems as they are needed. Note that I used the word administrator. Regular users don’t have this ability, by default. Well change that shortly.

Unix also has the additional flexibility of being able to mount different filesystems. In Windows, a floppy will always contain the FAT12 filesystem. BSD understands floppies formatted with either FAT12 or UFS, the Unix File System. As you might expect from the name, the UFS filesystem is assumed unless you specify otherwise.

For now, become the superuser and let’s pick apart the default invocation of the mount command:

  % su
 
Password:
 
# mount -t msdos /dev/fd0 /mnt
 
#

I used the type (-t) switch to indicate that this floppy was formatted from an msdos-based system. I could have used the mount_msdosfs command instead:

  # mount_msdosfs /dev/fd0 /mnt

Both commands take two arguments. The first indicates the device to be mounted. /dev/fd0 represents the first (0) floppy drive (fd) device ( /dev ).

The second argument represents the mount point. A mount point is simply an empty directory that acts as a pointer to the mounted filesystem. Your FreeBSD system comes with a default mount point called /mnt. If you prefer, create a different mount point with a more useful name. Just remember to keep that directory empty so it will be available as a mount point, because any files in your mount point will become hidden and inaccessible when you mount a device over it.

This can be a feature in itself if you have a filesystem that should always be mounted. Place a README file in /mnt/important_directory containing: “If you can see this file, contact the administrator at this number….”

In this example, I’ll create a mount point called /floppy, which Ill use in the rest of the examples in this hack:

  # mkdir /floppy

Common Error Messages

This is a good place to explain some common error messages. Trust me, I experienced them all before I became proficient at this whole mount business. At the time, I wished for a listing of error messages so I could figure out what I had done wrong and how to fix it.

Let’s take a look at the output of this command:

  # mount /dev/fd0 /mnt
 
mount: /dev/fd0 on /mnt: incorrect super block

Remember my first mount command? I know it worked, as I just received my prompt back. I know this command didnt work, because mount instead wrote me a message explaining why it did not do what I asked.

That error message isn’t actually as bad as it sounds. I forgot to include the type switch, meaning mount assumed I was using UFS. Since this is a FAT12 floppy, it simply didn’t understand the filesystem.

This error message also looks particularly nasty:

  fd0: hard error cmd=read fsbn 0 of 0-3 (No status )
  msdosfs: /dev/fd0: Input/output error

If you get that one, quickly reach down and push in the floppy before any one else notices. You forgot to insert it into the bay.

Here’s another error message:

  msdosfs: /dev/fd0: Operation not permitted

Oops. Looks like I didn’t become the superuser before trying that moun t command .

How about this one:

  mount: /floppy: No such file or directory

Looks like I forgot to make that mount point first. A mkdir /floppy should fix that one.

The one error message you do not want to see is a system panic followed by a reboot. It took me a while to break myself of the habit of just ejecting a floppy once I had copied over the files I wanted. That’s something you just don’t do in Unix land.

You must first warn your operating system that you have finished using a filesystem before you physically remove it from the computer. Otherwise, when it goes out looking for a file, it will panic when it realizes that it has just disappeared off of the edge of the universe! (Well, the computer’s universe anyway.) Put yourself in your operating system’s shoes for a minute. The user entrusted something important to your care. You blinked for just a split second and it was gone, nowhere to be found. You’d panic too!

Managing the Floppy

How do you warn your operating system that the universe has shrunk? You unmount the floppy before you eject it from the floppy bay. Note that the actual command used is missing the first n and is instead spelled umount :

  # umount /floppy

Also, the only argument is the name of your mount point. In this example, it’s /floppy.

How can you tell if a floppy is mounted? The disk free command will tell you:

Filesystem

1K-blocks

Used

Avail

Capacity

Mounted on

/dev/ad0s1a

      257838

    69838

 167374

   29%

/

devfs

         1

        1

      0

  100%

/dev

/dev/ad0s1e

    257838

      616

 236596

    0%

/tmp

/dev/ad0s1f

 13360662

 2882504

9409306

  23%

/usr

/dev/ad0sld

    257838

    28368

 208844

   12%

/var

/dev/fe0

      1424

        1

   1423

    0%

/floppy

 # df Command

as will the mount command with no arguments:

  # mount
  /dev/ad0s1a on / (ufs, local)
 
devfs on /dev (devfs, local)
 
/dev/ad0s1e on /tmp (ufs, local, soft-updates)
 
/dev/ad0s1f on /usr (ufs, local, soft-updates)
 
/dev/ad0s1d on /var (ufs, local, soft-updates)
 
/dev/fd0 on /floppy (msdosfs, local)

This system currently has a floppy /dev/fd 0 mounted on /floppy, meaning youll need to issue the umount command before ejecting the floppy.

Several other filesystems are also mounted, yet I only used the mount command on my floppy drive. When did they get mounted and how? The answer is in /etc/fstab, which controls which filesystems to mount at boot time. Here’s my /etc/fstab; it’s pretty similar to the earlier output from df :

# Device

Mountpoint

FStype

Options

 Dump

  Pass#

/dev/ad0s1b

none

swap

sw

0

0

/dev/ad0s1a

/

ufs

rw

1

1

/dev/ad0s1e

/tmp

ufs

rw

2

2

/dev/ad0s1f

/usr

ufs

rw

2

2

/dev/ad0s1d

/var

ufs

rw

2

2

/dev/acd0

/cdrom

cd9660

ro,noauto

0

0

proc

/proc

procfs

rw

0

0

linproc

/compat/linux/proc

linprocfs

rw

0

0

# more /etc/fstab command

Each mountable filesystem has its own line in this file. Each has its own unique mount point and its filesystem type listed. See how the /cdrom mount point has the options ro, noauto instead of rw? The noauto tells your system not to mount your CD-ROM at bootup. That is a good thing–if there’s no CD in the bay at boot time, the kernel will either give an error message or pause for a few seconds, looking for that filesystem.

However, you can mount a data CD-ROM at any time by simply typing:

  # mount /cdrom

That command was shorter than the usual mount command for one reason: there was an entry for /cdrom in /etc/fstab. That means you can shorten the command to mount a floppy by creating a similar entry for /floppy. Simply add this line to /etc/fstab:

  /dev/fd0 /floppy msdos rw,noauto 0 0

Test your change by inserting a floppy and issuing this command:

  # mount /floppy

If you receive an error, check /etc/fstab for a typo and try again.

Allowing Regular Users to Mount Floppies

Now that the superuser can quickly mount floppies, let’s give regular users this ability. First, we have to change the default setting of the vfs.usermount variable:

  # sysctl vfs.usermount=1
 
vfs.usermount: 0 -> 1

By changing the default 0 to a 1 , we’ve just enabled users to mount virtual filesystems. However, don’t worry about your users running amok with this new freedom–the devices themselves are still owned by root. Check out the permissions on the floppy device:

  # ls -l /dev/fd0
  crw-r—– 1 root operator 9, 0 Nov 28 08:31 /dev/fd0

If you’d like any user to have the right to mount a floppy, change the per missions so everyone has read and write access:

  # chmod 666 /dev/fd0

Now, if you don’t want every user to have this right, you could create a group, add the desired users to that group, and assign that group permissions to /dev/fd0.

You’re almost there. The only kicker is that the user has to own the mount point. The best place to put a user’s mount point is in his home directory. So, logged in as your usual user account:

  % mkdir ~/floppy

Now, do you think the mount command will recognize that new mount point?

  % mount ~/floppy 
  mount: /home/dru/floppy: unknown special file or file system

Oh boy. Looks like were back to square one, doesn’t it? Remember, that entry in /etc/fsta b only refers to root’s mount point, so I can’t use that shortcut to refer to my own mount point. While it’s great to have the ability to use the mount command, Im truly too lazy to have to type out mount -t msdos /dev/fd0 ~/floppy , let alone remember it.

Thank goodness for aliases. Try adding these lines to the alias section of your ~.cshrc file:

  alias mf  mount -t msdos /dev/fd0 ~/floppy
   alias uf  umount ~/floppy

Now you simply need to type mf whenever you want to mount a floppy and uf when it’s time to unmount the floppy. Or perhaps you’ll prefer to create a keyboard shortcut [Hack #4].

Formatting Floppies

Now that you can mount and unmount floppies with the best of them, it’s time to learn how to format them. Again, let’s start with the default invoca tions required to format a floppy, then move on to some ways to simplify the process.

When you format a floppy on a Windows or DOS system, several events occur:

  1. The floppy is low-level formatted, marking the tracks and sectors onto the disk.
  2. A filesystem is installed onto the floppy, along with two copies of its FAT table.
  3. You are given the opportunity to give the floppy a volume label.

The same process also has to occur when you format a floppy on a FreeBSD system. On a 5.x system, the order goes like this:

  % fdformat -f 1440 /dev/fd0
  Format 1440K floppy `/dev/fd0’? (y/n): y
  Processing ——————————
  % bsdlabel -w /dev/fd0 fd1440
  % newfs_msdos /dev/fd0
  /dev/fd0: 2840 sectors in 355 FAT12 clusters (4096 bytes/cluster)
  bps=512 spc=8 res=1 nft=2 rde=512 sec=2880 mid=0xf0 spf=2 spt=18 hds=2 hid=0

First, notice that we don’t use the mount command. You can’t mount a filesys tem before you have a filesystem! (You do have to have the floppy in the drive, though.) Take a look at the three steps:

  1. fdformat does the low-level format.
  2. bsdlabel creates the volume label.
  3. newfs_msdos installs the FAT12 filesystem.

If I see the following error message when I try to mount the floppy, I’ll realize that I forgot that third step:

  % mf
 
msdosfs: /dev/fd0: Invalid argument

Because my mf mount floppy alias uses the msdos filesystem, it will complain if the floppy isn’t formatted with FAT12.

Automating the Format Process

Any three-step process is just begging to be put into a shell script. I like to keep these scripts under ~/bin. If you don’t have this directory yet, create it. Then create a script called ff (for format floppy):

  % cd
  % mkdir bin
  % cd bin
  % vi ff
  #!/bin/sh
  #this script formats a floppy with FAT12
  #that floppy can also be used on a Windows system
 
# first, remind the user to insert the floppy
  echo "Please insert the floppy and press enter"
  read pathname
 
# then, proceed with the three format steps
 
fdformat -f 1440 /dev/fd0
  bsdlabel -w /dev/fd0 fd1440 
  newfs_msdos /dev/fd0
  echo "Format complete."

Note that this script is basically those three commands, with comments thrown in so I remember what the script does. The only new part is the read pathname line. I added it to force the user to press Enter before the script proceeds.

Remember to make the script executable:

  % chmod +x ff

I’ll then return to my home directory and see how it works. Since I use the C shell, I’ll use the rehash command to make the shell aware that there is a new executable in my path:

rehash command to make the shell aware that there is a new executable in my path: Ill then return to my home directory and see how it works. Since I use the C shell, Ill use the rehash command to make the shell aware that there is a new executable in my path:

  % cd
  % rehash
  % ff
  Please insert the floppy and press enter
 
Format 1440K floppy `/dev/fd0’? (y/n): y 
  Processing ——————————  /dev/fd0: 2840 sectors in 355 FAT12 clusters (4096 bytes/cluster)
  bps=512 spc=8 res=1 nft=2 rde=512 sec=2880 mid=0xf0 spf=2 spt=18 hds=2 hid=0 
  Format complete.

Not too bad. I can now manipulate floppies with my own custom mf , uf , and ff commands.

See Also

  1. man fstab
  2. man fdformat
  3. man bsdlabel
  4. man newfs
  5. The Creating and Using Floppies section of the FreeBSD Handbook (http:/ / www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/floppies.html)
  6. The Mounting and Unmounting File Systems section of the FreeBSD Handbook (http://www.freebsd.org/doc/en_US. ISO8859-1/books/handbook/mount-unmount.html)

{mospagebreak title=HACK#19: Access Windows Shares Without a Server}

Share files between Windows and FreeBSD with a minimum of fuss.

You’ve probably heard of some of the Unix utilities available for accessing files residing on Microsoft systems. For example, FreeBSD provides the mount_smbfs and smbutil utilities to mount Windows shares and view or access resources on a Microsoft network. However, both of those utilities have a caveat: they require an SMB server. The assumption is that somewhere in your network there is at least one NT or 2000 Server.

Not all networks have the budget or the administrative expertise to allow for commercial server operating systems. Sure, you can install and configure Samba, but isn’t that overkill for, say, a home or very small office network? Sometimes you just want to share some files between a Windows 9x system and a Unix system. It’s a matter of using the right-sized tool for the job. You don’t bring in a backhoe to plant flowers in a window box.

Installing and Configuring Sharity-Light

If your small network contains a mix of Microsoft and Unix clients, consider installing Sharity-Light on the Unix systems. This application allows you to mount a Windows share from a Unix system. FreeBSD provides a port for this purpose (see the Sharity-Light web site for other supported platforms):

  # cd /usr/ports/net/sharity-light
  # 
make install clean

Since Sharity-Light is a command-line utility, you should be familiar with UNC or the Universal Naming Convention. UNC is how you refer to Microsoft shared resources from the command line. A UNC looks like \NetBIOSnamesharename . It starts with double backslashes, then contains the NetBIOS name of the computer to access and the name of the share on that computer.

Since Sharity-Light is a command-line utility, you should be familiar with UNC or the Universal Naming Convention. UNC is how you refer to Microsoft shared resources from the command line. A UNC looks like

Before using Sharity-Light, you need to know the NetBIOS names of the computers you wish to access. If you have multiple machines running Microsoft operating systems, the quickest way to view each systems name is with nbtstat . From one of the Windows systems, open a command prompt and type:

  C:> nbtstat -A 192.168.2.10
        
NETBIOS Remote Machine Name Table
   
Name         Type        Status 
  —————————————–  LITTLE_WOLF  <00> UNIQUE    Registere d
  <snip>

Repeat for each IP address in your network. Your output will be several lines long, but the entry (usually the first) containing <00> is the one with the name you’re interested in. In this example, LITTLE_WOLF is the NetBIOS name associated with 192.168.2.10.

Even though nbtstat ? indicates that -A is used to view a remot e system, it also works with the IP address of the local system. This allows you to check all of the IP addresses in your network from the same system.

Once you know which IP addresses are associated with which NetBIOS names, you’ll need to add that information to /etc/hosts on your Unix systems:

  # more /etc/hosts
  127.0.0.1        localhost
  192.168.2.95     genisis       #this system 
  192.168.2.10     little_wolf   #98 system sharing cygwin2

You’ll also need to know the names of the shares you wish to access. Again, from a Microsoft command prompt, repeat this command for each Net-BIOS name and make note of your results:

  C:> net view \little_wolf
  Shared resources at \LITTLE_WOLF
  Sharename     Type      Comment

  ————————————-
  CYGWIN2      Disk
  The command was completed successfully.

Here the computer known as LITTLE_WOLF has only one share, the CYGWIN2 directory.

Finally, you’ll need a mount point on your Unix system, so you might as well give it a useful name. Since the typical floppy mount point is /flopp y and the typical CD mount point is /cdrom, let’s use /windows:

  # mkdir /windows

Accessing Microsoft Shares

Once you know the names of your computers and shares, using Sharity-Light is very easy. As the superuser, mount the desired share:

  # shlight    //little_wolf/cygwin2 /windows   
  Password:
  Using port 49923 for NFS.

Watch your slashes. Microsoft uses the backslash ( )  at the command line, whereas Unix and Sharity-Light use the forward slash ( / ).

Note that I was prompted for a password because Windows 9x and ME users have the option of password protecting their shares. This particular share did not have a password, so I simply pressed Enter.

Adding -n to the previous command will forego the password prompt. Type shlight -h to see all available options.

However, if the share is on a Windows NT Workstation, 2000 Pro, or XP system, you must provide a username and password valid on that system. The syntax is:

  # shlight //2000pro/cdrom /windows -U username -P password

Once the share is mounted, it works like any other mount point. Depending on the permissions set on the share, you should be able to browse that shared directory, copy over or add files, and modify files. When you’re finished using the share, unmount it:

    $ unshlight /windows

See Also

  • The Sharity-Light README and FAQ (/usr/local/share/doc/Sharity-Light/)
  • The Sharity-Light web site
    (http://www.obdev.at/products/sharity-light/index.html)
  • The Samba web site (http://www.samba.org/)

{mospagebreak title=HACK#20: Deal with Disk Hogs} 

Fortunately, you no longer have to be a script guru or a find wizard just to keep up with what is happening on your disks.

Think for a moment. What types of files are you always chasing after so they don’t waste resources? Your list probably includes temp files, core files, and old logs that have already been archived. Did you know that your system already contains scripts capable of cleaning out those files? Yes, Im talking about your periodic scripts.

Periodic Scripts

Youll find these scripts in the following directory on a FreeBSD system:

  % ls /etc/periodic/daily | grep clean
  100.clean-disks
  110.clean-tmps
  120.clean-preserve
  130.clean-msgs
  140.clean-rwho
  150.clean-hoststat

Are you using these scripts? To find out, look at your /etc/periodic.con f file. What, you don’t have one? That means you’ve never tweaked your default configurations. If that’s the case, copy over the sample file and take a look at what’s available:

  # cp  /etc/defaults/periodic.conf
  /etc/periodic.conf
  # 
more /etc/periodic.conf

daily_clean_disks. Let’s start with daily_clean_disks. This script is ideal for finding and deleting files with certain file extensions. You’ll find it about two pages into periodic.conf, in the Daily options section, where you may note that it’s not enabled by default. Fortunately, configuring it is a heck of a lot easier than using cron to schedule a complex find statement.

Before you enable any script, test it first, especiall y if it’ll delete files based on pattern-matching rules. Back up your system first!

For example, suppose you want to delete old logs with the .bz2 extension. If you’re not careful when you craft your daily_ clean_disks_files line, you may end up inadvertently deleting all files with that extension. Any user who has just compressed some important data will be very miffed when she finds that her data has mysteriously disappeared.

Let’s test this scenario. I’d like to prune all .core files and any logs older than .0.bz2. I’ll edit that section of /etc/periodic.conf like so:

  # 100.clean-disks

daily_clean_disks_enable="YES"

# Delete files daily

daily_clean_disks_files="*.[1-9].bz2 *.core"

# delete old logs, cores

daily_clean_disks_days=1

# on a daily basis

daily_clean_disks_verbose="YES"

# Mention files deleted

Notice my pattern-matching expression for the .bz2 files. My expression matches any filename ( *) followed by a dot and a number from one to nine ( .[1-9] ), followed by another dot and the .bz2 extension.

Now I’ll verify that my system has been backed up, and then manually run that script. As this script is fairly resource-intensive, I’ll do this test when the system is under a light load:

  # /etc/periodic/daily/100.clean-disks 
  Cleaning disks:   /usr/ports/distfiles/MPlayer-0.92.tar.bz2
  /usr/ports/distfiles/gnome2/libxml2-2.6.2.tar.bz2
  /usr/ports/distfiles/gnome2/libxslt-1.1.0.tar.bz2

Darn. Looks like I inadvertently nuked some of my distfiles. I’d better be a bit more explicit in my matching pattern. I’ll try this instead:

  # delete old logs, cores  
  daily_clean_disks_files="messages.[1-9].bz2 *.core"

  #
/etc/periodic/daily/100.clean-disks
  Cleaning disks: 
  /var/log/messages.1.bz2
  /var/log/messages.2.bz2   /var/log/messages.3.bz2
  /var/log/messages.4.bz2

That’s a bit better. It didnt delete /var/log/message s or /var/log/messages.1.bz2, which I like to keep on disk. Remember, always test your pattern matching before scheduling a deletion script. If you keep the verbose line at YES , the script will report the names of files it deletes.

daily_clean_tmps. The other cleaning scripts are quite straightforward to configure. Take daily_clean_tmps , for example:

  # 110.clean-tmps 
  daily_clean_tmps_enable="NO"     # Delete stuff daily
  daily_clean_tmps_dirs="/tmp"     # Delete under here
  daily_clean_tmps_days="3"        # If not accessed for
  daily_clean_tmps_ignore=".X*-lock quota.user quota.group" # Don’t delete
                        
# these
  daily_clean_tmps_verbose="YES"  # Mention files deleted

This is a quick way to clean out any temporary directories. Again, you get to choose the locations of those directories. Here is a quick way to find out which directories named tmp are on your system:

  # find / -type d -name tmp
  / tmp   /usr/tmp
  /var/spool/cups/tmp
  /var/tmp

That command asks find to start at root ( / ) and look for any directories ( -type d ) named tm p ( -name tmp ). If I wanted to clean those daily, I’d configure that section like so:

  # 110.clean-tmps
 
# Delete stuff dail y 
  daily_clean_tmps_enable="YES" 
  daily_clean_tmps_dirs="/tmp /usr/tmp / var/spool/cups/tmp /var/tmp"
 
# If not accessed for 
  daily_clean_tmps_days="1"
 
# Don’t delete these
  daily_clean_tmps_ignore=".X*-lock quota.user quota.group"
 
# Mention files deleted
  daily_clean_tmps_verbose="YES"

Again, I immediately test that script after saving my changes:

  # /etc/periodic/daily/110.
clean-tmps
 
  Removing old temporary files:
 
 /var/tmp/gconfd-root

This script will not delete any locked files or temporary files currently in use. This is an excellent feature and yet another reason to run this script on a daily basis, preferably at a time when few users are on the system.

daily_clean_preserve. Moving on, the next script is daily_clean_preserve :

   # 120.clean-preserve
  daily_clean_preserve_enable="YES"   #  
Delete files daily
 
daily_clean_preserve_days=7         # If not modified for
 
daily_clean_preserve_verbose="YES"  # Mention files deleted

What exactly is preserve ? The answer is in man hier . Use the manpage search function (the / key) to search for the word preserve :

  # man hier
  /preserve
        
preserve/ temporary home of files preserved after an accidental
                    death of an editor; see (ex)1

Now that you know what the script does, see if the default settings are suited for your environment. This script is run daily, but keeps preserved files until they are seven days old.

The last three clean scripts deal with cleaning out old files from msgs , rwho and sendmail’ s hoststat cache. See man periodic.conf for more details.

Incidentally, you don’t have to wait until it is time for periodic to do its thing; you can manually run any periodic script at any time. You’ll find them all in subdirectories of /etc/periodic/.

Limiting Files

Instead of waiting for a daily process to clean up any spills, you can tweak several knobs to prevent these files from being created in the first place. For example, the C shell itself provides limits, any of which are excellent candi dates for a customized dot.cshrc file [Hack #9].

To see the possible limits and their current values:

  % limit
 
cputime        unlimited
  filesize       unlimited
  datasize       524288 kbytes 
  stacksize      65536 kbytes
  coredumpsize   unlimited
  memoryuse      unlimited
  vmemoryuse     unlimited 
  descriptors    4557
  memorylocked   unlimited
  maxproc        2278
  sbsize         unlimited

You can test a limit by typing it at the command line; it will remain for the duration of your current shell. If you like the limit, make it permanent by adding it to .cshrc. For example:

  % limit filesize 2k
  % limit | grep filesize
 
filesize      2 kbytes

will set the maximum file size that can be created to 2 KB. The limit com mand supports both k for kilobytes and m for megabytes. Do note that this limit does not affect the total size of the area available to store files, just the size of a newly created file. See the Quotas section of the FreeBSD Handbook if you intend to limit disk space usage.

Having created a file limit, you’ll occasionally want to exceed it. For example, consider decompressing a file:

  % uncompress largefile.Z
  Filesize limit exceeded
 
% unlimit filesize
  % uncompress largefile.Z
  %
 

The unlimit command will allow me to override the file-size limit temporarily (for the duration of this shell). If you really do want to force your users to stick to limits, read man limits.

Now back to shell limits. If you don’t know what a core file is, you probably don’t need to collect them. Sure, periodic can clean those files out for you, but why make them in the first place? Core files are large. You can limit their size with:

  limit coredumpsize 1m

That command will limit a core file to 1 MB, or 1024 KB. To prevent core files completely, set the size to 0 :

  limit coredumpsize 0

If you’re interested in the rest of the built-in limits, you’ll find them in man tcsh . Searching for coredumpsize will take you to the right spot.

The Other BSDs

The preceding discussion is based on FreeBSD. Other BSD systems ship with similar scripts that do identical tasks, but they are kept in a single file instead of in a separate directory.

NetBSD. For daily, weekly, and monthly tasks, NetBSD uses the /etc/daily,  /etc/weekly, and /etc/monthly scripts, whose behavior is controlled with the /etc/daily.conf,  /etc/weekly.conf, and /etc/monthly.conf configuration files. For more information about them, read man daily.conf , man weekly.conf , and man monthly.conf .

OpenBSD. OpenBSD uses three scripts,  /etc/daily,  /etc/weekly, and /etc/ monthly. You can learn more about them by reading man daily .

See Also

  • man periodic.conf
  • man limits
  • man tcsh
  • The Quotas section of the FreeBSD Handbook (http://www.freebsd.org/doc/ en_US.ISO8859-1/books/handbook/quotas.html)

{mospagebreak title=HACK#21: Manage Temporary Files and Swap Space}

Add more temporary or swap space without repartitioning.

When you install any operating system, it’s important to allocate sufficient disk space to hold temporary and swap files. Ideally, you already know the optimum sizes for your system so you can partition your disk accordingly during the install. However, if your needs change or you wish to optimize your initial choices, your solution doesn’t have to be as drastic as a repartitionand– reinstall–of the system.

man tuning has some practical advice for guesstimating the appropriate size of swap and your other partitions.

Clearing /tmp

Unless you specifically chose otherwise when you partitioned your disk, the installer created a /tmp filesystem for you:

  % grep tmp /etc/fstab
  /dev/ad0s1e   /tmp  ufs  rw  2  2 
  
% df -h /tmp

Filesystem

Size

Used

Avil

Capacity

Mounted on

/dev/ad0s1e

252M

614K

 231M

    0%

/tmp

Here I searched /etc/fsta b for the /tmp filesystem. This particular filesystem is 256 MB in size. Only a small portion contains temporary files.

 

The df (disk free) command will always show you a number lower than the actual partition size. This is because eight percent of the filesystem is reserved to prevent users from inadvertently overflowing a filesystem. See man tunefs for details.

It’s always a good idea to clean out /tmp periodically so it doesn’t overflow with temporary files. Consider taking advantage of the built-in periodic script /etc/periodic/daily/110.clean-tmps [Hack #20].

You can also clean out /tmp when the system reboots by adding this line to /etc/rc.conf:

  clear_tmp_enable="YES"

Moving /tmp to RAM

Another option is to move /tmp off of your hard disk and into RAM. This has the built-in advantage of automatically clearing the filesystem when you reboot, since the contents of RAM are volatile. It also offers a performance boost, since RAM access time is much faster than disk access time.

Before moving /tmp, ensure you have enough RAM to support your desired /tmp size. This command will show the amount of installed RAM:

  % dmesg | grep memory
  real memory  = 335462400 (319 MB)
  avail memory = 320864256 (306 MB)

Also check that your kernel configuration file contains device md (or mem ory disk). The GENERIC kernel does; if you’ve customized your kernel, double-check that you still have md support:

  % grep -w md /usr/src/sys/ i386/conf/CUSTOM
 
device         md     # Memory "disks"

Changing the /tm p line in /etc/fstab as follows will mount a 64 MB /tmp in RAM:

  md /tmp mfs rw,-s64m 2 0

Next, unmount /tmp (which is currently mounted on your hard drive) and remount it using the new entry in /etc/fstab:

  # umount /tmp
  # 
mount /tmp
 
# df -h /tmp

Filesystem

Size

Used

Avail

Capacity

Mounted on

/dev/md0

 63M

8.OK

  58M

    0%

/tmp

Notice that the filesystem is now md0 , the first memory disk, instead of ad0s1e , a partition on the first IDE hard drive.

Creating a Swap File on Disk

Swap is different than /tmp. It’s not a storage area for temporary files; instead, it is an area where the filesystem swap s data between RAM and disk. A sufficient swap size can greatly increase the performance of your filesystem. Also, if your system contains multiple drives, this swapping process will be much more efficient if each drive has its own swap partition.

The initial install created a swap filesystem for you:

  % grep swap /etc/fstab
  /dev/ad0s1b     none   swap   sw  0   0
  % swapinfo

Device

1K-blocks

Used

Avail

Capacity

Type

/dev/ad0s1b

639688

  68

639620

    0%

Interleaved

 

Note that the swapinfo command displays the size of your swap files. If you prefer to see that output in MB, try the swapctl command with the -lh flags (which make the listing more human):

  % swapctl -lh
 
Device:      1048576-blocks Used:   
  /dev/ad0s1b         624     0

To add a swap area, first determine which area of disk space to use. For example, you may want to place a 128 MB swapfile on /usr. You’ll first need to use dd to create this as a file full of null (or zero) bytes. Here I’ll create a 128 MB swapfile as /usr/swap0:

  # dd if=/dev/zero of=/usr/swap0 bs=1024k count=128
  128+0 records in
  128+0 records out
  134217728 bytes transferred in 4.405036 secs (30469156 bytes/sec)

Next, change the permissions on this file. Remember, you don’t want users storing data here; this file is for the filesystem:

  # chmod 600 /usr/swap0

Since this is really a file on an existing filesystem, you can’t mount your swapfile in /etc/fstab. However, you can tell the system to find it at boot time by adding this line to /etc/rc.conf:

  swapfile="/usr/swap0"

To start using the swapfile now without having to reboot the system, use mdconfig :

  # mdconfig -a -t vnode -f /usr/swap0 -u 1 && swapon /dev/md1

The -a flag attaches the memory disk. -t vnode marks that the type of swap is a file, not a filesystem. The -f flag sets the name of that file: /usr/swap0.

The unit number -u 1 must match the name of the memory disk /dev/md1. Since this system already has /tm p mounted on /dev/md0, I chose to mount swap on /dev/md1. && swapon tells the system to enable that swap device, but only if the mdconfig command succeeded.

swapctl should now show the new swap partition:

  % swapctl -lh
   
Device:        1048576-blocks   Used:
 
/dev/ad0s1b           624       0
 
/dev/md1              128       0

Monitoring Swap Changes

Whenever you make changes to swap or are considering increasing swap, use systat to monitor how your swapfiles are being used in real time:

  % systat -swap

The output will show the names of your swap areas and how much of each is currently in use. It will also include a visual indicating what percentage of swap contains data.

OpenBSD Differences

You can make this hack work on OpenBSD, as long as you remember that the RAM disk device is rd and its configuration tool is rdconfig . Read the relevant manpages, and you’ll be hacking away.

See Also

  • man tuning ( practical advice on /tmp and swap)
  • man md
  • man mdconfig
  • man swapinfo
  • man swapctl
  • man systat
  • The BSD Handbook entry on adding swap (http://www.freebsd.org/doc/en_ US.ISO8859-1/books/handbook/adding-swap-space.html)

{mospagebreak title=HACK#22: Recreate a Directory Structure Using mtree}

Prevent or recover from rm disasters.

Someday the unthinkable may happen. You’re doing some routine maintenance and are distracted by a phone call or perhaps another employee’s question. A moment later, you’re faced with the awful realization that your fingers typed either a rm * or a rm -R in the wrong place, and now a portion of your system has evaporated into nothingness.

Painful thought, isn’t it? Let’s pause for a moment to catch our breath and examine a few ways to prevent such a scenario from happening in the first place.

Close your eyes and think back to when you were a fresh-faced newbie and were introduced to the omnipotent rm command. Return to the time when you actually read man rm and first discovered the -i switch. "What a great idea," you thought, "to be prompted for confirmation before irretrievably deleting a file from disk." However, you soon discovered that this switch can be a royal PITA. Face it, it’s irritating to deal with the constant question of whether you’re sure you want to remove a file when you just issued the command to remove that file.

Necessary Interaction

Fortunately, there is a way to request confirmation only when youre about to do something as rash as rm * . Simply make a file called -i. Well, actually, it’s not quite that simple. Your shell will complain if you try this:

  % touch -i
  touch: illegal option — i
  usage: touch [-acfhm] [-r file] [-t [[CC]Y]MMDDhhmm[.SS]] file …

You see, to your shell, - i looks like the -i switch, which touch doesn’t have. That’s actually part of the magic. The reason why we want to make a file called -i in the first place is to fool your shell: when you type rm * , the shell will expand * into all of the files in the directory. One of those files will be named -i, and, voila, you’ve just given the interactive switch to rm .

So, how do we get past the shell to make this file? Use this command instead:

  % touch ./-i

The ./ acts as a sort of separator instruction to the shell. To the left of the ./ go any options to the command touch ; in this case, there are none. To the right of the ./ is the name of the file to touch in "this directory."

In order for this to be effective, you need to create a file called -i in every directory that you would like to protect from an inadvertent rm * .

An alternative method is to take advantage of the rmstar shell variable found in the tcsh shell. This method will always prompt for confirmation of a rm * , regardless of your current directory, as long as you always use tcsh . Since the default shell for the superuser is tcsh , add this line to /root/.cshrc:

  set rmstar

This is also a good line to add to /usr/share/skel/dot.cshrc [Hack #9].

If you want to take advantage of the protection immediately, force the shell to reread its configuration file:

  # source /root/.cshrc

Using mtree

Now you know how to protect yourself from rm * . Unfortunately, neither method will save you from a rm -R . If you do manage to blow away a portion of your directory structure, how do you fix the mess with a minimum of fuss, fanfare, and years of teasing from your coworkers? Sure, you can always restore from backup, but that means filling in a form in triplicate, carrying it with you as you walk to the other side of the building where backups are stored, and sheepishly handing it over to the clerk in charge of tape storage.

Fortunately for a hacker, there is always more than one way to skin a cat, or in this case, to save your skin. That directory structure had to be created in the first place, which means it can be recreated.

When you installed FreeBSD, it created a directory structure for you. The utility responsible for this feat is called mtree.

To see which directory structures were created with mtree:

  % ls /etc/mtree/ 
 

./

BSD.root.dist

BSD.x11-4.dist

../

BSD.sendmail.dist

BSD.x11.dist

BSD.include.dist

BSD.usr.dist

 

BSD.local.dist

BSD.var.dist

 

Each of these files is in ASCII text, meaning you can read, and more interestingly, edit their contents. If you’re a hacker, I know what you’re thinking. Yes, you can edit a file to remove the directories you don’t want and to add other directories that you do.

Let’s start with a simpler example. Say you’ve managed to blow away /var. To recreate it:

  # mtree -deU -f /etc/mtree/BSD.var.dist -p /var

where:

-d

Ignores everything except directory files.

-e

Doesn’t complain if there are extra files.

-U

Recreates the original ownerships and permissions.

-f /etc/mtree/BSD.var.dist

Specifies how to create the directory structure; this is an ASCII text file if you want to read up ahead of time on what exactly is going to happen.

-p /var

Specifies where to create the directory structure; if you don’t specify, it will be placed in the current directory.

When you run this command, the recreated files will be echoed to standard output so you can watch as they are created for you. A few seconds later, you can:

  % ls /var

./

crash/

heimdal/

preserve/

yp/

../

cron/

lib/

run

 

account/

db/

log/

rwho/

 

at/

empty/

mail/

spool/

 

backups/

games/

msgs/

 

 

That looks a lot better, but don’t breathe that sigh of relief quite yet. You still have to recreate all of your log files. Yes, /var/lo g is still glaringly empty. Remember, mtree creates a directory structure, not all of the files within that directory structure. If you have a directory structure containing thousands of files, you’re better off grabbing your backup tape.

There is hope for /var/log, though. Rather than racking your brain for the names of all of the missing log files, do this instead:

  % more /etc/newsyslog.conf
 
# configuration file for newsyslog
  # $FreeBSD: src/etc/newsyslog.conf,v 1.42 2002/09/21 12:07:35 markm Exp $
  #
  # Note: some sites will want to select more restrictive protections than the
  # defaults. In particular, it may be desirable to switch many of the 644
  # entries to 640 or 600. For example, some sites will consider the
  # contents of maillog, messages, and lpd-errs to be confidential. In the
  # future, these defaults may change to more conservative ones.
  #

# logfilename

[owner:group]

mode

count

size

when

[ZJB]

[/pid_file] [sig_num]

 

 

 

 

 

 

/var/log/cron

 

600

3

100

*

J

/var/log/amd.log

 

644

7

100

*

J

/var/log/auth.log

 

600

7

100

*

J

/var/log/kerberos.log

 

600

7

100

*

J

/var/log/lpd-errs

 

644

7

100

*

J

/var/log/xferlog

 

600

7

100

*

J

/var/log/maillog

 

640

7

*

@T00

J

/var/log/sendmail.st

 

640

10

*

168

B

/var/log/messages

 

644

5

100

*

J

/var/log/all.log

 

600

7

*

@T00

J

/var/log/slip.log

root:network

640

3

100

*

J

/var/log/ppp.log

root:network

640

3

100

*

J

/var/log/security

 

600

10

100

*

J

/var/log/wtmp

 

644

3

*

@01T05

B

/var/log/daily.log

 

640

7

*

@T00

J

/var/log/weekly.log

 

640

5

1

$W6D0

J

/var/log/monthly.log

 

640

12

*

$M1D0

J

/var/log/console.log

 

600

5

100

*

J

There you go, all of the default log names and their permissions. Simply touch the required files and adjust their permissions accordingly with chmod .

Customizing mtree

Let’s get a little fancier and hack the mtree hack. If you want to be able to create a homegrown directory structure, start by perusing the instructions in /usr/src/etc/mtree/README.

The one rule to keep in mind is don’t use tabs. Instead, use four spaces for indentation. Here is a simple example:

  % more MY.test.dist
 
#home grown test directory structure
  /set type=dir uname=test gname=test mode=0755
  .
    
test1
     ..
      
test2
           subdir2a
           ..
           subdir2b
             
..
              subsubdir2c mode=01777
              ..
              ..
    
..

Note that you can specify different permissions on different parts of the directory structure.

Next, I’ll apply this file to my current directory:

  # mtree -deU -f MY.test.dist

and check out the results:

  # ls -F
  test1/
  test2/
  # ls -F test1
  #
  # ls -F test2
  subdir2a/
  subdir2b/
  # ls -F test2/subdir2b
  subsubdir2c/

As you can see, mtree can be a real timesaver if you need to create custom directory structures when you do installations. Simply take a few moments to create a file containing the directory structure and its permissions. You’ll gain the added bonus of having a record of the required directory structure.

See Also

  • man mtree
  • The Linux mtree port (http://www.wie-auch-immer.de/mtree/)

{mospagebreak title=HACK#23: Ghosting Systems}

Use an open source alternative to commercial ghosting software.

Do you find yourself installing multiple systems, all containing the same operating system and applications? As an IT instructor, Im constantly installing systems for my next class or trying to fix the ramifications of a misconfiguration from a previous class.

As any system administrator can attest to, ghostin g or hard drive-cloning software can be a real godsend. Backups are one thing; they retain your data. However, an image is a true timesaver–it’s a copy of the operating system itself, along with any installed software and all of your configurations and customizations.

I haven’t always had the luxury of a commercial ghosting utility at hand. As you can well imagine, I’ve tried every homegrown and open source ghosting solution available. I started with various invocations of dd , gzip , ssh , and dump , but kept running across the same fundamental problem: it was easy enough to create an image, but inconvenient to deploy that image to a fresh hard drive. It was doable in the labs that used removable drives, but, otherwise, I had to open up a system, cable in the drive to be deployed, copy the image, and recable the drive into its own system.

Forget the wear and tear on the equipment; that solution wasn’t working out to be much of a timesaver! What I really needed was a floppy that contained enough intelligence to go out on the network and retrieve and restore an image. I tried several open source applications and found that Ghost For Unix, g4u , best fit the bill.

Creating the Ghost Disk

You’re about two minutes away from creating a bootable g4u floppy. Simply download g4u-1.12fs from http://theatomicmoose.ca/g4u/  and copy it to a floppy:

  # cat g4u-1.12fs > /dev/fd0

Your only other requirement is a system with a drive capable of holding your images. It can be any operating system, as long as it has an installed FTP server. If it’s a FreeBSD system, you can configure an FTP server through /stand/sysinstall. Choose Configure from the menu, then Networking. Use your spacebar to choose Anon FTP.

Choose Yes to the configuration message and accept the defaults by tabbing to OK . The welcome message is optional. Exit sysinstall once you’re finished.

You’ll then need to remove the remark ( # ) in front of the FTP line in /etc/ inetd.conf, so it looks like this:

  ftp  stream  tcp  nowait root /usr/libexec/ftpd   ftpd -l

If inetd is already running, inform it of the configuration change using killall -1 inetd . Otherwise, start inetd by simply typing inetd . To ensure the service is running:

  # sockstat | grep 21
  root inetd 22433 4 tcp4 *:21  *:*

In this listing, the local system is listening for requests on port 21, and there aren’t any current connections listed in the remote address section ( *:* ).

g4u requires a username and a password before it will create or retrieve an image. The default account is install , but you can specify another user account when you use g4u . To create the install account on a FreeBSD FTP server:

  # pw useradd install -m -s /bin/csh

Make sure that the shell you give this user is listed in /etc/shells or FTP authentication will fail.

Then, use passwd install to give this account a password you will remember.

Creating an Image

Before you create an image, fully configure a test system. For example, in my security lab, I usually install the latest release of FreeBSD, add my customized /etc/motd and shell prompt, configure X, and install and configure the applications students will use during their labs.

It’s a good idea to know ahead of time how large the hard drive is on the test system and how it has been partitioned. There are several ways to find out on a FreeBSD system, depending upon how good you are at math. One way is to go back into /stand/sysinstall and choose Configure then Fdisk . The first long line will give the size of the entire hard drive:

  Disk name:      ad0
  DISK Geometry:  19885 cyls/16 heads/63 sectors = 20044080 sectors (9787MB)

Press q to exit this screen. If you then type fdisk at the command line, you’ll see the size of your partitions:

  # fdisk
 
<snip>
  The data for partition 1 is:
  sysid 165 (0xa5), (FreeBSD/NetBSD/386BSD)
      start 63, size 4095441 (1999 Meg), flag 80 (active)
  <snip>
  The data for partition 2 is:
  <UNUSED>
  The data for partition 3 is:
  <UNUSED>
  The data for partition 4 is:
  <UNUSED>

This particular system has a 9787 MB hard drive that has one 1999 MB par tition containing FreeBSD.

Whenever you’re using any ghosting utility, create an image using the smallest hard drive size that you have available, but which is also large enough to hold your desired data. This will reduce the size of the image and prevent the problems associated with trying to restore an image to a smaller hard drive.

Once you’re satisfied with your system, insert the floppy and reboot.

g4u will probe for hardware and configure the NIC using DHCP. Once it’s finished, you’ll be presented with this screen:

  Welcome to g4u Harddisk Image Cloning V1.12! 
  * To upload disk-image to FTP, type: uploaddisk serverIP [image] [disk]
  * To upload partition to FTP, type: uploadpart serverIP [image ]
  [disk+part] 
  * To install harddisk from FTP, type: slurpdisk serverIP [image] [disk]
 
* To install partition from FTP, type: slurppart serverIP [image]
  [disk+part]
 
* To copy disks locally, type: copydisk disk0 disk1
 
[disk] defaults to wd0 for first IDE disk, [disk+part] defaults to wd0d
  for the whole first IDE disk. Use wd1 for second IDE disk, sd0 for first
  SCSI disk, etc. Default image for slurpdisk is ‘rwd0d.gz’. Run ‘dmesg’ to
  see boot messages, ‘disks’ for recognized disks, ‘parts <disk>’ for list
  of (BSD-type!) partitions on disk ‘<disk>" (wd0, …), run any other
  commands without args to see usage message.

Creating the image is as simple as invoking uploaddisk with the IP address of the FTP server. If you wish, include a useful name for the image; in this example, I’ll call the image securitylab.gz:

  # uploaddisk 192.168.2.95 securitylab.gz
  ( cat $tmpfile ; dd progress=1 if=/dev/rwd0d bs=1m | gzip -9 ) | ftp -n
  tmpfile:
  open 192.168.2.95
  user install
  bin
  put – securitylab.gz
  bye
  5
  4
  3
  2
  1
  working…
  Connected to 192.168.2.95.
  220 genisis FTP server (Version 6.00LS) ready.
  331 Password required for install.
  Password: type_password_here
 
230 User install logged in.
  Remote system type is UNIX.
  Using binary mode to transfer files.
  200 Type set to I.
  remote: securitylab.gz
  227 Entering Passive Mode (192,168,2,95,192,1)
  150 Opening BINARY mode data connection for ‘securitylab.gz’.
  ……………….

This will take a while. How long depends upon the size of the drive and the speed of your network. When it is finished, you’ll see a summary:

  9787+1 records i n
  9787+1 records out
  10262568960 bytes transferred in 6033.533 secs (1700921 bytes/sec)
  226 Transfer complete.
  3936397936 bytes sent in 1:40:29 (637.58 KB/s)
  221 Goodbye.
  #

You can also check out the size of the image on the FTP server:

  % du -h ~install/securitylab.gz 
  3.7G /home/install/securitylab.gz

That’s not too bad. It took just over an hour and a half to compress that 9 GB drive to a 3.7 GB image. The g4u web site also has some hints for further reducing the size of the image or increasing the speed of the transfer.

If you use images on a regular basis, consider upgrading hubs or older switches to 100 MB switches. This can speed up your transfer rates significantly.

It’s also possible to create an image of each particular filesystem, but I find it easier just to image a fairly small drive. This is because an image of the entire drive includes the master boot record (MBR) or the desired partitioning scheme.

Deploying the Image

When you wish to install the image, use the floppy to boot the system to receive the image. Once you receive the prompt, specify the name of the image and the IP address of the FTP server:

  # slurpdisk 192.168.2.95 securitylab.gz

It doesn’t matter what was previously on that drive. Since the MBR is recreated, the new drive will just contain the imaged data. Once the deployment is finished, simply reboot the system without the floppy.

If the new drive is bigger than the image, you’ll have free space left over on the drive that you can partition with a partitioning utility. Remember, don’t try to deploy an image to a smaller drive!

See Also

  • The Ghost For Unix web site (http://www.feyrer.de/g4u/)
[gp-comments width="770" linklove="off" ]

chat