Home arrow BrainDump arrow Page 5 - More Amazing Things to Do With Pipelines

5.6 Summary - BrainDump

In this second part of a two-part series on pipelines in Unix, you will learn some fun ways to cheat at word puzzles and other, more useful tricks. This article is excerpted from chapter 5 of Classic Shell Scripting, written by Arnold Robbins and Nelson H.F. Beebe (O'Reilly; ISBN: 0596005954). Copyright © 2007 O'Reilly Media, Inc. All rights reserved. Used with permission from the publisher. Available from booksellers or direct from O'Reilly Media.

  1. More Amazing Things to Do With Pipelines
  2. 5.4 Word Lists
  3. Word Lists, continued
  4. 5.5 Tag Lists
  5. 5.6 Summary
By: O'Reilly Media
Rating: starstarstarstarstar / 5
July 02, 2008

print this article



This chapter has shown how to solve several text processing problems, none of which would be simple to do in most programming languages. The critical lessons of this chapter are:

  1. Data markup is extremely valuable, although it need not be complex. A unique single character, such as a tab, colon, or comma, often suffices.
  2. Pipelines of simple Unix tools and short, often inline, programs in a suitable text processing language, such as awk, can exploit data markup to pass multiple pieces of data through a series of processing stages, emerging with a useful report.
  3. By keeping the data markup simple, the output of our tools can readily become input to new tools, as shown by our little analysis of the output of the word-frequency filter, wf, applied to Shakespeare’s texts.
  4. By preserving some minimal markup in the output, we can later come back and massage that data further, as we did to turn a simple ASCII office directory into a web page. Indeed, it is wise never to consider any form of electronic data as final: there is a growing demand in some quarters for page-description languages, such as PCL, PDF, and PostScript, to preserve the original markup that led to the page formatting. Word processor documents currently are almost devoid of useful logical markup, but that may change in the future. At the time of this writing, one prominent word processor vendor was reported to be considering an XML representation for document storage. The GNU Project’s gnumeric spreadsheet, the Linux Documentation Project,* and the OpenOffice.org† office suite already do that. 
  5. Lines with delimiter-separated fields are a convenient format for exchanging data with more complex software, such as spreadsheets and databases. Although such systems usually offer some sort of report-generation feature, it is often easier to extract the data as a stream of lines of fields, and then to apply filters written in suitable programming languages to manipulate the data further. For example, catalog and directory publishing are often best done this way.



* On some systems, file formats are in Section 7; thus, you might need to use man 7 passwd instead.

* In addition to this book (listed in the Bibliography), hundreds of books on SGML and derivatives are listed at
http://www.math.utah.edu/pub/tex/bib/sgml.html and http://www.math.utah.edu/pub/tex/bib/sgml2000.html.

a E. F. Codd, A Relational Model of Data for Large Shared Data Banks, Communications of the ACM, 13(6) 377–387, June (1970), and Relational Database: A Practical Foundation for Productivity, Communications of the ACM, 25(2) 109–117, February (1982) (Turing Award lecture).

b By Kevin Kline and Daniel Kline, O’Reilly & Associates, 2000, ISBN 1-56592-744-3. See also
sqlbooks.html for an extensive list of SQL books.

* Available at http://www.math.utah.edu/pub/sgml/.

* Available at ftp://ftp.ox.ac.uk/pub/wordlists/, ftp://qiclab.scn.rain.com/pub/wordlists/, ftp://ibiblio.org/pub/
and http://www.phreak.org/html/wordlists.shtml. A search for “word list” in any Internet search engine turns up many more.

* Programming Pearls: A Literate Program: A WEB program for common words, Comm. ACM 29(6), 471–483, June (1986), and Programming Pearls: Literate Programming: Printing Common Words, 30(7), 594–599, July (1987). Knuth’s paper is also reprinted in his book Literate Programming, Stanford University Center for the Study of Language and Information, 1992, ISBN 0-937073-80-6 (paper) and 0-937073-81-4 (cloth).

* Programming Pearls: Associative Arrays, Comm. ACM 28(6), 570–576, June, (1985). This is an excellent introduction to the power of associative arrays (tables indexed by strings, rather than integers), a common
feature of most scripting languages.

† Available in the wonderful Project Gutenberg archives at

* Indeed, the only word related to the root of computer that Shakespeare used is computation, just once in each of two plays, Comedy of Errors and King Richard III. “Arithmetic” occurs six times in his plays, “calculate” twice, and “mathematics” thrice.

* See http://www.tldp.org/.

† See http://www.openoffice.org/.

>>> More BrainDump Articles          >>> More By O'Reilly Media

blog comments powered by Disqus
escort Bursa Bursa escort Antalya eskort


- Apple Founder Steve Jobs Dies
- Steve Jobs` Era at Apple Ends
- Google's Chrome Developer Tool Updated
- Google's Chrome 6 Browser Brings Speed to th...
- New Open Source Update Fedora 13 is Released...
- Install Linux with Knoppix
- iPad Developers Flock To SDK 3.2
- Managing a Linux Wireless Access Point
- Maintaining a Linux Wireless Access Point
- Securing a Linux Wireless Access Point
- Configuring a Linux Wireless Access Point
- Building a Linux Wireless Access Point
- Migrating Oracle to PostgreSQL with Enterpri...
- Demystifying SELinux on Kernel 2.6
- Yahoo and Microsoft Create Ad Partnership

Developer Shed Affiliates


Dev Shed Tutorial Topics: