Beginning Perl - Working with RegExps (Page 5 of 6 )
Now that we've matched a string, what do we do with it? Well, sometimes it's just useful to know whether a string contains a given pattern or not. However, a lot of the time we're going to be doing search-and-replace operations on text. We'll explain how to do that here. We'll also cover some of the more advanced areas of dealing with regular expressions.
Substitution Now we know all about matching text, substitution is very easy. Why? Because all of the clever things are in the 'search' part, rather than the 'replace': all the character classes, quantifiers and so on only make sense when matching. You can't substitute, say, a word with any number of digits. So, all we need to do is take the 'old' text, Our match, and tell perl what we want to replace it with. This we do with the s/// operator.
The s is for 'substitute' - between the first two slashes, we put our regular expression as before. Before the final slash, we put our text replacement. Just as with matching, we can use the =~ operator to apply it to a certain string. If this is not given, it applies to the default variable $_ :
#!/usr/bin/perl # subst1.plx use warnings; use strict; $_ = "Awake! Awake! Fear, Fire, Foes! Awake! Fire, Foes! Awake!"; # Tolkien, Lord of the Rings s/Foes/Flee/; print $_,"\n";
Here we have substituted the first occurrence of 'Foes' with the word 'Flee'. Had we wanted to change every occurrence, we would have needed to use another modifier. Just as the /i modifier for matching case-insensitively, the /g modifier on a substitution acts globally:
#!/usr/bin/perl# subst1.plxuse warnings;use strict; $_ = "Awake! Awake! Fear, Fire, Foes! Awake! Fire, Foes! Awake!";# Tolkien, Lord of the Rings
s/Foes/Flee/g;
print $_,"\n";
> perl subst1.plx Awake! Awake! Fear, Fire, Flee! Awake! Fire, Flee! Awake! > Like the left-hand side of the substitution, the right-hand side also works like a double-quoted string and is thus subject to variable interpolation. One useful thing, though, is that we can use the backreference variables we collected during the match on the right hand side. So, for instance, to swap the first two words in a string, we would say something like this:
#!/usr/bin/perl # subst2.plx use warnings; use strict; $_ = "there are two major products that come out of Berkeley: LSD and UNIX"; # Jeremy Anderson s/(\w+)\s+(\w+)/$2 $1/; print $_, "?\n";
>perl subst2.plx are there two major products that come out of Berkeley: LSD and UNIX? >
What would happen if we tried doing that globally? Well, let's do it and see:
#!/usr/bin/perl# subst2.plxuse warnings;use strict; $_ = "there are two major products that come out of Berkeley: LSD and UNIX";# Jeremy Anderson
s/(\w+)\s+(\w+)/$2 $1/g;
print $_, "?\n";
>perl subst2.plx are there major two that products out come Berkeley of: and LSD UNIX? >
Here, every word in a pair is swapped with its neighbor. When processing a global match, perl always starts where the previous match left off.
Changing Delimiters You may have noticed that // and s/// looks like q// and qq// . Well, just like q// and qq// , we can change the delimiters when matching and substituting to increase the readability of our regular expressions. The same rules apply: Any non-word character can be the delimiter, and paired delimiters such as <> , () , {}, and [] may be used - with two provisos.
First, if you change the delimiters on // , you must put an m in front of it. (m for 'match'). This is so that perl can still recognize it as a regular expression, rather than a block or comment or anything else. Second, if you use paired delimiters with substitution, you must use two pairs:
s/old text/new text/g;
becomes:
s{old text}{new text}g;
You may, however, leave spaces or new lines between the pairs for the sake of clarity:
s{old text} {new text}g;
The prime example of when you would want to do this is when you are dealing with file paths, which contain a lot of slashes. If you are, for instance, moving files on your Unix system from /usr/local/share/ to /usr/share/ , you may want to munge the file names like this:
s/\/usr\/local\/share\//\/usr\/share\//g;
However, it's far easier and far less ugly to change the delimiters in this case:
s#/usr/local/share/#/usr/share/#g;
Modifiers We've already seen the /i modifier used to indicate that a match should be case insensitive. We've also seen the /g modifier to apply a substitution. What other modifiers are there?
/m - treat the string as multiple lines. Normally, ^ and $ match the very start and very end of the string. If the /m modifier is in play, then they will match the starts and ends of individual lines (separated by \n ). For example, given the string: "one\ntwo" , the pattern /^two$/ will not match, but /^two$/m will. /s - treat the string as a single line. Normally, . does not match a new line character; when /s is given, then it will. /g - as well as globally replacing in a substitution, allows us to match multiple times. When using this modifier, placing the \G anchor at the beginning of the regexp will anchor it to the end point of the last match. /x - allow the use of whitespace and comments inside a match. Regular expressions can get quite fiendish to read at times. The /x modifier is one way to stop them becoming so. For instance, if you're matching a string in a log file that contains a time, followed by a computer name in square brackets, then a message, the expression you'll create to extract the information may easily end up looking like this:
# Time in $1, machine name in $2, text in $3/^([0-2]\d:[0-5]\d:[0-5]\d)\s+\[([^\]]+)\]\s+(.*)$/
However, if you use the /x modifier, you can stretch it out as follows:
/^( # First group: time [0-2]\d : [0-5]\d : [0-5]\d )\s+\[ # Square bracket ( # Second group: machine name [^\]]+ # Anything that isn't a square bracket )\] # End square bracket
\s+ ( # Third group: everything else .* )$/x
Another way to tidy this up is to put each of the groups into variables and interpolate them:
Split We briefly saw split earlier on in the chapter, where we used it to break up a string into a list of words. In fact, we only saw it in a very simple form. Strictly speaking, it was a bit of a cheat to use it at all. We didn't see it then, but split was actually using a regular expression to do its stuff!
Using split on its own is equivalent to saying:
split /\s+/, $_;
which breaks the default string $_ into a list of substrings, using whitespace as a delimiter. However, we can also specify our own regular expression: perl goes through the string, breaking it whenever the regexp matches. The delimiter itself is thrown away.
For instance, on the UNIX operating system, configuration files are sometimes a list of fields separated by colons. A sample line from the password file looks like this:
kake:x:10018:10020::/home/kake:/bin/bash
To get at each field, we can split when we see a colon:
#!/usr/bin/perl # split.plx use warnings; use strict; my $passwd = "kake:x:10018:10020::/home/kake:/bin/bash"; my @fields = split /:/, $passwd; print "Login name : $fields[0]\n"; print "User ID : $fields[2]\n"; print "Home directory : $fields[5]\n";
>perl split.plx Login name : kake User ID : 10018 Home directory : /home/kake >
Note that the fifth field has been left empty. Perl will recognize this as an empty field, and the numbering used for the following entries takes account of this. So $fields[5] returns /home/kake , as we'd otherwise expect. Be careful though - if the line you are splitting contains empty fields at the end, they will get dropped.
Join To do the exact opposite, we can use the join operator. This takes a specified delimiter and interposes it between the elements of a specified array. For example:
#!/usr/bin/perl
# join.plx
use warnings;use strict; my $passwd = "kake:x:10018:10020::/home/kake:/bin/bash";my @fields = split /:/, $passwd;print "Login name : $fields[0]\n";print "User ID : $fields[2]\n";print "Home directory : $fields[5]\n";
my $passwd2 = join "#", @fields; print "Original password : $passwd\n"; print "New password : $passwd2\n";
>perl join.plx Login name : kake User ID : 10018 Home directory : /home/kake Original password : kake:x:10018:10020::/home/kake:/bin/bash New password : kake#x#10018#10020##/home/kake#/bin/bash >
Transliteration While we're looking at regular expressions, we should briefly consider another operator. While it's not directly associated with regexps, the transliteration operator has a lot in common with them and adds a very useful facility to the matching and substitution techniques we've already seen.
What this does is to correlate the characters in its two arguments, one by one, and use these pairings to substitute individual characters in the referenced string. It uses the syntax tr/one/two/ and (as with the matching and substitution operators) references the special variable $_ unless otherwise specified with =~ or !~ . In this case, it replaces all the 'o's in the referenced string with 't's, all the 'n's with 'w's, and all the 'e's with 'o's.
Let's say you wanted to replace, for some reason, all the numbers in a string with letters. You might say something like this:
$string =~ tr/0123456789/abcdefghij/;
This would turn, say, "2011064" into "cabbage". You can use ranges in transliteration but not in any of the character classes. We could write the above as:
$string =~ tr/0-9/a-j/;
The return value of this operator is, by default, the number of characters matched with those in the first argument. You can therefore use the transliteration operator to count the number of occurrences of certain characters. For example, to count the number of vowels in a string, you can use:
my $vowels = $string =~ tr/aeiou//;
Note that this will not actually substitute any of the vowels in the variable $string. As the second argument is blank, there is no correlation, so no substitution occurs. However, the transliteration operator can take the /d modifier, which will delete occurrences on the left that do not have a correlating character on the right. So, to get rid of all spaces in a string quickly, you could use this line:
$string =~ tr/ //d;
Common Blunders There are a few common mistakes people tend to make when writing regexps. We've already seen that /a*b*c*/ will happily match any string at all, since it matches each letter zero times. What else can go wrong?
Forgetting To Group/Bam{2}/ will match 'Bamm', while /(Bam){2}/ will match 'BamBam', so be careful when choosing which one to use. The same goes for alternation: /Simple|on/ will match 'Simple' and 'on', while /Sim(ple|on)/ will match both 'Simple' and 'Simon' Group each option separately. Getting The Anchors Wrong^ goes at the beginning, $ goes at the end. A dollar anywhere else in the string makes perl try and interpolate a variable. Forgetting To Escape Special Characters . Do you want them to have a special meaning? These are the characters to be careful of: . * ? + [ ] ( ) { } ^ $ | and of course \ itself. Not Counting from Zero The first entry in an array is given the index zero. Counting from Zero I know, I know! All along I've been telling you that computers start counting from zero. Nevertheless, there's always the odd exception - the first backreference is $1 . Don't blame Perl though - it took this behavior from a language called awk which used $1 as the first reference variable.
