We've now moved from matching a specific character to a more general type of character - when we don't know (or don't care) exactly what the character will be. Now we're going to see what happens when we want to talk about a more general quantity of characters: more than three digits in a row; two to four capital letters, and so on. The metacharacters that we use to deal with a number of characters in a row are called quantifiers .
Indefinite Repetition The easiest of these is the question mark. It should suggest uncertainty - something may be there, or it may not. That's exactly what it does: stating that the immediately preceding character(s) - or metacharacter(s) - may appear once, or not at all. It's a good way of saying that a particular character or group is optional. To match the word 'he or she', you can put:
> perl matchtest.plx Enter some text to find: \bs?he\b The text matches the pattern '\bs?he\b'. >
To make a series of characters (or metacharacters) optional, group them in parentheses as before. Did he say 'what the Entish is' or 'what the Entish word is'? Either will do:
> perl matchtest.plx Enter some text to find: what the Entish (word )?is The text matches the pattern 'what the Entish (word )?is'. >
Notice that we had to put the space inside the group: otherwise we end up with two spaces between 'Entish' and 'is', whereas our text only has one:
> perl matchtest.plx Enter some text to find: what the Entish (word)? is 'what the Entish (word)? is' was not found. >
As well as matching something one or zero times, you can match something one or more times. We do this with the plus sign - to match an entire word without specifying how long it should be, you can say:
> perl matchtest.plx Enter some text to find: \b\w+\b The text matches the pattern '\b\w+\b'. >
In this case, we match the first available word - I.
If, on the other hand, you have something which may be there any number of times but might not be there at all - zero or one or many - you need what's called 'Kleene's star': the * quantifier. So, to find a capital letter after any - but possibly no - spaces at the start of the string, what would you do? The start of the string, then any number of whitespace characters, then a capital:
> perl matchtest.plx Enter some text to find: ^\s*[A-Z] '^\s*[A-Z]' was not found.
>
Of course, our test string begins with a quote, so the above pattern won't match, but, sure enough, if you take away that first quote, the pattern will match fine. Let's review the three qualifiers:
/bea?t/
Matches either 'beat' or 'bet'
/bea+t/
Matches 'beat', 'beaat', 'beaaat'Ö
/bea*t/
Matches 'bet', 'beat', 'beaat'Ö
Novice Perl programmers tend to go to town on combinations of dot and star, and the results often surprise them, particularly when it comes to searching-and-replacing. We'll explain the rules of the regular expression matcher shortly, but bear the following in mind:
A regular expression should hardly ever start or finish with a starred character.
You should also consider the fact that .* and .+ in the middle of a regular expression will match as much of your string as they possibly can. We'll look more at this 'greedy' behavior later on.
Well-Defined Repetition If you want to be more precise about how many times a character or roups of characters might be repeated, you can specify the maximum and minimum number of repeats in curly brackets. '2 or 3 spaces' can be written as follows:
> perl matchtest.plx Enter some text to find: \s{2,3} '\s{2,3}' was not found. >
So we have no doubled or trebled spaces in our string. Notice how we construct that - the minimum, a comma, and the maximum, all inside braces. Omitting either the maximum or the minimum signifies 'or more' and 'or fewer' respectively. For example, {2,} denotes '2 or more', while {,3} is '3 or fewer'. In these cases, the same warnings apply as for the star operator.
Finally, you can specify exactly how many things are to be in a row by simply putting that number inside the curly brackets. Here's the five-letter-word example tidied up a little:
> perl matchtest.plx Enter some text to find: \b\w{5}\b '\b\w{5}\b' was not found. >
Summary Table To refresh your memory, here are the various metacharacters we've seen so far:
Metacharacter
Meaning
[abc]
any one of the characters a , b , or c.
[^abc]
any one character other than a , b, or c.
Table continued on following page
Metacharacter
Meaning
[a-z]
any one ASCII character between a and z.
\d \D
a digit; a non-digit.
\w \W
a 'word' character; a non-'word' character.
\s \S
a whitespace character; a non-whitespace character.
\b
the boundary between a \w character and a \W character.
.
any character (apart from a new line).
(abc)
the phrase 'abc ' as a group.
?
preceding character or group may be present 0 or 1 times.
+
preceding character or group is present 1 or more times.
*
preceding character or group may be present 0 or more times.
{x,y}
preceding character or group is present between x and y times.
{,y}
preceding character or group is present at most y times.
{x,}
preceding character or group is present at least x times.
{x}
preceding character or group is present x times.
Backreferences What if we want to know what a certain regular expression matched? It was easy when we were matching literal strings: we knew that 'Case' was going to match those four letters and nothing else. But now, what matches? If we have /\w{3}/, which three word characters are getting matched?
Perl has a series of special variables in which it stores anything that's matched with a group in parentheses. Each time it sees a set of parentheses, it copies the matched text inside into a numbered variable - the first matched group goes in $1 , the second group in $2 , and so on. By looking at these variables, which we call the backreference variables, we can see what triggered various parts of our match, and we can also extract portions of the data for later use.
First, though, let's rewrite our test program so that we can see what's in those variables:
Try it out : A Second Pattern Tester
#!/usr/bin/perl # matchtest2.plx use warnings; use strict; $_ = '1: A silly sentence (495,a) *BUT* one which will be useful. (3)'; print "Enter a regular expression: "; my $pattern = <STDIN>; chomp($pattern);
if (/$pattern/) { print "The text matches the pattern '$pattern'.\n"; print "\$1 is '$1'\n" if defined $1; print "\$2 is '$2'\n" if defined $2; print "\$3 is '$3'\n" if defined $3; print "\$4 is '$4'\n" if defined $4; print "\$5 is '$5'\n" if defined $5; } else { print "'$pattern' was not found.\n"; }
Note that we use a backslash to escape the first 'dollar' symbol in each print statement, thus displaying the actual symbol, while leaving the second in each to display the contents of the appropriate variable. We've got our special variables in place, and we've got a new sentence to do our matching on. Let's see what's been happening:
> perl matchtest2.plx Enter a regular expression: ([a-z]+) The text matches the pattern '([a-z]+)'. $1 is 'silly'
> perl matchtest2.plx Enter a regular expression: (\w+) The text matches the pattern '(\w+)'.
$1 is '1' > perl matchtest2.plx Enter a regular expression: ([a-z]+)(.*)([a-z]+) The text matches the pattern '([a-z]+)(.*)([a-z]+)'. $1 is 'silly' $2 is ' sentence (495,a) *BUT* one which will be usefu' $3 is 'l'
> perl matchtest2.plx
Enter a regular expression: e(\w|n\w+) The text matches the pattern 'e(\w|n\w+)'. $1 is 'n'
How It Works By printing out what's in each of the groups, we can see exactly what caused perl to start and stop matching, and when. If we look carefully at these results, we'll find that they can tell us a great deal about how perl handles regular expressions.
How the Engine Works We've now seen most of the syntax behind regular expression matching and plenty of examples of it in action. The code that does all the matching is called perl's 'regular expression engine'. You might now be wondering about the exact rules applied by this engine when determining whether or not a piece of text matches. And how much of it matches what. From what our examples have shown us, let us make some deductions about the engine's operation. Our first expression, ([a-z]+) plucked out a set of one-or-more lower-case letters. The first such set that perl came across was 'silly '. The next character after 'y ' was a space, and so no longer matched the expression.
Rule one: Once the engine starts matching, it will keep matching a character at a time for as long as it can. Once it sees something that doesn't match, however, it has to stop. In this example, it can never get beyond a character that is not a lower case letter. It has to stop as soon as it encounters one. Next, we looked for a series of word characters, using (\w+ ). The engine started looking at the beginning of the string and found one, '1'. The next character was not a word character (it was a colon), and so the engine had to stop.
Rule two: Unlike me, the engine is eager . It's eager to start work and eager to finish, and it starts matching as soon as possible in the string; if the first character doesn't match, try and start matching from the second. Then take every opportunity to finish as quickly as possible.
Then we tried this:([a-z]+)(.*)([a-z]+) . The result we got with this was a little strange. Let's look at it again:
> perl matchtest2.plx Enter a regular expression: ([a-z]+)(.*)([a-z]+) The text matches the pattern '([a-z]+)(.*)([a-z]+)'. $1 is 'silly' $2 is ' sentence (495,a) *BUT* one which will be usefu' $3 is 'l' >
Our first group was the same as what matched before - nothing new there. When we could no longer match lower case letters, we switched to matching anything we could. Now, this could take up the rest of the string, but that wouldn't allow a match for the third group. We have to leave at least one lower-case letter.
So, the engine started to reverse back along the string, giving characters up one by one. It gave up the closing bracket, the 3, then the opening bracket, and so on, until we got to the first thing that would satisfy all the groups and let the match go ahead - namely a lower-case letter: the 'l' at the end of 'useful'.
From this, we can draw up the third rule:
Rule three: Like me, in this case, the engine is greedy. If you use the + or * operators, they will try and steal as much of the string as possible. If the rest of the expression does not match, it grudgingly gives up a character at a time and tries to match again, in order to find the fullest possible match. We can turn a greedy match into a non-greedy match by putting the ? operator after either the plus or star. For instance, let's turn this example into a non-greedy version: ([a-z]+)(.*?)([a-z]+) . This gives us an entirely different result:
> perl matchtest2.plx Enter a regular expression: ([a-z]+)(.*?)([a-z]+) The text matches the pattern '([a-z]+)(.*?)([a-z]+)'. $1 is 'silly' $2 is ' ' $3 is 'sentence' >
Now we've shut off rule three, rule two takes over. The smallest possible match for the second group was a single space. First, it tried to get nothing at all, but then the third group would be faced with a space. This wouldn't match. So, we grudgingly accept the space and try and finish again. This time the third group has some lower case letters, and that can match as well.
What if we turn off greediness in all three groups, and say this: ([a-z]+?)(.*?)([a-z]+?)
> perl matchtest2.plx Enter a regular expression: ([a-z]+?)(.*?)([a-z]+?) The text matches the pattern '([a-z]+?)(.*?)([a-z]+?)'. $1 is 's' $2 is '' $3 is 'i' >
What about this? Well, the smallest possible match for the first group is the 's' of silly. We asked it to find one character or more, and so the smallest it could find was one. The second group actually matched no characters at all. This left the third group facing an 'i', which it took to complete the match.
Our last example included an alternation:
> perl matchtest2.plx
Enter a regular expression: e(\w|n\w+) The text matches the pattern 'e(\w|n\w+)'. $1 is 'n' >
The engine took the first branch of the alternation and matched a single character, even though the second branch would actually satisfy greed. This leads us onto the fourth rule:
Rule four: Again like me, the regular expression engine hates decisions . If there are two branches, it will always choose the first one, even though the second one might allow it to gain a longer match. To summarize:
The regular expression engine starts as soon as it can, grabs as much as it can, then tries to finish as soon as it can, while taking the first decision available to it.
