Yogsototh's last blogs entries

split a file by keyword

2010-02-18T13:29:14Z

Strangely enough, I didn’t find any built-in tool to split a file by keyword. I made one myself in awk. I put it here mostly for myself. But it could also helps someone else. The following code split a file for each line containing the word UTC.

#!/usr/bin/env awk
BEGIN{i=0;}
/UTC/ { 
    i+=1;
    FIC=sprintf("fic.%03d",i); 
} 
{print $0>>FIC}

In my real world example, I wanted one file per day, each line containing UTC being in the following format:

Mon Dec  7 10:32:30 UTC 2009

I then finished with the following code:

#!/usr/bin/env awk
BEGIN{i=0;}
/UTC/ {
    date=$1$2$3; 
    if ( date != olddate ) {
        olddate=date;
        i+=1;
        FIC=sprintf("fic.%03d",i); 
    }
} 
{print $0>>FIC}

split a file by keyword

2010-02-18T13:29:14Z

#!/usr/bin/env awk
BEGIN{i=0;}
/UTC/ { 
    i+=1;
    FIC=sprintf("fic.%03d",i); 
} 
{print $0>>FIC}

In my real world example, I wanted one file per day, each line containing UTC being in the following format:

Mon Dec  7 10:32:30 UTC 2009

I then finished with the following code:

#!/usr/bin/env awk
BEGIN{i=0;}
/UTC/ {
    date=$1$2$3; 
    if ( date != olddate ) {
        olddate=date;
        i+=1;
        FIC=sprintf("fic.%03d",i); 
    }
} 
{print $0>>FIC}

All but something regexp (2)

2010-02-16T08:33:21Z

In my previous post I had given some trick to match all except something. On the same idea, the trick to match the smallest possible string. Say you want to match the string between ‘a’ and ‘b’, for example, you want to match:

a.....a......b..b..a....a....b...

Here are two common errors and a solution:

/a.*b/
a.....a......b..b..a....a....b...

/a.*?b/
a.....a......b..b..a....a....b...

/a[^ab]*b/
a.....a......b..b..a....a....b...

The first error is to use the evil .*. Because you will match from the first to the last. The next natural way, is to change the greediness. But it is not enough as you will match from the first a to the first b. Then a simple constatation is that our matching string shouldn’t contain any a nor b. Which lead to the last elegant solution.

Until now, that was, easy. Now, how do you manage when instead of a you have a string?

Say you want to match:

<li>...<li>

This is a bit difficult. You need to match

<li>[anything not containing <li>]</li>

The first method would be to use the same reasoning as in my previous post. Here is a first try:

<li>([^<]|<[^l]|<l[^i]|<li[^>])*</li>

But what about the following string:

<li>...<li</li>

That string should not match. This is why if we really want to match it correctly^† we need to add:

<li>([^<]|<[^l]|<l[^i]|<li[^>])*(|<|<l|<li)</li>

Yes a bit complicated. But what if the string I wanted to match was even longer?

Here is the algorithm way to handle this easily. You reduce the problem to the first one letter matching:

# transform a simple randomly choosen character
# to an unique ID 
# (you should verify the identifier is REALLY unique)
# beware the unique ID must not contain the 
# choosen character
s/X/_was_x_/g
s/Y/_was_y_/g

# transform the long string in this simple character
s/<li>/X/g
s/<\/li>/Y/g

# use the first method
s/X([^X]*)Y//g

# retransform choosen letter by string
s/X/<li>/g
s/Y/<\/li>/g

# retransform the choosen character back
s/_was_x_/X/g
s/_was_y_/Y/g

And it works in only 9 lines for any beginning and ending string. This solution should look less I AM THE GREAT REGEXP M45T3R, URAN00B, but is more convenient in my humble opinion. Further more, using this last solution prove you master regexp, because you know it is difficult to manage such problems with only a regexp.

^† I know I used an HTML syntax example, but in my real life usage, I needed to match between and. And sometimes the string could finish with e::.

All but something regexp (2)

2010-02-16T08:33:21Z

a.....a......b..b..a....a....b...

Here are two common errors and a solution:

/a.*b/
a.....a......b..b..a....a....b...

/a.*?b/
a.....a......b..b..a....a....b...

/a[^ab]*b/
a.....a......b..b..a....a....b...

Until now, that was, easy. Now, how do you manage when instead of a you have a string?

Say you want to match:

<li>...<li>

This is a bit difficult. You need to match

<li>[anything not containing <li>]</li>

The first method would be to use the same reasoning as in my previous post. Here is a first try:

<li>([^<]|<[^l]|<l[^i]|<li[^>])*</li>

But what about the following string:

<li>...<li</li>

That string should not match. This is why if we really want to match it correctly^† we need to add:

<li>([^<]|<[^l]|<l[^i]|<li[^>])*(|<|<l|<li)</li>

Yes a bit complicated. But what if the string I wanted to match was even longer?

Here is the algorithm way to handle this easily. You reduce the problem to the first one letter matching:

# transform a simple randomly choosen character
# to an unique ID 
# (you should verify the identifier is REALLY unique)
# beware the unique ID must not contain the 
# choosen character
s/X/_was_x_/g
s/Y/_was_y_/g

# transform the long string in this simple character
s/<li>/X/g
s/<\/li>/Y/g

# use the first method
s/X([^X]*)Y//g

# retransform choosen letter by string
s/X/<li>/g
s/Y/<\/li>/g

# retransform the choosen character back
s/_was_x_/X/g
s/_was_y_/Y/g

^† I know I used an HTML syntax example, but in my real life usage, I needed to match between ``. And sometimes the string could finish with e::.

All but something regexp

2010-02-15T09:16:12Z

Sometimes you cannot simply write:

if str.match(regexp) and 
    not str.match(other_regexp)
        do_something

and you have to make this behaviour with only one regular expression. The problem is the complementary of regular languages is not regular. Then, for some expression it is absolutely not impossible.

But sometimes with some simple regular expression it should be possible^†. Say you want to match everything containing the some word say bull but don’t want to match bullshit. Here is a nice way to do that:

# match all string containing 'bull' (bullshit comprised)
/bull/

# match all string containing 'bull' except 'bullshit'
/bull([^s]|$)|
bulls([^h]|$)|
bullsh([^i]|$)|
bullshi([^t]|$)/

# another way to write it would be
/bull([^s]|$|s([^h]|$)|sh([^i]|$)|shi([^t]|$))/

Let look closer. In the first line the expression is: bull([^s]|$), why does the $ is needed? Because, without it the word bull would be no more matched. This expression means:

The string finish by bull
or,
contains bull followed by a letter different from s.

And this is it. I hope it could help you.

Notice this method is not always the best. For example try to write a regular expression equivalent to the following conditional expression:

# Begin with 'a': ^a
# End with 'a': c$
# Contain 'b': .*b.*
# But isn't 'axbxc'
if str.match(/^a.*b.*c$/) and 
        not str.match(/^axbxc$/)
    do_something
end

A nice solution is:

/abc|           # length 3
a.bc|           # length 4
ab.c|
a[^x]b[^x]c|    # length 5
a...*b.*c|      # length >5
a.*b...*c/

This solution uses the maximal length of the string not to be matched. There certainly exists many other methods. But the important lesson is it is not straightforward to exclude something of a regular expression.

† It can be proved that any regular set minus a finite set is also regular.