Monday, December 31, 2012

Sed: Mutli-Line Replacement Between Two Patterns

This post has some useful sed commands which can be used to perform replacements and deletes between two patterns across multiple lines. For example, consider the following file:
$ cat file
line 1
line 2
foo
line 3
line 4
line 5
bar
line 6
line 7
1) Replace text on each line between two patterns (inclusive):
To perform a replacement on each line between foo and bar, including the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{s/./x/g}' file
line 1
line 2
xxx
xxxxxx
xxxxxx
xxxxxx
xxx
line 6
line 7
2) Replace text on each line between two patterns (exclusive):
To perform a replacement on each line between foo and bar, excluding the lines containing foo and bar, use the following:
$ sed '/foo/,/bar/{/foo/n;/bar/!{s/./x/g}}' file
line 1
line 2
foo
xxxxxx
xxxxxx
xxxxxx
bar
line 6
line 7
3) Delete lines between two patterns (inclusive):
To delete all lines between foo and bar, including the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.

$ sed '/foo/,/bar/d' file
line 1
line 2
line 6
line 7
4) Delete lines between two patterns (exclusive):
To delete all lines between foo and bar, excluding the lines containing foo and bar, use the same replacement sed command as shown above, but simply change the replacement expression to a delete.
$ sed '/foo/,/bar/ {/foo/n;/bar/!d}' file
line 1
line 2
foo
bar
line 6
line 7
5) Replace all lines between two patterns (inclusive):
To perform a replacement on a block of lines between foo and bar, including the lines containing foo and bar, use:
$ sed -n '/foo/{:a;N;/bar/!ba;N;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
REPLACEMENT
line 6
line 7
How it works:
/foo/{                   # when "foo" is found
  :a                     # create a label "a"
    N                    # store the next line
  /bar/!ba               # goto "a" and keep looping and storing lines until "bar" is found
  N                      # store the line containing "bar"
  s/.*\n/REPLACEMENT\n/  # delete the lines
}
p                        # print
6) Replace all lines between two patterns (exclusive):
To perform a replacement on a block of lines between foo and bar, excluding the lines containing foo and bar, use:
$ sed -n '/foo/{p;:a;N;/bar/!ba;s/.*\n/REPLACEMENT\n/};p' file
line 1
line 2
foo
REPLACEMENT
bar
line 6
line 7
References:
Sed - An Introduction and Tutorial by Bruce Barnett

13 comments:

  1. Awesome post..
    Really helpful...

    ReplyDelete
  2. very useful, thank you!!

    ReplyDelete
  3. Nice article. In #5, how can I replace contents from another file? i.e. REPLACEMENT is contents of another file

    ReplyDelete
  4. Thanks for taking the time to put this post together, it was very helpful!

    ReplyDelete
  5. Awesome article, quite helpful.

    However I must say that the number 6 needs a bit of tweaking. If you run the command with sed -n the output is fine. But if you use sed -n -i, it double each line in the file.

    To get it work, I had to remove the first "p" after the bracket and also add my first pattern in the replacement (dunno why the second one was not affected)

    Excuse for my bad english, I am french from Québec, Canada

    Larow

    ReplyDelete
  6. Very helpful! Thanks!

    ReplyDelete
  7. Excellent work Fahd.

    However just a missing piece that I couldn't work with above. How would you remove everything between foo and bar without deleting full lines?

    So for example if I have a source of this text.

    line 1
    line 2 foo line 3
    line 4
    line 5 bar line 6
    line 7

    And I want to remove everything between foo and bar only where output will look like below.
    line 1
    line 2 line 6
    line 7

    Many thanks in advance.

    ReplyDelete
  8. Really cool, thanks!

    ReplyDelete
  9. Hello,

    Apparently you can change this:

    sed '/foo/,/bar/ {/foo/n;/bar/!d}' file

    to this:

    sed '/foo/,/bar/ {//!d}' file

    I am still trying to figure out how that works as there is not much documentation on it. It is probably something of the sort "use the last regexp" or something.

    ReplyDelete
  10. Wow Sir very impressive your sed domain!

    ReplyDelete
  11. Great Article.Thanks a lot for help

    ReplyDelete
  12. How to use a variable instead of "REPLACEMENT" string in number 6 example.
    v="server 192.168.0.1"
    sed -n '/foo/{p;:a;N;/bar/!ba;s/.*\n/$v\n/};p' file

    ReplyDelete