How can I view log files in Linux and apply custom filters while viewing?

I need to read through some gigantic log files on a Linux system. There's a lot 开发者_运维百科of clutter in the logs. At the moment I'm doing something like this:

cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less

But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.

I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.

Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.

Changing the code that generates the log to generate less is not an option.

Use &pattern command within less.

From the man page for less

&pattern

          Display  only  lines which match the pattern; lines which do not
          match the pattern are not displayed.  If pattern  is  empty  (if
          you  type  &  immediately  followed  by ENTER), any filtering is
          turned off, and all lines are displayed.  While filtering is  in
          effect,  an  ampersand  is  displayed  at  the  beginning of the
          prompt, as a reminder that some lines in the file may be hidden.

          Certain characters are special as in the / command:

          ^N or !
                 Display only lines which do NOT match the pattern.

          ^R     Don't interpret regular expression  metacharacters;  that
                 is, do a simple textual comparison.

Try the multitail tool - as well as letting you view multile logs at once, I'm pretty sure it lets you apply regex filters interactively.

Based on ghostdog74's answer and the less manpage, I came up with this:

~/.bashrc:

export LESSOPEN='|~/less-filter.sh %s'
export LESS=-R  # to allow ANSI colors

~/less-filter.sh:

#!/bin/sh
case "$1" in
*logfile*.log*) ~/less-filter.sed < $1
  ;;
esac

~/less-filter.sed:

/deleteLinesLikeThis/d  # to filter out lines
s/this/that/  # to change text on lines (useful to colorize using ANSI escapes)

Then:

• less logfileFooBar.log.1 -- applies the filter applies automatically.
• cat logfileFooBar.log.1 | less -- to see the log without filtering

This is adequate for now but I would still like to be able to edit the filters on the fly.

see the man page of less. there are some options you can use to search for words for example. It has line editing mode as well.

There's an application by Casstor Software Solutions called LogFilter (www.casstor.com) that can edit Windows/Mac/Linux text files and can easily perform file filtering. It supports multiple filters as well as regular expressions. I think it might be what you're looking for.