Shell script that continuously checks a text file for log data and then runs a program

I have a java program that stops often due to erro开发者_运维百科rs which is logged in a .log file. What can be a simple shell script to detect a particular text in the last/latest line say

[INFO] Stream closed

and then run the following command

java -jar xyz.jar

This should keep on happening forever(possibly after every two minutes or so) because xyz.jar writes the log file.

The text stream closed can arrive a lot of times in the log file. I just want it to take an action when it comes in the last line.

How about

while [[ true ]];
do
  sleep 120
  tail -1 logfile | grep -q "[INFO] Stream Closed"
  if [[ $? -eq 1 ]]
  then
    java -jar xyz.jar &
  fi
done

There may be condition where the tailed last log "Stream Closed" is not the real last log and the process is still logging the messages. We can avoid this condition by checking if the process is alive or not. If the process exited and the last log is "Stream Closed" then we need to restart the application.

#!/bin/bash

java -jar xyz.jar &
PID=$1

while [ true ]
do
   tail -1 logfile | grep -q "Stream Closed" && kill -0 $PID && sleep 20 && continue
   java -jar xyz.jar &
   PID=$1
done

I would prefer checking whether the corresponding process is still running and restart the program on that event. There might be other errors that cause the process to stop. You can use a cronjob to periodically (like every minute) perform such a check.

Also, you might want to improve your java code so that it does not crash that often (if you have access to the code).

i solved this using a watchdog script that checks directly (grep) if program(s) is(are) running. by calling watchdog every minute (from cron under ubuntu), i basically guarantee (programs and environment are VERY stable) that no program will stay offline for more than 59 seconds.

this script will check a list of programs using the name in an array and see if each one is running, and, in case not, start it.

#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#
# This needs to be run as root or a user that can start system services.
#
# Revisions: 0.1 (20100506), 0.2 (20100507)

# first prog to check
NAME[0]=soc_gt2
# 2nd
NAME[1]=soc_gt0
# 3rd, etc etc
NAME[2]=soc_gp00


# START=/usr/sbin/$NAME
NOTIFY=you@gmail.com
NOTIFYCC=you2@mail.com
GREP=/bin/grep
PS=/bin/ps
NOP=/bin/true
DATE=/bin/date
MAIL=/bin/mail
RM=/bin/rm


for nameTemp in "${NAME[@]}"; do
    $PS -ef|$GREP -v grep|$GREP $nameTemp >/dev/null 2>&1
    case "$?" in
    0)
        # It is running in this case so we do nothing.
        echo "$nameTemp is RUNNING OK. Relax."

        $NOP
        ;;
    1)
        echo "$nameTemp is NOT RUNNING. Starting $nameTemp and sending notices."
        START=/usr/sbin/$nameTemp 
        $START 2>&1 >/dev/null &
        NOTICE=/tmp/watchdog.txt
        echo "$NAME was not running and was started on `$DATE`" > $NOTICE
        # $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
        $RM -f $NOTICE
        ;;
    esac
done

exit

i do not use the log verification, though you could easily incorporate that into your own version (just change grep for log check, for example).

if you run it from command line (or putty, if you are remotely connected), you will see what was working and what wasnt. have been using it for months now without a hiccup. just call it whenever you want to see what's working (regardless of it running under cron).

you could also place all your critical programs in one folder, do a directory list and check if every file in that folder has a program running under the same name. or read a txt file line by line, with every line correspoding to a program that is supposed to be running. etcetcetc

A good way is to use the awk command:

tail -f somelog.log | awk '/.*[INFO] Stream Closed.*/ { system("java -jar xyz.jar") }'

This continually monitors the log stream and when the regular expression matches its fires off whatever system command you have set, which is anything you would type into a shell.

If you really wanna be good you can put that line into a .sh file and run that .sh file from a process monitoring daemon like upstart to ensure that it never dies.

Nice and clean =D