Check Logfiles Only a Few Minutes Back

This is an update post. Previously I had a post here: http://blog.ls-al.com/check-logfiles-for-recent-entries-only/

The code has been problematic around when a new year starts because of the lack of a year in the log entries. I updated the code a little bit to account for the year ticking over. I may still need to come up with a better way but below seem to work ok.

#!/usr/bin/python
#

#: Script Name  : checkLogs.py
#: Version      : 0.0.1.1
#: Description  : Check messages for last x minutes.  Used in conjunction with checkLogs.sh and a cron schedule

from datetime import datetime, timedelta

#suppressPhrases = ['ssd','offline']
suppressPhrases = []

#now = datetime(2015,3,17,7,28,00)						## Get time right now. ie cron job execution
now = datetime.now()
day_of_year = datetime.now().timetuple().tm_yday   		## Used for special case when year ticks over. Older log entries should be one year older.

## How long back to check. Making it 11 mins because cron runs every 10 mins
checkBack = 11

lines = []

#print "log entries newer than " + now.strftime('%b %d %H:%M:%S') + " minus " + str(checkBack) + " minutes"

with open('/var/adm/messages', 'r') as f:
    for line in f:
      myDate = str(now.year) + " " + line[:15]          ## Solaris syslog format like this: Mar 11 12:47:23 so need to add year

      if day_of_year >= 1 and day_of_year <= 31:        ## Brain dead log has no year so special case during January
        if not "Jan" in myDate:         #2015 Dec 30
          myDate = str(now.year -1) + " " + line[:15]

      if myDate[3] == " ":								## What about "Mar  1" having double space vs "Mar 15". That will break strptime %d.
        myDate = myDate.replace(myDate[3],"0")			## zero pad string position 4 to make %d work?

      #print "myDate: %s and now: %s" % (myDate,now)
      lt = datetime.strptime(myDate,'%Y %b %d %H:%M:%S')
      diff = now - lt
      if diff.days <= 0:
        if lt > now - timedelta(minutes=checkBack):
          #print myDate + " --- diff: " + str(diff)
          match = False
          for s in suppressPhrases:
            i = line.find(s)
            if i > -1:
              match = True
          if not match:
            lines.append(line)

if lines:
    message = '\n'.join(lines)
    print message										    # do some grepping for my specific errors here.. send message per mail...

Check Logfiles For Recent Entries Only

Frequently I have a cron job to check for specific entries in log files but want to avoid being notified of something already checked. For example I want my 10 minute cron job to only look for most recent 10 minute entries.

Here is what I did in python.

from datetime import datetime, timedelta

## Get time right now. ie cron job execution
#now = datetime(2015,3,15,8,55,00)
now = datetime.now()

## How long back to check. Making it 11 mins because cron runs every 10 mins
checkBack = 11

lines = []

print "log entries newer than " + now.strftime('%b %d %H:%M:%S') + " minus " + str(checkBack) + " minutes"

with open('/var/log/syslog', 'r') as f:
    for line in f:
      ## Linux syslog format like this:
      ## Mar 15 08:50:01 EP45-DS3L postfix/sendmail[6492]: fatal
      ## Brain dead log has no year. So this hack will not work close to year ticking over
      myDate = str(now.year) + " " + line[:15]

      ## What about "Mar  1" having double space vs "Mar 15". That will break strptime %d.
      ## zero pad string position 4 to make %d work?
      if myDate[3] == " ":
        myDate = myDate.replace(myDate[3],"0")

      lt = datetime.strptime(myDate,'%Y %b %d %H:%M:%S')
      diff = now - lt
      if diff.days <= 0:
        if lt > now - timedelta(minutes=checkBack):
          # print myDate + " --- diff: " + str(diff)
          lines.append(line)

if lines:
    # message = '\n'.join(lines)
    # do some grepping for my specific errors here..
    # send message per mail...

Just for reference here is an older test where no year is used. This is doing a string compare but I have not tested this one good enough. Most likely it will fail when month ticks over Apr will not be bigger than Mar. Also what about midnight 23:59 > 00:00?

from datetime import datetime, timedelta
now = datetime.now()
lookback = timedelta(minutes=5)

## Linux syslog format "Mar 15 07:30:10 ..."
## Probably need to zero pad string position 4 to make %d work?
oldest = (now - lookback).strftime('%b %d %H:%M:%S')

with open('/var/log/syslog', 'r') as f:
    for line in f:
        if line[:15] > oldest:
          print "entry: " + line[:15] + " --- " + line[16:50]