Category: Python

Jan 22

AWS API and Python Boto

Quick note on connection to EC2 to list instances.

- Ensure IAM User permissions. In my case I tried EC2FullAccess.
- Ensure you have your access and secret key handy.
- This example just cycle through regions and list any instances.

import argparse
import boto.ec2

access_key = ''
secret_key = ''

def get_ec2_instances(region):
    ec2_conn = boto.ec2.connect_to_region(region,
                aws_access_key_id=access_key,
                aws_secret_access_key=secret_key)
    reservations = ec2_conn.get_all_reservations()
    for reservation in reservations:    
        print region+':',reservation.instances

    for vol in ec2_conn.get_all_volumes():
        print region+':',vol.id

def main():
    regions = ['us-east-1','us-west-1','us-west-2','eu-west-1','sa-east-1',
                'ap-southeast-1','ap-southeast-2','ap-northeast-1']
    parser = argparse.ArgumentParser()
    parser.add_argument('access_key', help='Access Key');
    parser.add_argument('secret_key', help='Secret Key');
    args = parser.parse_args()
    global access_key
    global secret_key
    access_key = args.access_key
    secret_key = args.secret_key
    
    for region in regions: get_ec2_instances(region)

if __name__ =='__main__':main()

Example:

$ python list.py myaccess_key mysecret_key
us-east-1: [Instance:i-1aac5699]
us-east-1: vol-d121290e

Comments Off on AWS API and Python Boto
comments

Jan 22

Python Dict for Arrays

Some more examples of Python dicts and array general use.

import pprint

###  Example python dict as an associative multi array
###  Example is using an unique key
pp = pprint.PrettyPrinter(indent=2)

table = {}
table[1]={'LastName':'Sable','name':'Sam'}
table[2]={'LastName':'Sable','name':'Samantha'}
table[3]={'LastName':'Sable','name':'Stevie'}

pp.pprint(table)

print table[2]
print table[3]['name']

Comments Off on Python Dict for Arrays
comments

Jan 05

Oracle OVM rest api example

If you hate the Oracle OVM CLI and having to use expect scripts you may want to look into the web services API.  Note it looks like SOAP will be decommissioned soon so use REST.

$ cat ovm_using_rest.py 
import requests

s=requests.Session()
s.auth=('admin','pwd_removed')
s.verify=False #disables SSL certificate verification

s.headers.update({'Accept': 'application/json', 'Content-Type': 'application/json'})

baseUri='https://ovmm:7002/ovm/core/wsapi/rest'

print "\nServer List:"
print "##############"
r=s.get(baseUri+'/Server')
for i in r.json():
  # do something with the content
  print '{:20} {:20}'.format(i['serverRunState'],i['name'])

print "\nVM List:"
print "########"
r=s.get(baseUri+'/Vm')
for i in r.json():
  # do something with the content
  print '{:20} {:20}'.format(i['vmRunState'],i['name'])
  #print '{name} '.format(name=i['name'])

Output:

$ python ovm_using_rest.py 

Server List:
##############
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
RUNNING              ovms2            
RUNNING              ovms3             
RUNNING              ovms1             

VM List:
########
TEMPLATE             EXALYTICS_BASE_LINUX_OEL5_GUEST_VM_TEMPLATE_2.2.0.0.0.el5
TEMPLATE             EXALYTICS_BASE_LINUX_OEL6_GUEST_VM_TEMPLATE_2.2.0.0.0.el6
RUNNING              OBIEXA3           
STOPPED              obiexa4           
[..]
STOPPED              EXALYTICS_BASE_LINUX_OEL5_GUEST_VM_TEMPLATE_2.0.1.4.0
TEMPLATE             OBIEE12C_TEMPLATE.0 
RUNNING              OBIEXA01         

Comments Off on Oracle OVM rest api example
comments

Dec 10

Nagios Downtime using a ServiceGroup

This is not a complete script. I was only interested in scheduling a SERVICEGROUP through the Nagios command file (NAGIOSCMD). This is for obvious reasons if you ever used Nagios in large environments. It is painful to schedule and cancel downtimes.

Since I wanted to be able to delete multiple downtime entries and I used a feature added somewhere in 3.x called DEL_DOWNTIME_BY_START_TIME_COMMENT. The cancellation of downtime is done by providing a Start date and a comment.

import sys, argparse, time
from datetime import datetime, timedelta

##  EXAMPLES
## # python nagios_downtime.py --action add --Servicegroup PRDVIPS --begin "2016-12-10 8:36" --duration 10 --author rrosso --comment "Test Scheduling Downtime in Nagios"
## [1481387741.0] SCHEDULE_SERVICEGROUP_SVC_DOWNTIME;PRDVIPs;1481387760.0;1481388360.0;1;0;600;rrosso;Test Scheduling Downtime in Nagios
## # python nagios_downtime.py --action cancel --begin "2016-12-10 8:36" --comment="Test Scheduling Downtime in Nagios"
## [1481387769.0] DEL_DOWNTIME_BY_START_TIME_COMMENT;1481387760.0;Test Scheduling Downtime in Nagios
##

VERSION = "0.4"
VERDATE = "2016-12-10"

NAGIOSCMD =  "/usr/local/nagios/var/rw/nagios.cmd"
now = datetime.now()
cmd = '[' + str(time.mktime(now.timetuple())) + '] '
execline = ''

parser = argparse.ArgumentParser(description='Nagios Downtime Scheduler.')
parser.add_argument('--action', dest='action', help='use add or cancel as action for downtime entries', required=True)
parser.add_argument('--Servicegroup', dest='servicegroup', help ='Schedule Downtime a specific ServiceGroup')
parser.add_argument('--duration', dest='duration', help='Duration of downtime, in minutes.)
parser.add_argument('--begin', dest='begin', help='Beginning of Downtime. ex: 2016-12-10 18:10', required=True)
parser.add_argument('--author', dest='author', default='admin', help='Author: Who is scheduling the downtime?')
parser.add_argument('--comment', dest='comment', help='Comment: Reason for scheduling the downtime.', required=True)
parser.add_argument('--dryrun', action='store_true', help='Dry run.  Do not do anything but show action.')

args = parser.parse_args()

## need some argument checking here.  what is required what conflicts etc..
if (args.action not in ['add','cancel']):
  sys.exit(1)

if (args.begin != None):
  #check for proper format here...
  #beginTime = datetime.datetime(2016,12,8,13,0).strftime('%s')
  beginTime = datetime.strptime(args.begin,'%Y-%m-%d %H:%M')

if (args.action == 'add'):
  if (args.servicegroup):
    cmd = cmd + 'SCHEDULE_SERVICEGROUP_SVC_DOWNTIME;'
    endTime = beginTime + timedelta(minutes=int(args.duration))
    execline=cmd + 'PRDVIPs;' + str(time.mktime(beginTime.timetuple())) + ';' + str(time.mktime(endTime.timetuple())) + ';1;0;' + '600' +';' + args.author + ';' + args.comment + '\n'

if (args.action == 'cancel'):
  cmd = cmd + 'DEL_DOWNTIME_BY_START_TIME_COMMENT;'
  execline=cmd + str(time.mktime(beginTime.timetuple())) + ';' + args.comment + '\n'

print 'Nagios CMD interaction will be: ' + execline

if (args.dryrun):
  print "Note that this is a dry run ie so not committing transaction"
else:
  print "Note that this is not a dry run ie --dryrun was not used so committing transaction"
  f = open(NAGIOSCMD,'w')
  f.write(execline)

Comments Off on Nagios Downtime using a ServiceGroup
comments

Jun 23

Solaris Find Process Id tied to IP Address

Recently I needed to find out who is connecting to an Oracle database and at the same time I wanted to see the load the specific connection add to the CPU. So in short I needed IP Address and Port tied to a Unix Pid.

I wrote this quick and dirty python script.

#!/usr/bin/python
import subprocess

## No doubt you would want to exclude some non local or expected IP addresses
excludeIPs="10.2.16.86|10.2.16.62|10.2.16.83|\*.\*"

p = subprocess.Popen("/usr/bin/netstat -an | grep 1521 | awk '{print $2}' | egrep -v '" + excludeIPs + "'", stdout=subprocess.PIPE, shell=True)
nonlocals= p.stdout
 
if nonlocals <> '':
  p = subprocess.Popen("pfiles `ls /proc` 2>/dev/null", stdout=subprocess.PIPE, shell=True)
  try:
    outs, errs = p.communicate()
  except TimeoutExpired:
    p.kill()
    outs, errs = p.communicate()

  pfiles = outs

  for line in nonlocals:
    line=line.strip()
    (IP,port) = line.rsplit('.',1)
    print ("Going to find PID for connection with IP %s and port %s" % (IP,port) )

    for line in pfiles.splitlines():
      if line[:1].strip() <> '':
        pid = line
      if "port: " + port in line:
        print pid

I plan to enhance this script a little bit but for now it did exactly what I needed.

Comments Off on Solaris Find Process Id tied to IP Address
comments

Jan 16

Get Third Sunday of The Month

Sometimes you have to rely on a routine to reliably provide dates. In this example I want to know for certain what the third Sunday of every month is. Could be various reasons but mostly for me it is to predict maintenance windows or use during scripting for automation.

$ cat /tmp/thirdweek.py 
from datetime import date, timedelta

def allsundays(year):
   d = date(year, 1, 1)                    # January 1st
   d += timedelta(days = 6 - d.weekday())  # First Sunday
   while d.year == year:
      yield d
      d += timedelta(days = 7)

def SundaysInMonth(year,month):
  d = date(year,month,1)		  # First day of the month
  d += timedelta(days = 6 - d.weekday())  # First Sunday
  while d.year == year and d.month == month:
      yield d
      d += timedelta(days = 7)

def xSundayOfMonth(x,Sundays):
  i=0
  for Sunday in Sundays:
    i=i+1
    if i == x: 
      #print "%s - %s" % (i,Sunday)
      return Sunday

print "All Sundays in Year"
for d in allsundays(2016):
   print d

print "All Third Sundays in Year"
for month in range(1,13): 
  Sundays=SundaysInMonth(2016,month)
  print xSundayOfMonth(3,Sundays)

Comments Off on Get Third Sunday of The Month
comments

Jan 04

Check Logfiles Only a Few Minutes Back

This is an update post. Previously I had a post here: http://blog.ls-al.com/check-logfiles-for-recent-entries-only/

The code has been problematic around when a new year starts because of the lack of a year in the log entries. I updated the code a little bit to account for the year ticking over. I may still need to come up with a better way but below seem to work ok.

#!/usr/bin/python
#

#: Script Name  : checkLogs.py
#: Version      : 0.0.1.1
#: Description  : Check messages for last x minutes.  Used in conjunction with checkLogs.sh and a cron schedule

from datetime import datetime, timedelta

#suppressPhrases = ['ssd','offline']
suppressPhrases = []

#now = datetime(2015,3,17,7,28,00)						## Get time right now. ie cron job execution
now = datetime.now()
day_of_year = datetime.now().timetuple().tm_yday   		## Used for special case when year ticks over. Older log entries should be one year older.

## How long back to check. Making it 11 mins because cron runs every 10 mins
checkBack = 11

lines = []

#print "log entries newer than " + now.strftime('%b %d %H:%M:%S') + " minus " + str(checkBack) + " minutes"

with open('/var/adm/messages', 'r') as f:
    for line in f:
      myDate = str(now.year) + " " + line[:15]          ## Solaris syslog format like this: Mar 11 12:47:23 so need to add year

      if day_of_year >= 1 and day_of_year <= 31:        ## Brain dead log has no year so special case during January
        if not "Jan" in myDate:         #2015 Dec 30
          myDate = str(now.year -1) + " " + line[:15]

      if myDate[3] == " ":								## What about "Mar  1" having double space vs "Mar 15". That will break strptime %d.
        myDate = myDate.replace(myDate[3],"0")			## zero pad string position 4 to make %d work?

      #print "myDate: %s and now: %s" % (myDate,now)
      lt = datetime.strptime(myDate,'%Y %b %d %H:%M:%S')
      diff = now - lt
      if diff.days <= 0:
        if lt > now - timedelta(minutes=checkBack):
          #print myDate + " --- diff: " + str(diff)
          match = False
          for s in suppressPhrases:
            i = line.find(s)
            if i > -1:
              match = True
          if not match:
            lines.append(line)

if lines:
    message = '\n'.join(lines)
    print message										    # do some grepping for my specific errors here.. send message per mail...

Comments Off on Check Logfiles Only a Few Minutes Back
comments

Mar 15

Check Logfiles For Recent Entries Only

Frequently I have a cron job to check for specific entries in log files but want to avoid being notified of something already checked. For example I want my 10 minute cron job to only look for most recent 10 minute entries.

Here is what I did in python.

from datetime import datetime, timedelta

## Get time right now. ie cron job execution
#now = datetime(2015,3,15,8,55,00)
now = datetime.now()

## How long back to check. Making it 11 mins because cron runs every 10 mins
checkBack = 11

lines = []

print "log entries newer than " + now.strftime('%b %d %H:%M:%S') + " minus " + str(checkBack) + " minutes"

with open('/var/log/syslog', 'r') as f:
    for line in f:
      ## Linux syslog format like this:
      ## Mar 15 08:50:01 EP45-DS3L postfix/sendmail[6492]: fatal
      ## Brain dead log has no year. So this hack will not work close to year ticking over
      myDate = str(now.year) + " " + line[:15]

      ## What about "Mar  1" having double space vs "Mar 15". That will break strptime %d.
      ## zero pad string position 4 to make %d work?
      if myDate[3] == " ":
        myDate = myDate.replace(myDate[3],"0")

      lt = datetime.strptime(myDate,'%Y %b %d %H:%M:%S')
      diff = now - lt
      if diff.days <= 0:
        if lt > now - timedelta(minutes=checkBack):
          # print myDate + " --- diff: " + str(diff)
          lines.append(line)

if lines:
    # message = '\n'.join(lines)
    # do some grepping for my specific errors here..
    # send message per mail...

Just for reference here is an older test where no year is used. This is doing a string compare but I have not tested this one good enough. Most likely it will fail when month ticks over Apr will not be bigger than Mar. Also what about midnight 23:59 > 00:00?

from datetime import datetime, timedelta
now = datetime.now()
lookback = timedelta(minutes=5)

## Linux syslog format "Mar 15 07:30:10 ..."
## Probably need to zero pad string position 4 to make %d work?
oldest = (now - lookback).strftime('%b %d %H:%M:%S')

with open('/var/log/syslog', 'r') as f:
    for line in f:
        if line[:15] > oldest:
          print "entry: " + line[:15] + " --- " + line[16:50]

Comments Off on Check Logfiles For Recent Entries Only
comments

Jan 23

Python Output Align by Column

I have used several ways in the past to have very simple column headers and subsequent output lines line up. Of course everything can be static but I like to have column headers adjust their size based on some of the data.

For example if I have column headers in an array like this:
header = ['File Name','File Size']

I want to use this array to print the header line but before printing it I want to adjust the column width for 'File Name' to the longest file name that is in the list yet to be processed. The following example is one way of doing it.

I played with other options also(see update 01.24.15 below).

Honestly I think the main reason for doing it dynamically is I like to print the headers and data with minimal lines of code.

#!/usr/bin/env python
import os,glob,sys
FORMAT = [('File Name',10),('Size',12)]

## In a real use case this function will probably do a lot more
## for example MB to GB conversions etc.
## But important here is where I make the adjustment to the main array
## to stretch the 'File Name' field.
def makeEntry(fileN,fileS):
  global FORMAT
  entry = []
  entry = [fileN, fileS]

  if len(fileN) > FORMAT[0][1]:
    n = dict(FORMAT)			# tuples are immutable so have to change to something writeable
    n['File Name'] = len(fileN) + 2	# expanding field width
    FORMAT = n.items()			# overwrite original

  return entry

os.chdir('/DATA/')
video_files = []
for files in ('*.m4v', '*.mp4', '*.ts','*.dvr-ms','*.avi','*.mkv'):
  video_files.extend(glob.glob(files))

## Loop through files and build an array
fArr = []
for fileN in video_files:
  fArr.append(makeEntry(fileN, os.path.getsize(fileN)))

## Print header
for item in FORMAT:
  print "{header:{width}}".format (header=item[0], width=item[1]),
print

## Print entries
for entry in fArr:
  #print ("%-" + str(FORMAT[0][1]) + "s %-" + str(FORMAT[1][1]) + "s") % (entry[0], entry[1])		# old style
  print "{f0:{w0}} {f1:{w1}}".format ( f0=entry[0], w0=FORMAT[0][1], f1=entry[1], w1=FORMAT[1][1] )	# new style

Update 01.24.15
Above example is fine except I did not like the way I updated the tuple. Below example use a dict for the header.

#!/usr/bin/env python
import os,glob,sys

FORMAT = {'File Name':10,'Size':12} 

def makeEntry(fileN,fileS):
  global FORMAT
  entry = []
  entry = [fileN, fileS]
  if len(fileN) > FORMAT['File Name']:
    FORMAT['File Name'] = len(fileN) + 2	# expanding field width
  return entry

os.chdir('/DATA/')
video_files = []
for files in ('*.m4v', '*.mp4', '*.ts','*.dvr-ms','*.avi','*.mkv'):
  video_files.extend(glob.glob(files))

fArr = []
for fileN in video_files:
  fArr.append(makeEntry(fileN, os.path.getsize(fileN)))

## Print header.
print "{header:{width}}".format (header='File Name', width=FORMAT['File Name']),
print "{header:{width}}".format (header='Size', width=FORMAT['Size'])
print

for entry in fArr:
  print "{f0:{w0}} {f1:{w1}}".format ( f0=entry[0], w0=FORMAT['File Name'], f1=entry[1], w1=FORMAT['Size'] )

Comments Off on Python Output Align by Column
comments

Jan 15

Dictionaries or Associative Arrays in Python

I have a couple previous articles around a similar topic but since I have not added any python code here is a short how to for future reference. It may be called associative arrays in other languages but Python calls it dicts or dictionaries.

Related links:

Creating a javascript array with one to many type relationship

Multi-Array in Bash

Multidimensional array in python

Manually constructing the dictionary:

mailLists = {}
mailLists['dba'] = ['joe', 'jim', 'jeff', 'john']
mailLists['sa'] = ['mike', 'matt' ]

#print mailLists

#for key in mailLists:
#  print key

print mailLists['dba']
print mailLists['sa']

for k,v in mailLists.iteritems():
  print k + ": " + str(v)

For me the real value in this is for example when you build the lists. Think looping through an input file with lines like this:
jon : dba

Now you add jon to the array with key 'dba'. If the key does not exist we simply add one. If the key exists a new item is added to it.

mailLists = {}

with open('b.txt') as input_file:
  for i, line in enumerate(input_file):
    #print line,
    v,k = line.strip().split(':')
    k = k.strip()
    if k not in mailLists.keys():
      mailLists[k] = []
    mailLists[k].append(v)

for k,v in mailLists.iteritems():
  print k + ": " + str(v)
# python b.py
dba: ['jon ', 'jeff ']
sa: ['matt ']

Comments Off on Dictionaries or Associative Arrays in Python
comments