Category: Python

Sep 23

Quick Python Mailer

import sys,smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

msg = MIMEMultipart()
msg['From'] = 'root@domain.ad'
msg['To'] = 'user@domain.com'
msg['Subject'] = 'Process finished -' + sys.argv[1]
message = 'here is the email'
msg.attach(MIMEText(message))

mailserver = smtplib.SMTP('mail.domain.com',25)
# identify ourselves to smtp gmail client
mailserver.ehlo()
# secure our email with tls encryption
mailserver.starttls()
# re-identify ourselves as an encrypted connection
mailserver.ehlo()

mailserver.sendmail('root@domain.ad','user@domain.com',msg.as_string())

mailserver.quit()

** This script need to be optimized. I am duplicating from and to fields

Comments Off on Quick Python Mailer
comments

Feb 12

Python Split Using Space vs None Separator

Quick reminder of how to split a "ls -l" listing. It is problematic because the last field which is the directory or file name can have spaces itself, so splitting on spaces which is the only option here still need a bit of additional work.

import config
import subprocess

## First experiment doing manual manipulation of the fields.  
## Basically split on all spaces and then assemble the file name.
def dir_listing():
  ls_lines = subprocess.check_output(['ls', '-l']).splitlines()
  ls_arr= []
  for item in ls_lines:
    if not "total" in item:
      f = item.split()
      ## fname = item.join(str(v) for v in item.index if v > 7)
      fname = ""
      for idx,val in enumerate(f):
        if idx > 7:
          fname = fname + str(val) + " "
      fname = fname[:-1]
      ls_fields = [f[0],f[1],f[2],f[3],f[4],f[5]+"-"+f[6]+"-"+f[7],fname]
      ls_arr.append(ls_fields)
  return ls_arr

## Second attempt is split on spaces with a max split defined. 
## This sounds like the obvious way to do this but I needed to use "None" and not " "
## as shown below.
def dir_listing_optimized():
  ls_lines = subprocess.check_output(['ls', '-l']).splitlines()
  ls_arr= []
  for item in ls_lines:
    if not "total" in item:
      ## None will use arbitrary strings of whitespace characters (space, tab, newline, return, formfeed)
      ## When using split(" ",8) I had more separation than I expected and therefor my last field 
      ## was not working right.
      f = item.split(None,8)
      ls_arr.append(f)
  return ls_arr

for dir in dir_listing():
  print dir

for dir in dir_listing_optimized():
  print dir

Comments Off on Python Split Using Space vs None Separator
comments

Feb 10

webpy Example

A short write up on how to bootstrap a webpy app and at the same time show how to make a system call with popen.  I based this short app on the skeleton example here: http://webpy.org/skeleton/0.3

Just a note on where my test live for my own reference later.

$ pwd
/DATA/src/hvInfo

$ ls
call1.py call.pyc config.pyc hvInfo.pyc my test with spaces_ and more... templates view.pyc web.py-0.37
call.py config.py hvInfo.py index.html simpleHTTP.py view.py web web.py-0.37.tar

Main python app:

hvInfo.py

import web
import view, config
from view import render

urls = (
  '/', 'index'
)

class index:
  def GET(self):
    return render.base(view.listing())

if __name__ == "__main__": 
  app = web.application(urls, globals())
  app.run()

Define some functions here:

call.py

import config
#from subprocess import Popen, PIPE
import subprocess

def listing_experiment1():
    cmd = "ls -l ~/"
    p = Popen(cmd , shell=True, stdout=PIPE, stderr=PIPE)
    #out, err = p.communicate()
    #print "Return code: ", p.returncode
    #return out.rstrip(), err.rstrip()
    outputlines = filter(lambda x:len(x)>0,(line.strip() for line in p.stdout))
    return outputlines
    #return out.replace('\n','<br>')

    ##return config.DB.select('items', **k)

def dir_listing():
  ## http://stackoverflow.com/questions/8880461/python-subprocess-output-to-list-or-file
  ls_lines = subprocess.check_output(['ls', '-l']).splitlines()
  ## I want to pass output as separate fields but space as a delimiter won't do the trick since filenames can have spaces...
  ls_arr= []
  for item in ls_lines:
    if not "total" in item:
      f = item.split()
      #fname = item.join(str(v) for v in item.index if v > 7)
      fname = ""
      for idx,val in enumerate(f):
        if idx > 7:
          fname = fname + str(val) + " "
      fname = fname[:-1]
      ls_fields = [f[0],f[1],f[2],f[3],f[4],f[5]+"-"+f[6]+"-"+f[7],fname]
      ls_arr.append(ls_fields)
  return ls_arr

Lets maintain a config file. Especially when DB access becomes necessary.

config.py

import web
#DB = web.database(dbn='postgres', db='appname', user='username', pw='')
cache = False

Following MVC type framework.  It makes sense to keep presentation separate.

view.py

import web
import call
import config

t_globals = dict(
  datestr=web.datestr,
)
render = web.template.render('templates/', cache=config.cache, 
    globals=t_globals)
render._keywords['globals']['render'] = render

def listing():
    l = call.dir_listing()
    return render.listing(l)

Templates as follow:

$ cat templates/base.html
$def with (page, title=None)
<html><head>
<title>hvInfo\
$if title: : $title\
</title>
</head><body>
<h1><a href="/">hvInfo</a></h1>
$:page   
</body></html>
$ cat templates/listing.html
$def with (items)
  <table>
  $for item in items:
    <tr><td>$item[0]<td>$item[1]<td>$item[2]<td>$item[3]<td>$item[4]<td>$item[5]<td><b>$item[6]</b>
  </table>
$ cat templates/item.html
$def with (item)

$item

Run web server as follow:

$ python hvInfo.py
http://0.0.0.0:8080/

Test in browser at this link: http://localhost:8080/

Comments Off on webpy Example
comments

Dec 30

Sanitizing Email Recipient List

I wrote a couple articles on respectively using sendmail and postfix to block outbound email and only allow selective domains and selective email addresses.  I ran into problems with both sendmail and postfix doing exactly what I want.  Sendmail was fairly easy to satisfy the requirements for forwarding only to certain domains. Postfix got very close on doing domains plus a selective few email addresses.  I could not get the multi instance postfix method working 100% with my header checks though.

Below is another method to sanitize outbound email addresses using python smtplib and sendmail.

Sendmail answer on non standard port:

root@myhost:/etc/mail/cf/cf# more usla-utility.mc
divert(-1)
...
divert(0)dnl
VERSIONID(`sendmail.mc (Sun)')
OSTYPE(`solaris11')dnl
DOMAIN(`solaris-generic')dnl
DAEMON_OPTIONS(`Port=10026,Addr=127.0.0.1, Name=MTA')dnl
MASQUERADE_AS(`arbonne.com')
FEATURE(masquerade_envelope)
FEATURE(`mailertable')
MAILER(`local')dnl<br />

root@myhost:/etc/mail/cf/cf# /usr/ccs/bin/m4 ../m4/cf.m4 myhost.mc > /etc/mail/sendmail.cf

I needed some specific transports so I have a mailertable:

root@myhost:/etc/mail# makemap hash mailertable < mailertable

Run a small python smtplib SMTP forwarder on port 25:

More information here: http://docs.python.org/2/library/smtplib.html#smtp-example

Run python in the background:

root@myhost:# nohup python sanitizer.py &

My code snippet to sanitize. Probably need to look at CC and BCC also.

...
allowedDomains = ['domain1.com','domain2.com','domain3.com','domain4.com']
      allowedRecipients = ['user@domain5.com']
      for rcpt in rcpttos[:]:
        user,domain = rcpt.split("@")
        if not (rcpt.lower() in allowedRecipients or domain.lower() in allowedDomains):
          #log.debug("%s not allowed per our policy" % rcpt)
          i = rcpttos[:].index(rcpt)
          del rcpttos[i]
      log.debug("sanitized list %s" % rcpttos)
...

Monitor the custom log:

root@myhost:~# tail -f sanitizer.log 
... DEBUG:__main__:Received message from: ('10.1.11.86', 32841). From: root@myclient | To: ['user1@domain1.com', 'user2@domain2.com', 'user3@domain3.com', 'user4@domain1.com'] DEBUG:__main__:sanitized list ['user3@domain3.com']

Inject a test from client:

root@myclient:/tmp# cat /tmp/test_all.eml
To: user1@domain1.com,user2@domain2.com,user3@domain3.com,user4@domain1.com 
Subject: MAILHOST TEST DL
From: luser@mydomain.com
body...

root@myclient:/tmp# sendmail -d7.99 -d38.99 -vt < /tmp/test_all.eml

Monitor the real maillog of sendmail to see what happened:

Comments Off on Sanitizing Email Recipient List
comments

Apr 06

Python Manipulating XML

A short script with descriptions for reading and manipulating xml.  It seems like the python ElementTree module should be the easiest and best suited for XML manipulation.  However I had a complicated XML structure with multiple namespaces and lxml handled it better.  ElementTree could only handle one namespace with it;s register function. In addition lxml has pretty_print which might be useful.  Although in my case when I do inserts pretty_print did not work even with the FAX fix for remove_blank_text.

import lxml.etree as ET

f = open('config.xml','rb')
## http://lxml.de/FAQ.html#why-doesn-t-the-pretty-print-option-reformat-my-xml-output
#parser = ET.XMLParser(remove_blank_text=True)
#tree = ET.parse(f, parser)
tree = ET.parse(f)

#for element in tree.iter():
#    element.tail = None

root = tree.getroot()
namespace="http://xmlns.oracle.com/weblogic/domain"
servers = tree.findall('.//{%s}server' % namespace)

## Loop through the nodes we found
for server in servers:
  print "New SERVER node detected:"
  for child in server:
    tag = child.tag
    val = child.text
    ## Remove any existing children
    if tag == "{http://xmlns.oracle.com/weblogic/domain}ssl":
      print "found server.ssl and will remove",
      server.remove(child)
    if tag == "{http://xmlns.oracle.com/weblogic/domain}log":
      print "found server.log and will remove",
      server.remove(child)
    if tag == "{http://xmlns.oracle.com/weblogic/domain}data-source":
      print "found server.data-source and will remove",
      server.remove(child)
    print tag, val
  
  ## Add the 3 children we want 
  child = ET.Element("ssl")
  child.text=''
  server.insert(1,child)
  ##  Check out why xsi:nil is not working. UTF???
  ##  gchild = ET.Element("hostname-verifier",attrib={'xsi:nil':'true'})
  gchild = ET.Element("hostname-verifier",attrib={'xsi_nil':'true'})
  gchild.text=''
  child.insert(1,gchild)
  gchild = ET.Element("hostname-verification-ignored")
  gchild.text='true'
  child.insert(2,gchild)
  gchild = ET.Element("client-certificate-enforced")
  gchild.text='true'
  child.insert(3,gchild)
  gchild = ET.Element("two-way-ssl-enabled")
  gchild.text='false'
  child.insert(3,gchild)
  
  child = ET.Element("log")
  child.text=''
  server.insert(2,child)
  gchild = ET.Element("rotation-type")
  gchild.text='byTime'
  child.insert(1,gchild)
  gchild = ET.Element("number-of-files-limited")
  gchild.text='true'
  child.insert(2,gchild)
  gchild = ET.Element("rotate-log-on-startup")
  gchild.text='true'
  child.insert(3,gchild)
  
  child = ET.Element("data-source")
  child.text=''
  server.insert(3,child)
  gchild = ET.Element("data-source-log-file")
  gchild.text=''
  child.insert(1,gchild)
  ggchild = ET.Element("rotation-type")
  ggchild.text='byTime'
  gchild.insert(1,ggchild)
  ggchild = ET.Element("number-of-files-limited")
  ggchild.text='true'
  gchild.insert(2,ggchild)
  ggchild = ET.Element("rotate-log-on-startup")
  ggchild.text='true'
  gchild.insert(3,ggchild)

## Check out why pretty_print is not making newlines in new tags  
#print(ET.tostring(tree, pretty_print=True))
tree.write("wc-out.xml", pretty_print=True)

Comments Off on Python Manipulating XML
comments

Nov 26

Unix Find and Remove Python Style

No doubt the Unix find and remove command comes in very useful when cleaning up large folders.  However find can quickly bump into the old "/usr/bin/find: Argument list too long" problem.

For reference here is a command that works well.  Except of course when too many files or directories involved.

find /usr/local/forms/*/output -name "*.html" -mtime +4 -exec rm {} \;

There is of course other ways to get this done with find, but I like python so I resorted to python as the example show below.

Here is an example that worked for me:


#!/usr/bin/env python
##  Adjust the humandays variable.  Set to 2000 days until we feel more comfortable.

import os
import glob
import time
import shutil

humandays = 2000
computerdays = 86400*humandays
now = time.time()
inputDirs = glob.glob('/usr/local/forms/*/input')

print "Script running on %s " % time.ctime(now)
print "using physical path /usr/local/forms/*/input and only removing directories older than %s days." % (humandays)

for inputDir in inputDirs:
  for r,d,f in os.walk(inputDir):
    for dir in d:
         timestamp = os.path.getmtime(os.path.join(r,dir))
         if now-computerdays > timestamp:
             try:
		  print "modified: %s" % time.ctime(timestamp),
		  removeDir=os.path.join(r,dir)
                  print "  remove ",removeDir,
                  ## Better be 100% sure before you uncomment the rmtree line!
                  ## shutil.rmtree(removeDir)
                  ## Better be 100% sure before you uncomment the rmtree line!
             except Exception,e:
                  print e
                  pass
             else:
                  print " -> success"

Comments Off on Unix Find and Remove Python Style
comments

Nov 06

Python Inject SMTP

Sometimes I want to email from Python and this is pretty easy. Or even just testing mail flow I prefer this over good old "telnet host 25" since we have well constructed headers with smtplib.

#!/usr/bin/python
import datetime
import os
import sys
import smtplib

def warnEmail(SMTPserver, Subject, Body):
 fromaddr = "rrosso@domain.com"
 #toaddrs = ["rcpt1@domain.com","rcpt2@domain.com"]
 toaddrs = ["hostmaster@domain.com"]

 msg = ("From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (fromaddr, ", ".join(toaddrs),Subject))
 
 server = smtplib.SMTP(SMTPserver)
 msg = msg + Body
 #server.set_debuglevel(1)
 server.sendmail(fromaddr, toaddrs, msg)
 server.quit()

SMTPserver =  'mailhost'
warnEmail(SMTPserver, "Test through " + SMTPserver , "Body test goes \n here!")

Comments Off on Python Inject SMTP
comments

Nov 06

Python-ldap Query MS Active Directory

I use Python to pull Active Directory information sometimes. In my case mainly to report or view information but also to create files in a LDIF or PowerShell format.  These can be manually run on the Domain Controller later. For instance find all users in a Distribution List or Group and create a rule or ldif entry that can manually be executed line by line. Off course there is also ways with PowerShell and vbscript to do this, but I prefer Python for text manipulation and it is not too cumbersome for me to batch run these files manually later.

I noticed on Ubuntu 12.10 that my query failed with the following error:

ldap.LDAP_CONTROL_PAGE_OID,True,(page_size,'')
AttributeError: 'module' object has no attribute 'LDAP_CONTROL_PAGE_OID'

I found a comment from the developers saying that with python-ldap 2.4
"there have been changes in the API for LDAPv3 extended controls. Please see Demo/page_control.py (and Demo/paged_search_ext_s.py) how to use the simple paged control with 2.4."

I found the source here: http://python-ldap.cvs.sourceforge.net/viewvc/python-ldap/python-ldap/Demo/page_control.py?view=log

I made a new script and tested as shown below:
** Note the following first:

  1. This is a "paged" query meaning you should not have issues if LDAP return a limted number of results.  More detail here: http://support.microsoft.com/kb/315071
  2. You need to install python-ldap of course.  apt-get install python-ldap should work on apt systems.
  3. Check comment about using 512 as userAccountControl.
DistributionList = "CN=IT Infrastructure,CN=Distribution Lists,CN=Users,DC=domain,DC=com"

url = "ldap://usdc101"
base = "dc=domain,dc=com"
#  search_flt = r'(objectClass=*)'
##  I used userAccountControl=512 but that would most certainly exclude some 
##  user accounts in your domain. For instance "66048 Enabled, Password Doesn't Expire"
##  values listed here: http://www.netvision.com/ad_useraccountcontrol.php
search_flt = r'(&(objectCategory=user) (userAccountControl=512) )'

page_size = 10

import ldap,pprint
from ldap.controls import SimplePagedResultsControl

searchreq_attrlist=["displayName","cn","distinguishedName","mail","memberOf"]

ldap.set_option(ldap.OPT_REFERRALS, 0)
l = ldap.initialize(url,trace_level=0)
l.protocol_version = 3
l.simple_bind_s("ADaccount@domain.com", "passsword")

req_ctrl = SimplePagedResultsControl(True,size=page_size,cookie='')

known_ldap_resp_ctrls = {
SimplePagedResultsControl.controlType:SimplePagedResultsControl,
}

# Send search request
msgid = l.search_ext(
base,
ldap.SCOPE_SUBTREE,
search_flt,
attrlist=searchreq_attrlist,
serverctrls=[req_ctrl]
)

pages = 0
i = 0
print "listing users in the list:" + DistributionList

while True:
  pages += 1
  rtype, rdata, rmsgid, serverctrls = l.result3(msgid,resp_ctrl_classes=known_ldap_resp_ctrls)

  for dn, entry in rdata:
    ##  Lets check if the user is a member of the AD List / Group
    try:
      membership = entry['memberOf']
    except:
      membership = 'none'
    if DistributionList in membership:
      i += 1
      print " \"%d\" | \"%s\" " % (i , dn)

  pctrls = [
    c
    for c in serverctrls
    if c.controlType == SimplePagedResultsControl.controlType
  ]
  if pctrls:
    if pctrls[0].cookie:
      # Copy cookie from response control to request control
      req_ctrl.cookie = pctrls[0].cookie
      msgid = l.search_ext(
        base,
        ldap.SCOPE_SUBTREE,
        search_flt,
        attrlist=searchreq_attrlist,
        serverctrls=[req_ctrl]
      )
    else:
      break
  else:
    print "Warning: Server ignores RFC 2696 control."
    break

l.unbind_s()

Comments Off on Python-ldap Query MS Active Directory
comments

Nov 06

Python Simple Sort

Quick sort routine...

def sort(array):
  if len(array) <= 1: return array
  mid = len(array) // 2
  return merge (sort(array[0:mid]), sort(array[mid:]))

# this may not be the most thoroughly idiomatic python, or the
# most efficient merge (it duplicates data when "Transmitting")
# but it works
def merge(left, right):
    merged = []

    i = 0
    j = 0
    while(len(merged) < len(left)+len(right)):
        if left[i] < right[j]:
            merged.append(left[i])
            i += 1
            if i == len(left):
                # Knuth, TaoCP Vol 3 5.2.4 Calls this the "transmit"       
                y = right[j:]
                for x in y:
                    merged.append(x)
                break
        else:
            merged.append(right[j])
            j += 1
            if j == len(right):
                y = left[i:]
                for x in y:
                    merged.append(x)
                break

    return merged

a=[1,3,2,4]
b=sort(a)
print b

Comments Off on Python Simple Sort
comments

Nov 06

Python Nodes In A List

Quick and simple Python list with node insert capabilty.

class Node:
    def __init__(self,value):
        self.data = value
        self.next = 0

class List:
    def __init__(self):
        self.firstNode = Node(0)
    def __ShowNodeData(self,aNode):
        if aNode.next != 0:
           print aNode.data
           self.__ShowNodeData(aNode.next)
    def Dump(self):
        self.__ShowNodeData(self.firstNode)
    def InsertAfter(self,aNode,aNewNode):
        aNewNode.next = aNode.next
        aNode.next = aNewNode
    def InsertBeginning(self,aNewNode):
        aNewNode.next = self.firstNode
        self.firstNode = aNewNode   

nodeA = Node("A")
nodeB = Node("B")
nodeC = Node("C")
nodeD = Node("D")

aList = List()

aList.InsertBeginning(nodeB)
aList.InsertAfter(nodeB,nodeD)
aList.InsertAfter(nodeD,nodeC)
aList.InsertAfter(nodeC,nodeA)
 
aList.Dump()

Comments Off on Python Nodes In A List
comments