The Troll-Range » Python

Poor mans offsite backup

Ghworg — Sat, 02 May 2009 07:00:34 +0000

Before I got some proper offsite backup space* I used to make tarballs of files and dump them in my gmail. Not the most elegant solution but it worked and is free (I know about gmailfs, but I could never get it to work reliably).

This is the script I used to send the files:

#!/usr/bin/env python
import getpass
import optparse
import smtplib
import sys

from email import Encoders
from email.MIMEBase import MIMEBase
from email.MIMEMultipart import MIMEMultipart

opts = optparse.OptionParser()
opts.add_option(‘-u’, ‘–username’)
opts.add_option(‘-p’, ‘–password’)
(options, args) = opts.parse_args()
if len(args) < 1:
print "Usage: send_file_to_gmail.py [-u user] [-p pass] filename"
sys.exit(0)

if options.username == None:
options.username = raw_input("Username: ")
if options.password == None:
options.password = getpass.getpass()
if options.username[len(options.username)-10:] != "@gmail.com":
options.username += "@gmail.com"

msg = MIMEMultipart()
msg[‘Subject’] = "Backup of %s" % args[0]
msg[‘From’] = options.username
msg[‘To’] = options.username

msg.preamble = "Backup of %s" % args[0]
msg.epilogue = " "

fp = open(args[0], "rb")
atfile = MIMEBase("application", "octet-stream")
atfile.set_payload(fp.read())
fp.close()
Encoders.encode_base64(atfile)
atfile.add_header("Content-Disposition", "attachment", filename=args[0])
msg.attach(atfile)

s = smtplib.SMTP()
#s.connect("localhost", "8025")
s.connect("smtp.gmail.com")
s.ehlo()
s.starttls()
s.ehlo()
s.login(options.username, options.password)
print "Sending %s" % args[0]
s.sendmail(options.username, options.username, msg.as_string())
s.close()

The above code is made available under the MIT license.

* While I am writing this the Dreamhost backup space I use has been down for about two weeks (it came back briefly a couple of days ago but is back down again now). I can’t complain too much as it is a free extra they throw in with the hosting package but I may have to restart my gmail sending if this keeps up.

Convert files to UTF-8

Ghworg — Thu, 16 Apr 2009 05:00:54 +0000

I occasionally come across text files with weird squiggles or numbers in them were there should be characters. Usually it’s accented characters, but in extreme cases I’ve seen it happen with speech marks.

The problem of course is that the files are not ASCII, and text files don’t store what character set they were created with. So if the set my system happens to use doesn’t match the one the file uses then the result is screwed up letters.

To solve this I created a little python script that converts the file into using the UTF-8 character set as that is nice and universal. You can specify what codec the input is in with the -c option, if you don’t bother then it assumes the Windows 1252 codepage as that is usually what it is in my experience. There is also a force option for when the conversion comes across characters that don’t match the input codec but you want it to convert anyway.

I thought about making it autodetect the codec of the input file, but it is a lot of work for little benefit. The current code works in 99% of cases for me.

#!/usr/bin/env python

import codecs
import optparse
import sys

if len(sys.argv) < 2:
print ‘convert_cp1252_to_UTF-8.py [-f] $filename’
sys.exit(0)

opts = optparse.OptionParser()
# ‘help’, ‘config=’, ‘logfile=’
opts.add_option(‘-f’, ‘–force’, action=‘store_true’)
opts.add_option(‘-c’, ‘–codec’, default=‘cp1252′)
(parsedOpts, args) = opts.parse_args()
filename = args[0]

try:
if parsedOpts.force:
textfile = codecs.open(filename, ‘r’, encoding=parsedOpts.codec, errors=‘ignore’)
else:
textfile = codecs.open(filename, ‘r’, encoding=parsedOpts.codec)
utffile = codecs.open(filename+‘.utf-8′, ‘w’, encoding=‘utf-8′)
utffile.write(textfile.read())
except:
print ‘Error converting to UTF-8, source file probably doesn’t use cp1252‘

The above code is made available under the MIT license.

Build numbers in interpreted code

Ghworg — Thu, 02 Apr 2009 06:00:20 +0000

When I write a program I like to include a build number in the version. Usually I go version.revision.build. The version only ever gets bumped for things that are practically a complete rewrite, and sometimes not even then. The revision gets bumped for new features or major bug fixes and the build gets bumped for every compile.

This gives the huge of advantage of being able to identify the exact time a particular version of a program was created and thus make bug finding easier.

For compiled code this works really well, I just have the build process increment the number somehow. For interpreted code, such as python, however there is no build process. So how do I make sure the build number gets incremented every time? There is no way I would remember to increment it manually.

When I was using subversion I hacked something together using $LastChangedRevision$, so the build number was based on the most recent svn commit number. It worked well enough and I was happy.

Now however I’m switching to git, which doesn’t have keywords like CVS and Subversion so I’m a bit stumped. I tried hacking a script that auto-increments a number then runs a git commit, but not only is it logically flawed, but it relies on me remembering to run it rather than direct git commands which is never going to be reliable.

Maybe I could do something with the current date, but again that would rely on me running something. Probably repo hooks are the answer somehow but I’ve no idea how. The build no. itself needs to be stored in git so as to sync between computers. Can a git repo hook update a file then commit that update back to itself automatically (without an infinite recursion destroying the repo)?

Update (2009-04-05): I’ve worked out how to generate an auto-incrementing number via git hooks. Use the following script as a pre-commit hook (save as .git/hooks/pre-commit). It’s setup for python but it could easily be adapted for whatever language you need.

#!/bin/bash
#
# Called by git-commit with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# This script will automatically increment the __revision__ number in
# the file version.py on every git commit.

f="$GIT_DIR/../version.py"

REVISION=$(grep ‘^__revision__’ $f | sed ‘s/^__revision__ = //’)
#echo "Current revision = $REVISION"
let REVISION=$REVISION+1
#echo "New revision = $REVISION"
sed -r "s/^(__revision__ = )([0-9]+)$/\1$REVISION/" < $f > $f.new
mv $f.new $f
git add $f

Twitter to XMPP (Jabber)

Ghworg — Sat, 21 Mar 2009 21:23:50 +0000

The other day I noticed that irssi (the IRC client I use) had a XMPP plugin. It occurred to me that since I’m on IRC a lot it would be really cool to pull microblog feeds into irssi via this plugin.

For identi.ca this was a doddle, since laconica supports XMPP natively. Twitter however was a problem. I believe twitter used to support XMPP but they pulled that feature for who knows what reason.

There are a couple of apps that will provided a twitter->XMPP gateway for you. I looked at twitterspy, but it sounded like a pain to install. So at first I tried tweet.im, which worked after a fashion. It got the tweets into irssi, but it did it in some weird way so that irssi didn’t realise new messages had come in and hence it didn’t indicate there was activity when I was in a different irssi window.

At this point the little voice in my head that is the bane of my life started saying “You know, you could probably write your own gateway app. How hard could it be?”. Like a fool I listened to this voice, but for once it didn’t turn out disastrously. A few hours later I had a functioning (one-way) twitter->XMPP app, written in python.

I’ve since learned that twitterspy has been rewritten in python and would have been a doddle to install. Not only that but there is a public instance of it I could have used. Still, I’m not immune to NIH syndrome and having my own app to do it is nice. If you want a twitter->XMPP gateway of your own though I suggest you try twitterspy first as it is much more capable (and mature) than my noddy little script.

But if you are interested, and because I like to boast about my tiny little achievements, here is twixm (I’ll give it its own page when I get round to it, but till then it is going to live in this blog post).

#!/usr/bin/python -u
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see .
#

__author__ = "Ghworg"
__version__ = "1.0"
__description__ = "Twitter to XMPP gateway"

import logging, os, signal, sys, threading, time, urllib

import simplejson

try:
from iniparse.compat import ConfigParser
except ImportError:
from ConfigParser import ConfigParser

from pyxmpp.jid import JID
from pyxmpp.jabber.simple import send_message

class WebApiOpener(urllib.FancyURLopener):
"""
Provides a way for HTTP Basic authentication to take place without
prompting the user for a username and password like FancyURLopener
would.
"""
def __init__(self, username, password):
urllib.FancyURLopener.__init__(self,{})
self.username = username
self.password = password

def prompt_user_passwd(self, host, realm):
"""Handle Basic Auth automatically."""
return (self.username, self.password)

class CaseSensitiveConfigParser(ConfigParser):
"""Case Sensitive version of ConfigParser"""
def __init__(self):
ConfigParser.__init__(self)

def optionxform(self, option):
"""Make keys case-sensitive"""
return str(option)

def getAndType(self, section, option):
"""Try to guess variable type and convert to type before returning"""
rawVal = self.get(section, option)
isInt = True
isFloat = True
for c in rawVal:
if not c.isdigit():
isInt = False
if c != ‘.’:
isFloat = False
if isInt:
return int(rawVal)
elif isFloat:
return float(rawVal)
else:
return rawVal

def msgToXMPP(msg):
"""Send a message to Jabber (XMPP)"""
jid = JID(config[‘jid’])
if not jid.resource:
jid = JID(jid.node, jid.domain, ‘send_message’)
recpt = JID(config[‘jid’])
logging.debug(‘Sending to XMPP %s’ % msg)
send_message(jid, config[‘XMPPpass’], recpt, msg)

def getTweets(site, maxTweets=1):
"""Read in a list of tweets made since the last check"""
if site[0] in cache:
url = ‘%s?since_id=%d’ % (site[1], cache[site[0]])
else:
url = ‘%s?count=%d’ % (site[1], maxTweets)

opener = WebApiOpener(site[2], site[3])
try:
logging.debug(‘Requesting %s’ % url)
webPage = opener.open(url)
jsonData = simplejson.load(webPage)
if (‘error’ in jsonData) and (jsonData[‘error’] != ”):
logging.warning(‘Error from server %s: %s’ % (url, jsonData[‘error’]))
jsonData = None
except IOError, e:
logging.error(e)
logging.error(‘URL = %s’ % src)
except ValueError, e:
logging.error(e)
logging.error(‘URL = %s’ % src)

maxID = 0
for tweet in jsonData:
if tweet[u‘id’] > maxID:
maxID = tweet[u‘id’]
msg = ‘%s (%s): %s’ % (tweet[u‘user’][u‘screen_name’], site[0], tweet[u‘text’])
logging.debug(‘Read tweet %s’ % msg)
msgToXMPP(msg)
return maxID

def run():
"""Thread"""
while True:
startSignal.wait()
if quitSignal.isSet():
return
startSignal.clear()

for site in config[‘site’]:
maxID = getTweets(site)
if (site[0] not in cache) or (maxID > cache[site[0]]):
cache[site[0]] = maxID

def readConfig():
"""Load config settings"""
config = {}

cfgfile = os.path.join(os.path.expanduser(‘~’), ‘.twixmrc’)
parser = CaseSensitiveConfigParser()
parser.read(cfgfile)
settings = {}
for section in parser.sections():
if ‘XMPP’ == section:
for optname in parser.options(section):
config[optname] = parser.getAndType(section, optname)
else:
siteName = section
siteUrl = parser.getAndType(section, ‘URL’)
siteUser = parser.getAndType(section, ‘Username’)
sitePass = parser.getAndType(section, ‘Password’)
site = (siteName, siteUrl, siteUser, sitePass)
if ‘site’ not in config:
config[‘site’] = []
config[‘site’].append(site)

config[‘cycleTime’] = 60.0

# !!! Check for missing config settings and bail out

logging.debug(‘XMPP ID = %s’ % config[‘jid’])
logging.debug(‘XMPP Pass = %s’ % config[‘XMPPpass’])
logging.debug(‘Cycle Time = %s’ % config[‘cycleTime’])
for site in config[‘site’]:
logging.debug(‘Site: %s’ % site[0])
logging.debug(‘\tURL: %s’ % site[1])
logging.debug(‘\tUser: %s’ % site[2])
logging.debug(‘\tPass: %s’ % site[3])

return config

def setupLogging():
"""Setup python logging to stdout"""
log = logging.getLogger()
logformat = ‘%(asctime)s %(levelname)s %(message)s’
dateformat = ‘%Y-%m-%d %H:%M:%S’
frmttr = logging.Formatter(logformat, dateformat)
#if options.logfile:
# fhdlr = logging.FileHandler(options.logfile)
# fhdlr.setFormatter(frmttr)
# log.addHandler(fhdlr)
if sys.stdin.isatty():
shdlr = logging.StreamHandler(sys.stdout)
shdlr.setFormatter(frmttr)
log.addHandler(shdlr)
#log.setLevel(logging.DEBUG)
log.setLevel(logging.INFO)

def handler(signum, frame):
"""Handle various termination signals gracefully"""
logging.warning(‘Signal handler called with signal %d’ % signum)
quitSignal.set()
startSignal.set()

###############################################################################
# Main
###############################################################################

# Set the signal handlers
signal.signal(signal.SIGTERM, handler)
signal.signal(signal.SIGQUIT, handler)
signal.signal(signal.SIGINT, handler)

setupLogging()

config = readConfig()
cache = {}

quitSignal = threading.Event()
startSignal = threading.Event()

runThread = threading.Thread(target=run)
runThread.start()

while False == quitSignal.isSet():
startSignal.set()
count = 0.0
while count < config[‘cycleTime’]:
quitSignal.wait(0.5)
count += 0.5
if quitSignal.isSet():
break
logging.info(‘Exiting’)

Update 2009-11-25: I have created a page for twixm now. Go there for the latest code and info.