code.random()

Python: display refreshing status (like top)

philipp.keller — Sun, 04 Apr 2010 21:24:50 +0000

In scripts I often want to display some status information (e.g. progress), this can be achieved with e.g.:

PYTHON:

print "progress: %i %%\r" % i,
sys.stdout.flush()

but this just works for one-liners. I wanted to have what top or less do: open a new "window" and being able to write everywhere in the window, not just on the last line.

Found out that curses does exactly that - but I didn't find a stripped-to-the-bare-necessities-example, so here you are:

PLAIN TEXT

PYTHON:

import curses, time
from datetime import datetime
w = curses.initscr()
try:
while True:
w.erase()
w.addstr("some status..\ncurrent time\n%s" % datetime.now())
w.refresh()
time.sleep(1)
finally:
curses.endwin()

Python: Print list of dicts as ascii table

philipp.keller — Sat, 27 Feb 2010 10:55:35 +0000

Sometimes I want to print list of dicts as an ascii table, like this:

| Programming Language | Language Type | Years of Experience |
+----------------------+---------------+---------------------+
| python               | script        |                   4 |
| php                  | script        |                   5 |
| java                 | compiled      |                  11 |
| assember             | compiled      |                  15 |

I searched on Google - but without luck.

That's what I came up with - it's not particularly nice but it does the job:

PLAIN TEXT

PYTHON:

def table_print(data, title_row):
"""data: list of dicts,
title_row: e.g. [('name', 'Programming Language'), ('type', 'Language Type')]
"""
max_widths = {}
data_copy = [dict(title_row)] + list(data)
for col in data_copy[0].keys():
max_widths[col] = max([len(str(row[col])) for row in data_copy])
cols_order = [tup[0] for tup in title_row]
def custom_just(col, value):
if type(value) == int:
return str(value).rjust(max_widths[col])
else:
return value.ljust(max_widths[col])
for row in data_copy:
row_str = " | ".join([custom_just(col, row[col]) for col in cols_order])
print "| %s |" % row_str
if data_copy.index(row) == 0:
underline = "-+-".join(['-' * max_widths[col] for col in cols_order])
print '+-%s-+' % underline

Use it like that:

PLAIN TEXT

PYTHON:

data = [dict(name='python', type='script', years_experience=4),
dict(name='php', type='script', years_experience=5),
dict(name='java', type='compiled', years_experience=11),
dict(name='assember', type='compiled', years_experience=15)
]
titles = [('name', 'Programming Language'),
('type', 'Language Type'),
('years_experience', 'Years of Experience')]
table_print(data, titles)

It will produce the table printed above. It's not fancy - the only 'smart' thing it does is right-adjusting integers, strings are left-adjusted.

P.S. no, I don't have 15 years of experience of Assembler - I just know it since 15 years - it's one of the first programming languages I learned - and I even wrote a text editor with it - then I learned that's probably not the best language to write an editor

dealing with “MySQL backend does not support timezone-aware datetimes”

philipp.keller — Wed, 30 Dec 2009 22:27:00 +0000

When fetching events from an iCal feed and saving this into a database I got

MySQL backend does not support timezone-aware datetimes

This did the trick for me:

Install pytz (download here)

PLAIN TEXT

PYTHON:

import pytz
that_datetime_in_utc.astimezone(pytz.timezone('Europe/Zurich')).replace(tzinfo=None)

Caution: I don't really understand what I'm writing here - it feels like those posters in PHP forums who explain 'how to ...' and then trying to explain something they have no clue of, but well..

Fetch publicly available google calendar data with python

philipp.keller — Tue, 29 Dec 2009 20:31:18 +0000

I tried accessing the google data api with python - that api seems either overly complicated or just not suited for just grabbing events from a public google calendar.

This worked for me:

find out ical address (subscribe to the calendar, other calendars->arrow down->calendar settings)
install icalendar

Then this is possible:

PLAIN TEXT

PYTHON:

from icalendar import Calendar
import urllib
ics = urllib.urlopen('http://www.google.com/calendar/ical/fchppllvcaupb6fgguigobkfj4@group.calendar.google.com/public/basic.ics').read()
ical=Calendar.from_string(ics)
for vevent in ical.subcomponents:
if vevent.name != "VEVENT":
continue
title = str(vevent.get('SUMMARY'))
description = str(vevent.get('DESCRIPTION'))
location = str(vevent.get('LOCATION'))
start = vevent.get('DTSTART').dt # a datetime
end = vevent.get('DTEND').dt # a datetime

Send javascript errors by mail

philipp.keller — Sat, 06 Oct 2007 12:33:42 +0000

I'm running a Django-powered site for a closed user group and added a bit of JavaScript magic here and there (mainly Prototype and Tooltip).

Now Django sends me a mail whenever a 404 or 500 error occurs. But when one of my users encounters a JavaScript-Error, I'm not informed. I thought anyone in the web has solved this problem but didn't find anything, so here's my take: Just send any error using Ajax (here: using Prototypes Ajax abstraction) to the server

PLAIN TEXT

JAVASCRIPT:

onerror = Extranet.mailError;
function mailError(msg, url, line) {
var postBody = 'url=' + url + '&line=' + line + '&message=' + escape(msg) + '&useragent=' + escape(navigator.userAgent) + '&user=' + escape(user_name);
var myAjax = new Ajax.Request('/api/jserror/', {method: 'post', postBody: postBody});
}

user_name is a JavaScript variable holding the Django username (so I know whom I can inform when the error is fixed).

On the server side, I just send me mails containing the JavaScript error message, the username and the user agent:

PLAIN TEXT

PYTHON:

def jserror(request):
from django.core.mail import mail_admins
omit_messages = ['pointerobj is not defined', 'tipobj is not defined', 'ns6 is not defined', 'enabletip is not defined']
if request.POST.get('message', '') not in omit_messages:
message = """url: %s (%s)
%s
user-agent: %s
username: %s
""" % (request.POST.get('url', ''), request.POST.get('line', ''), request.POST.get('message', ''), request.POST.get('useragent', ''), request.POST.get('user', ''))
mail_admins("javascript error", urldecode(message))
return HttpResponse()

Yeah, that's all very trivial but I wonder what other solutions exist for this problem...

Django: Serve big files via fcgid

philipp.keller — Wed, 03 Oct 2007 07:14:28 +0000

I've got a django project running which requires you to login to access files.
That means that I have to serve the files via python, like this:

PLAIN TEXT

PYTHON:

@login_required
def download(request, filename):
# ... some code specific to my site ...
response = HttpResponse(mimetype=postUpload.mimetype)
response['Content-Disposition'] = "attachment; filename=" + original_filename
response['Content-Length'] = os.path.getsize(filename_path)
response.write(open(filename_path).read())
return response

The problem: If the download of a file exceeded 5 minutes (big files and/or low bandwidth) the download was canceled on the server side by a timeout. This Apache configuration for mod_fcgid solved the problem (see mod_fcgid documentation for BusyTimeout)

PLAIN TEXT

CODE:

c>
BusyTimeout 1200

The problem was that the apache module scanned every minute for processes that run for more than BusyTimeout seconds. These processes are potentially in bad health (infinite loop et al.) and have to be killed. Not so with my processes (since I know what I'm doing..). The setting of the busy timeout to 1200 seconds now lets my processes run for a maximum of one hour.

As this setting can't by overwritten in a htaccess file by default I needed to bug my web hosting provider with the request, which was handled in 24 hours, so thanks for that one!

PS: If you know of another way how to serve protected static files via a single sign on (no HTTP basic auth), please let me know.

Python: Sort a list of dicts by dict-key

philipp.keller — Mon, 23 Jul 2007 13:52:29 +0000

I always forget that one and end up searching for half an hour:

PLAIN TEXT

PYTHON:

>>> list = [ dict(a=1,b=2,c=3),
... dict(a=2,b=2,c=2),
... dict(a=3,b=2,c=1)]
>>> list.sort(key=operator.itemgetter('c'))
>>> list
[{'a': 3, 'c': 1, 'b': 2}, {'a': 2, 'c': 2, 'b': 2}, {'a': 1, 'c': 3, 'b': 2}]
>>> list.sort(key=operator.itemgetter('a'))
>>> list
[{'a': 1, 'c': 3, 'b': 2}, {'a': 2, 'c': 2, 'b': 2}, {'a': 3, 'c': 1, 'b': 2}]

Python: Find out cpu time of a certain process

philipp.keller — Fri, 08 Jun 2007 09:52:52 +0000

To find out how many percentage a certain process uses the cpu:

PLAIN TEXT

PYTHON:

import os, time
# find out the pid by username.
# "-o pid h" omits the header and just prints the pid
pid = os.popen('ps -U my_user_name -o pid h').read().strip()
# 14th column is utime, 15th column is stime:
# The time the process has been scheduled in user/kernel mode
# The time value is in jiffies. One jiffie is appox 1/100 second
# see man proc for more info
stat = os.popen('cat /proc/%s/stat' % pid).read().strip()
cpu_time1=int(stat.split()[14]) + int(stat.split()[15])
time1=time.time()
time.sleep(1)
stat = os.popen('cat /proc/%s/stat' % pid).read().strip()
cpu_time2=int(stat.split()[14]) + int(stat.split()[15])
time2=time.time()
print str(float(cpu_time2 - cpu_time1) / (time2 - time1)) + "%"

I don't know though if the number is accurate.
What is "cpu time" anyway? It's the time the process is running (using the cpu for 100%) divided by the time the process is laid asleep by the scheduler.
Then, jiffies seem to be not a safe number for time measurements.
But for relative measurements it should do the trick.

add bandwidth to a file download in python

philipp.keller — Sun, 25 Feb 2007 01:51:05 +0000

Another short code snippet I came up when writing the module download script

The following class keeps a download under a certain ratio and prints live download stats (KB/s).

PLAIN TEXT

PYTHON:

class BandwidthFile:
"""
limits the file download to a certain amount of bytes per second and prints download stats
"""
def __init__(self, filename, filesize, bytes_per_second = 20000, mode = "wb"):
self.fp = open(filename, "wb")
self.bytecounter = 0
self.started = datetime.datetime.now()
self.bytes_per_second = bytes_per_second
self.filesize = filesize
self.filename = filename
def __del__(self):
self.fp.close()
def write(self, bytestring):
self.bytecounter += len(bytestring)
seconds_since_started = (datetime.datetime.now() - self.started).seconds + 1
while (self.bytecounter / seconds_since_started)> self.bytes_per_second:
seconds_since_started = (datetime.datetime.now() - self.started).seconds + 1
time.sleep(0.1)
self.fp.write(bytestring)
sys.stdout.write("\rdownloading %s (%i/%i KB, %i KB/s)" % (self.filename, (self.bytecounter / 1024), (int(self.filesize) / 1024), (self.bytecounter /seconds_since_started / 1024)))
sys.stdout.flush()

To use the class with ftp:

PLAIN TEXT

PYTHON:

from ftplib import FTP
ftp = FTP("ftp.modland.com")
filesize = ftp.size(remote_filename)
f = BandwidthFile(local_filename, ftp_filesize)
ftp.retrbinary('RETR '+remote_filename, f.write)

The status line looks like this:

downloading /home/phred/modules/Skaven/beyond the network.it (95/4411 KB, 19 KB/s)

If you have suggestions how to enhance the code snippet feel free to leave a comment!

set timeout for a shell command in python

philipp.keller — Sun, 18 Feb 2007 23:04:06 +0000

I wanted to run a shell command in python without knowing if the shell command is going to exit within reasonable time (adplay that was, sometimes it simply hangs).

Update: the "task" module of Rob Hooft seems to solve this exact problem. At the time I wrote this, the python.net website was down. I leave my solution here just for archive purpose.

PLAIN TEXT

PYTHON:

def timeout_command(command, timeout):
"""call shell-command and either return its output or kill it
if it doesn't normally exit within timeout seconds and return None"""
import subprocess, datetime, os, time, signal
start = datetime.datetime.now()
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while process.poll() is None:
time.sleep(0.1)
now = datetime.datetime.now()
if (now - start).seconds> timeout:
os.kill(process.pid, signal.SIGKILL)
os.waitpid(-1, os.WNOHANG)
return None
return process.stdout.read()

PLAIN TEXT

PYTHON:

>>> output = timeout_command(["sleep", "10"], 2)
None
>>> output = timeout_command(["sleep", "1"], 2)

The process can be killed when it has run for too long (the os.waitpid waits for the kill to end and avoids defunct-processes) and furthermore the Popen'ed process' printed is caught and returned if it doesn't timeout. However, subprocess.Popen is called with a list as argument. That means, that the command isn't passed to a shell and furthermore you can just call one command with options, nothing more.