code.random() » python

Python: Print list of dicts as ascii table

philipp.keller — Sat, 27 Feb 2010 10:55:35 +0000

Sometimes I want to print list of dicts as an ascii table, like this:

| Programming Language | Language Type | Years of Experience |
+----------------------+---------------+---------------------+
| python               | script        |                   4 |
| php                  | script        |                   5 |
| java                 | compiled      |                  11 |
| assember             | compiled      |                  15 |

I searched on Google - but without luck.

That's what I came up with - it's not particularly nice but it does the job:

PLAIN TEXT

PYTHON:

def table_print(data, title_row):
"""data: list of dicts,
title_row: e.g. [('name', 'Programming Language'), ('type', 'Language Type')]
"""
max_widths = {}
data_copy = [dict(title_row)] + list(data)
for col in data_copy[0].keys():
max_widths[col] = max([len(str(row[col])) for row in data_copy])
cols_order = [tup[0] for tup in title_row]
def custom_just(col, value):
if type(value) == int:
return str(value).rjust(max_widths[col])
else:
return value.ljust(max_widths[col])
for row in data_copy:
row_str = " | ".join([custom_just(col, row[col]) for col in cols_order])
print "| %s |" % row_str
if data_copy.index(row) == 0:
underline = "-+-".join(['-' * max_widths[col] for col in cols_order])
print '+-%s-+' % underline

Use it like that:

PLAIN TEXT

PYTHON:

data = [dict(name='python', type='script', years_experience=4),
dict(name='php', type='script', years_experience=5),
dict(name='java', type='compiled', years_experience=11),
dict(name='assember', type='compiled', years_experience=15)
]
titles = [('name', 'Programming Language'),
('type', 'Language Type'),
('years_experience', 'Years of Experience')]
table_print(data, titles)

It will produce the table printed above. It's not fancy - the only 'smart' thing it does is right-adjusting integers, strings are left-adjusted.

P.S. no, I don't have 15 years of experience of Assembler - I just know it since 15 years - it's one of the first programming languages I learned - and I even wrote a text editor with it - then I learned that's probably not the best language to write an editor

dealing with “MySQL backend does not support timezone-aware datetimes”

philipp.keller — Wed, 30 Dec 2009 22:27:00 +0000

When fetching events from an iCal feed and saving this into a database I got

MySQL backend does not support timezone-aware datetimes

This did the trick for me:

Install pytz (download here)

PLAIN TEXT

PYTHON:

import pytz
that_datetime_in_utc.astimezone(pytz.timezone('Europe/Zurich')).replace(tzinfo=None)

Caution: I don't really understand what I'm writing here - it feels like those posters in PHP forums who explain 'how to ...' and then trying to explain something they have no clue of, but well..

Fetch publicly available google calendar data with python

philipp.keller — Tue, 29 Dec 2009 20:31:18 +0000

I tried accessing the google data api with python - that api seems either overly complicated or just not suited for just grabbing events from a public google calendar.

This worked for me:

find out ical address (subscribe to the calendar, other calendars->arrow down->calendar settings)
install icalendar

Then this is possible:

PLAIN TEXT

PYTHON:

from icalendar import Calendar
import urllib
ics = urllib.urlopen('http://www.google.com/calendar/ical/fchppllvcaupb6fgguigobkfj4@group.calendar.google.com/public/basic.ics').read()
ical=Calendar.from_string(ics)
for vevent in ical.subcomponents:
if vevent.name != "VEVENT":
continue
title = str(vevent.get('SUMMARY'))
description = str(vevent.get('DESCRIPTION'))
location = str(vevent.get('LOCATION'))
start = vevent.get('DTSTART').dt # a datetime
end = vevent.get('DTEND').dt # a datetime

Send javascript errors by mail

philipp.keller — Sat, 06 Oct 2007 12:33:42 +0000

I'm running a Django-powered site for a closed user group and added a bit of JavaScript magic here and there (mainly Prototype and Tooltip).

Now Django sends me a mail whenever a 404 or 500 error occurs. But when one of my users encounters a JavaScript-Error, I'm not informed. I thought anyone in the web has solved this problem but didn't find anything, so here's my take: Just send any error using Ajax (here: using Prototypes Ajax abstraction) to the server

PLAIN TEXT

JAVASCRIPT:

onerror = Extranet.mailError;
function mailError(msg, url, line) {
var postBody = 'url=' + url + '&line=' + line + '&message=' + escape(msg) + '&useragent=' + escape(navigator.userAgent) + '&user=' + escape(user_name);
var myAjax = new Ajax.Request('/api/jserror/', {method: 'post', postBody: postBody});
}

user_name is a JavaScript variable holding the Django username (so I know whom I can inform when the error is fixed).

On the server side, I just send me mails containing the JavaScript error message, the username and the user agent:

PLAIN TEXT

PYTHON:

def jserror(request):
from django.core.mail import mail_admins
omit_messages = ['pointerobj is not defined', 'tipobj is not defined', 'ns6 is not defined', 'enabletip is not defined']
if request.POST.get('message', '') not in omit_messages:
message = """url: %s (%s)
%s
user-agent: %s
username: %s
""" % (request.POST.get('url', ''), request.POST.get('line', ''), request.POST.get('message', ''), request.POST.get('useragent', ''), request.POST.get('user', ''))
mail_admins("javascript error", urldecode(message))
return HttpResponse()

Yeah, that's all very trivial but I wonder what other solutions exist for this problem...

Django: Serve big files via fcgid

philipp.keller — Wed, 03 Oct 2007 07:14:28 +0000

I've got a django project running which requires you to login to access files.
That means that I have to serve the files via python, like this:

PLAIN TEXT

PYTHON:

@login_required
def download(request, filename):
# ... some code specific to my site ...
response = HttpResponse(mimetype=postUpload.mimetype)
response['Content-Disposition'] = "attachment; filename=" + original_filename
response['Content-Length'] = os.path.getsize(filename_path)
response.write(open(filename_path).read())
return response

The problem: If the download of a file exceeded 5 minutes (big files and/or low bandwidth) the download was canceled on the server side by a timeout. This Apache configuration for mod_fcgid solved the problem (see mod_fcgid documentation for BusyTimeout)

PLAIN TEXT

CODE:

c>
BusyTimeout 1200

The problem was that the apache module scanned every minute for processes that run for more than BusyTimeout seconds. These processes are potentially in bad health (infinite loop et al.) and have to be killed. Not so with my processes (since I know what I'm doing..). The setting of the busy timeout to 1200 seconds now lets my processes run for a maximum of one hour.

As this setting can't by overwritten in a htaccess file by default I needed to bug my web hosting provider with the request, which was handled in 24 hours, so thanks for that one!

PS: If you know of another way how to serve protected static files via a single sign on (no HTTP basic auth), please let me know.

Python: Sort a list of dicts by dict-key

philipp.keller — Mon, 23 Jul 2007 13:52:29 +0000

I always forget that one and end up searching for half an hour:

PLAIN TEXT

PYTHON:

>>> list = [ dict(a=1,b=2,c=3),
... dict(a=2,b=2,c=2),
... dict(a=3,b=2,c=1)]
>>> list.sort(key=operator.itemgetter('c'))
>>> list
[{'a': 3, 'c': 1, 'b': 2}, {'a': 2, 'c': 2, 'b': 2}, {'a': 1, 'c': 3, 'b': 2}]
>>> list.sort(key=operator.itemgetter('a'))
>>> list
[{'a': 1, 'c': 3, 'b': 2}, {'a': 2, 'c': 2, 'b': 2}, {'a': 3, 'c': 1, 'b': 2}]

Python: Find out cpu time of a certain process

philipp.keller — Fri, 08 Jun 2007 09:52:52 +0000

To find out how many percentage a certain process uses the cpu:

PLAIN TEXT

PYTHON:

import os, time
# find out the pid by username.
# "-o pid h" omits the header and just prints the pid
pid = os.popen('ps -U my_user_name -o pid h').read().strip()
# 14th column is utime, 15th column is stime:
# The time the process has been scheduled in user/kernel mode
# The time value is in jiffies. One jiffie is appox 1/100 second
# see man proc for more info
stat = os.popen('cat /proc/%s/stat' % pid).read().strip()
cpu_time1=int(stat.split()[14]) + int(stat.split()[15])
time1=time.time()
time.sleep(1)
stat = os.popen('cat /proc/%s/stat' % pid).read().strip()
cpu_time2=int(stat.split()[14]) + int(stat.split()[15])
time2=time.time()
print str(float(cpu_time2 - cpu_time1) / (time2 - time1)) + "%"

I don't know though if the number is accurate.
What is "cpu time" anyway? It's the time the process is running (using the cpu for 100%) divided by the time the process is laid asleep by the scheduler.
Then, jiffies seem to be not a safe number for time measurements.
But for relative measurements it should do the trick.

add bandwidth to a file download in python

philipp.keller — Sun, 25 Feb 2007 01:51:05 +0000

Another short code snippet I came up when writing the module download script

The following class keeps a download under a certain ratio and prints live download stats (KB/s).

PLAIN TEXT

PYTHON:

class BandwidthFile:
"""
limits the file download to a certain amount of bytes per second and prints download stats
"""
def __init__(self, filename, filesize, bytes_per_second = 20000, mode = "wb"):
self.fp = open(filename, "wb")
self.bytecounter = 0
self.started = datetime.datetime.now()
self.bytes_per_second = bytes_per_second
self.filesize = filesize
self.filename = filename
def __del__(self):
self.fp.close()
def write(self, bytestring):
self.bytecounter += len(bytestring)
seconds_since_started = (datetime.datetime.now() - self.started).seconds + 1
while (self.bytecounter / seconds_since_started)> self.bytes_per_second:
seconds_since_started = (datetime.datetime.now() - self.started).seconds + 1
time.sleep(0.1)
self.fp.write(bytestring)
sys.stdout.write("\rdownloading %s (%i/%i KB, %i KB/s)" % (self.filename, (self.bytecounter / 1024), (int(self.filesize) / 1024), (self.bytecounter /seconds_since_started / 1024)))
sys.stdout.flush()

To use the class with ftp:

PLAIN TEXT

PYTHON:

from ftplib import FTP
ftp = FTP("ftp.modland.com")
filesize = ftp.size(remote_filename)
f = BandwidthFile(local_filename, ftp_filesize)
ftp.retrbinary('RETR '+remote_filename, f.write)

The status line looks like this:

downloading /home/phred/modules/Skaven/beyond the network.it (95/4411 KB, 19 KB/s)

If you have suggestions how to enhance the code snippet feel free to leave a comment!

set timeout for a shell command in python

philipp.keller — Sun, 18 Feb 2007 23:04:06 +0000

I wanted to run a shell command in python without knowing if the shell command is going to exit within reasonable time (adplay that was, sometimes it simply hangs).

Update: the "task" module of Rob Hooft seems to solve this exact problem. At the time I wrote this, the python.net website was down. I leave my solution here just for archive purpose.

PLAIN TEXT

PYTHON:

def timeout_command(command, timeout):
"""call shell-command and either return its output or kill it
if it doesn't normally exit within timeout seconds and return None"""
import subprocess, datetime, os, time, signal
start = datetime.datetime.now()
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while process.poll() is None:
time.sleep(0.1)
now = datetime.datetime.now()
if (now - start).seconds> timeout:
os.kill(process.pid, signal.SIGKILL)
os.waitpid(-1, os.WNOHANG)
return None
return process.stdout.read()

PLAIN TEXT

PYTHON:

>>> output = timeout_command(["sleep", "10"], 2)
None
>>> output = timeout_command(["sleep", "1"], 2)

The process can be killed when it has run for too long (the os.waitpid waits for the kill to end and avoids defunct-processes) and furthermore the Popen'ed process' printed is caught and returned if it doesn't timeout. However, subprocess.Popen is called with a list as argument. That means, that the command isn't passed to a shell and furthermore you can just call one command with options, nothing more.

Turn demoscene modules into mp3s

philipp.keller — Sun, 18 Feb 2007 17:39:11 +0000

tunes of purple motion turned into mp3s are playing in itunes

If you were part of the demoscene in your former life, if you were and still are fond of modules (those sound files with the mod, xm, s3m, it, ... ending), if you are a linux user and if you still want to listen to this music on your computer without doing all the tweaks of installing (or even compiling) music player plugins for itunes or amarok or if you simply want to listen to Purple Motions tunes on your mp3 player then this little tutorial is for you. If not, then you won't have read that far anyway..

short version for the impatient

download downloadmod.py and mod2mp3.py into /usr/local/bin/

PLAIN TEXT

CODE:

sudo apt-get install xmp adplay unrar lame
mkdir ~/modules/
downloadmod.py ~/modules/ "Purple Motion"
mod2mp3.py ~/modules/

First, get those modules

The first task is to get those modules onto your computer (as you most certainly deleted them - either per accident or when you were in needed of some space on your hard drive back when hard drives where small and expensive)

Afaik the most complete module resource is modland. The crux is that it's just an ftp site with deep directory structure and without a search facility. All it has got is a complete list of all the modules in a RAR file (which is kept up to date by a cron job) that holds a text file with all the available modules and their path.

To download all modules of a certain artist, I wrote a little python script that downloads the module list, unrars it, caches it for further search requests, searches for the artist and downloads all modules of that particular artist.

downloadmod.py ~/modules/ "Michael Land" downloads all Adlib songs of Michael Land and places them into ~/modules/Michael Land/ and writes some artist/album meta information into those newly created directories.

Alternatively you can adapt a quick and dirty bash function. It uses wget and is quite messy but it should do the task as well. You have to download and unpack the module list before you call the shell function.

Then, convert those modules into mp3 files

To convert the modules into mp3 files you usually take a module player and call with some special parameters in order to "play" the module into a wav-File and convert that wav into an mp3 file.

On linux, I'd suggest to install this module players (both exist as debian packages and work without hassle on my ubuntu installation)

xmp is *nix "native"
adplay supports many adlib formats

I wrote a python script that converts the downloaded modules into mp3s using the just mentioned module players. You need at least xmp or adplay. If you install both, you'd be able to play about 90% of all modules.

Just call mod2mp3.py with your root module directory as argument and the script creates a mp3 file for each module that is convertible and isn't already converted. If you downloaded the modules with my download script, this script will use the meta information of the first script to write the id3 tags of the mp3s.

If everything worked, you now should have mp3 files with correct id3 tags that can be imported into your amarok/itunes library.

What about us windows users?

Sorry guys. I've got to let you down. One whole evening I tried to get any module player writing it's output to a wave file from the command line. I tried modplug and Winamp with a module player plugin but without luck. If you succeed, let me know. Up to then, just take a linux box, copy/download all modules there and let the box convert your modules over night and import it back into your iTunes player. That's what I did.