atom

RiDDiM: A streaming audio server in python

2010-07-19T00:02:39Z

1.0

For some time I was looking for a way to stream music from my home server to other computers, across the internet. SHOUTcast accomplishes this, but it’s closed-source, seems essentially unlicensed, and comes with a pretty spooky disclaimer. At this point using closed source is pretty much out of the question, especially for network services hung off the wan. There are other projects out there, but mostly they seem abandoned and/or to be suffering from the same opacity as SHOUTcast.

So what’s an enterprising hacker to do? Not sure. But I rolled up my sleeves, broke out $EDITOR and python, and came up with a streaming audio server that I call riddim.* Right now it only handles mp3, but it has a few nice features, including persistent playlist management through an on-disk cache, fast forward/rewind, metadata, and because it’s multi-threaded, multiple clients. I have added in remote procedure calls with python’s xmlrpclib, part of the standard library, so everything can be remotely administered.

My design goal for this project was to keep it as simple as possible, while creating functionality that a streaming music server should probably do, all while minimizing dependencies. Right now the only non-standard python library required is mutagen, but riddim could be made to work without it. I have not done any benchmarks for multiple clients, but it should scale to support quite a few because it’s an extremely lightweight model. RAM and CPU usage is virtually zero with one client connection. That’s significant, because riddim replaces my old setup, which involved mounting directories remotely with sshfs, and that is pretty resource-intensive. And if you think about it, it really doesn’t make sense to waste cycles encrypting mp3 data. That wastefulness was probably my main motivation for spending time writing riddim.

Yes, but what does it do?

The basic flow is this: start up the riddim server, enqueue some tracks. This is done with riddim.py. The server waits for an HTTP connection on port 18944, and when it gets a client connection it starts streaming mp3 data into which metadata has been chunked to the client. Simultaneously, the server listens for RPC connections, which it delegates to the proper methods.

Patent (not) pending

As an aside: this is a project that’s been knocking around in my head for quite some time, and I’m pretty satisfied with how it turned out. I think it’s illuminating in this case to note how the unformed idea, months in incubation, swelled up into my consciousness, until the imperative to make it real became overwhelming, at that point I expressed the idea in code. Patent lawyers make a lot of the idea/expression dichotomy, and a basic tenet of patent law is that ideas are not patentable, while expressions are. Maybe now that I have expressed this idea I should try to get a patent on it. Assuming of course that there’s no prior art on the subject. ;) In sum, I’m not sure if riddim will be useful to other people, but it’s something I’m going to be using for years to come.

*(According to wikipedia, riddims are instrumental song versions, usually consisting of a bassline and percussion.)

LaTeX Tips: XeTeX and Makefiles

2010-06-14T17:40:59Z

Any geek worth his salt knows that there is only one word processor; and that is LaTeX, of course. After all, when a pronunciation guide is required to say the very name of a piece of software, how can potential challengers hope to compete?

Makefiles For Speed

Simple Makefiles and vim-wizardry add even more awesome to LaTeX:

main:
        xelatex file.tex

The above compiles file.tex into file.pdf with XeTeX. (XeTeX is basically Knuth TeX + unicode.)

If you use evince as a pdf viewer, the change will become immediately visible. Thus, it’s nice to have an editor window and an evince pdf viewer window open side-by-side. This speeds up the the edit-compile-run workflow about as much as is humanly possible.

If you’re using vim, you can do this after editing:

:w
CTRL-z
% make
<look at pdf>
% fg
<make more edits>
. . .

However, this is faster:

:w|!make
<look at pdf>
<make more edits>
. . .

If you’re not using vim, kill yourself (I kid!)

XeTeX Makes It Hot

I’m using XeTeX exclusively these days because of its support for OpenType fonts. These fonts look great and are very easy to use.

This is about as simple as it gets:

The argument to \setromanfont can be anything, provided it’s an installed font. The list can be gotten by otfinfo --family somefont.otf.

`rtorrent:` organize .torrents by tracker

2010-06-03T03:42:21Z

In the words of its creator, libtorrent is:

[A] BitTorrent library written in C++ for *nix, with a focus on high performance and good code. The library differentiates itself from other implementations by transfering directly from file pages to the network stack.

I have been using rtorrent—a text-based ncurses client written on top of libtorrent—for a few years now, and it is by far one of the most solid pieces of software I have yet to encounter. As of this writing, the latest version of rtorrent is:

% pacman -Q rtorrent
rtorrent 0.8.6-2

One of the benefits of rtorrent is that it supports huge numbers of seeding files, with a low memory footprint. I have had hundreds of torrents seeding simultaneously, and rtorrent only consumes a few megabytes of RAM. This encourages me to seed my torrents for a long time, weeks and in some cases, months. It’s also nice that rtorrent doesn’t require Xwindows. However, a problem created by this resource efficiency is that the download directories (of which there is only one by default) become quite crowded without constant pruning. Reading 20 pages of output from ls|less to find out what’s been downloaded sucks.

Luckily, the rtorrent devs have a solution for this problem, which is a series of hooks that allow registration of event callbacks; for example, when a torrent download completes.

Combining some of those hooks

And this directory structure,

/download
        ├──   tracker1.com
        ├──   tracker2.com
        ├──   torrents
        │   └──   .sessions
        │   └──   tracker1.com
        │   └──   tracker2.com
        └──   queue

makes rtorrent mv finished torrents to a directory on a per-tracker basis, and updates the torrent files with the new (target) directory path. For instance, when a torrent registered at tracker1.com finishes, it is mved into /download/tracker1.com, and the session torrent’s directory path is updated to reflect that change (this is important so that when rtorrent is closed and re-opened, it doesn’t start downloading the file again, instead it checks the hash of the original). Thus, the .sessions directory is NOT optional.

The Automatic Save Folder firefox extension can be used to automatically create the torrents/ hierarchy.

It’s hard to complain about rtorrent, but my chief criticism is the rtorrent.rc file’s hacky configuration syntax. It is positively terrible; seemingly made up by the authors in an attempt to create Yet Another Configuration Syntax, and with the added demerit of being very finicky about escaping of characters. (However, in my experience, this enamoredness with creating ad-hoc configuration languages is shared by many C/C++ programmers.) Sometimes syntax errors throw an error, sometimes they fail silently. Limited debugging help is available in the client by pressing lowercase ‘L’. With a program as otherwise clean and featureful as rtorrent, it seems like such a waste to have a configuration syntax that is constantly in flux from rtorrent version to version, and is constantly being re-written. If the devs had used something like YAML from the get-go this wouldn’t be a problem and wouldn’t make my eyes bleed.

`iptables` is for masochists

2010-05-29T11:01:57Z

Sun May 30 20:50:58 CDT 2010:
This post seems to have stirred up some anti-shorewall, pro-iptables angst. To that end, I would recommend checking out conntrack-tools and netfilter hacks (alpha-masochists only!).

Also catting /proc/net/ip_conntrack isn’t a bad idea.

. . .

Writing iptables rules by hand is for masochists. The iptables syntax is needlessly complicated, and this complexity has no benefit. Additionally, writing iptables rules by hand violates the Rule of Generation. It is also difficult to tell whether iptables is actually working as it should.

Enter shorewall, a rule engine that abstracts away the nitty gritty details of iptables, and makes the rules themselves more stateful than the combination of command-line entries and raw syntax of /etc/iptables/*.rules, by relegating them to text files with macro support for common port configurations.

Shorewall is “iptables made easy” and it is also iptables made more powerful. For example, it is trivial to set up a DMZ with shorewall. What is overly complicated with iptables, is a few lines of configuration code with shorewall.

An syntactic example will illustrate the above points nicely. Which is clearer, as an example of opening some oft-used ports? Iptables,

iptables -A OUTPUT -p tcp --dport ftp -j ACCEPT
iptables -A OUTPUT -p tcp --dport ssh -j ACCEPT
iptables -A OUTPUT -p tcp --dport www -j ACCEPT
iptables -A OUTPUT -p tcp --dport https -j ACCEPT

Or shorewall?

ACCEPT  net     $FW     tcp 21,22,80,433

Aside from the verbosity of the rules, there is another reason why using iptables directly is a bad idea: As noted by Ahmed Mekkawy, the main problem with userspace firewalls is that they are inherently volatile. In other words, because software firewalls have to be applied to change the state of a running system using an init script or daemon, a good sysadmin has cause to wonder, “is my firewall working? Did I forget to restart the daemon after the last reboot? Did an upgrade break it?”

With iptables, the only surefire way to answer these questions is to use nmap or iptables-save to check that the rules are in place. However, a properly-configured system may have hundreds of rules, which makes scanning the output for correctness a headache. And port scans take time.

Shorewall provides an answer to the “am I running” question,

# shorewall status
Shorewall-4.4.3 Status at downbe - Fri May 28 13:27:16 CDT 2010

Shorewall is running
State:Started (Fri Apr  9 17:24:07 CDT 2010)

as well as the “which rules are defined” question:

# shorewall show     
Shorewall 4.4.3 filter Table at downbe - Fri May 28 13:28:58 CDT 2010

Counters reset Fri Apr  9 17:24:07 CDT 2010

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source destination         
 . . .

In sum, accessing iptables directly is completely unnecessary, not to mention error-prone. Raw iptables rules are less stateful than other solutions, and exist at a too-low level of abstraction. So unless you are a masochist, you should probably be using shorewall.

Going paperless with procmail, fetchmail, gmail, python, and a scanner

2010-02-07T10:06:57Z

Although I’m generally wary (and weary) of the GTD crowd, I read something interesting lately called “30 Days to a More Organized Life”; and was motivated to implement day six, entitled, “Scan Everything You Can”.

Between work, school, and general mail; there is a lot of paper in my life. Some of it is important, but most of it is trash. Nonetheless I hold on to it “just in case”. For example, bank statements. These are in that gray area between “totally dispensable” and “might need it someday.” It’s hard to say, and this creates two problems. First, I have a bunch of sensitive documents in my apartment, either laying around or filed in a filing cabinet. This in insecure and un-backed-up. Second, it’s just more clutter.

30 Days suggests a remedy:

Sit down at your computer with the scanner on your desk and all your miscelanious [sic] paperwork at your side. Go through each sheet and ask yourself: “do I need this?”

If you don’t need it, throw it in the shredder. If you do, ask yourself, “do I need the original?” If yes, then put it in a “to file” pile. If you don’t need the original, run the document through the scanner.

So when I saw a great deal on a wireless scanner with built-in email functionality, I jumped at the opportunity. Now I have a setup that allows me to do the following:

Scan documents and email to a dedicated gmail document cache account at the push of a button;
Download all new email from that account periodically (fetchmail);
Extract the attachments and file them away intelligently (procmail/python).

The flow is simple; it only took me an hour or so to figure it out and iron out the kinks.

    push:  scanner -> gmail
    pull:  cron -> fetchmail -> procmail -> python

Fetchmail pulls the files down from gmail every hour, and passes the result to procmail:

Procmail passes content-type matches to my python script, which does the real work. I get an email for each new scanned file, and it would be pretty trivial to include a link to that file and serve it up on a password-protected webserver:

I did say that the files are archived intelligently, that’s done by munging the subject header of the email message into an archive directory path. For example, if I scan a document and email it to my cache gmail account with the subject “bank.citi.statements” then my python script will put the files attached to that email into $archive_root/bank/citi/statements. Pretty slick. Python, to its credit, makes this kind of stuff stupid-easy. It’s true that python is one of the better-documented languages out there, while also being highly expressive.

And that’s pretty much it. A few ancillary benefits of this email-first approach (as opposed to scanning directly to the local computer, and then batch archiving to gmail) are that the scanner doesn’t have to be made to work with Linux (no small benefit), you still get the cloud-based archive for free, and what I really have here is a completely self-contained system whereby I can email any file as an attachment to a gmail address, and it will eventually end up archived on my server.