Most of the info I could find on this simply claims that the encoding of the POST request is the same as the page that contained the submitted form. Therefor, if you serve pages as UTF-8, any forms that are submitted back to you will also be in UTF-8.

That may be true, but that doesn’t really help you, if you’re an idealist who wants to treat the HTTP request like the stateless request that it really is. Such as yours truly.

Looking around in specs, there are two methods a browser can use to indicate the character encoding in a POST request:

• By specifying it in the Content-Type header, such as “application/x-www-form-urlencoded; charset=UTF-8”. It looks like there’s a Mozilla bug from back in 1999, in which this was discussed. Eventually, they didn’t opt for this method because it caused breakage on several HTTP server implementations at the time.


• For forms that use the application/x-www-form-urlencoded encoding (most forms that don’t do file uploads), a hidden field named ‘_charset_’ can be included. Browsers will override its value on submission with the encoding used. This will be in HTML5, and you can find it in the current draft.

Neither of these methods are handled by Rails or Rack for Ruby 1.9, and all you get is strings with the #encoding set to US-ASCII, while the string actually contains UTF-8. A nice contradiction and source of exceptions elsewhere deep in your application.

I set out to get this sorted in my app, and wrote a monkey-patch. The patch automatically adds the hidden field when using FormHelper, and tries to deal with both that field and the Content-Type header in requests. It’s been briefly tested in Firefox 3.6 only. You can find it in a gist on Github.

But now I had to, and not realizing it, I spent a good hour fighting a vague problem from various angles, before I finally made the breakthrough.

This config file contains a line “LANG=C” by default. This has many consequences, but one of them is that all file operations in Ruby 1.9 expect files to be in ASCII encoding, while the rest of the system operates in UTF-8.

I ran into this with a Rails application hosted in Apache with Passenger, and a view containing non-ASCII characters. Ruby’s errors when it encounters incompatible encodings are… very terse.

But I imagine the reason that particular line in that particular config file is really there, because someone else banged his or her head against a wall for a good hour too, because LANG wasn’t C.

If we applied the same ferocity the media industry has applied to fighting piracy, to fighting spam instead, how much better would our internet be?

If we were to make a list of every IP address of an SMTP server or rogue machine filling our inbox with spam, or of a machine probing and brute-forcing for SSH access to our systems, or attempting to exploit vulnerabilites in our HTTP servers, FTP servers, SMB servers, you name it.

And then we would go and send abuse mail to providers of every single one of them. With logs. Detailed. And thorough inquiries after the perpetrator’s identity.

And we would go all the way to take legal action against these basement crawlers or their shady providers.

Point is…

We take an awful lot of this crap as just an everyday chore for our systems to deal with, and put all of our services in virtual bunkers to prevent anything bad from happening because of it. Which is good sysadmin practice, of course, but…

Then there’s nobody putting these guys on the stand for what they are doing. And it piles up. This often isn’t even that far from our doorstep. Most of the attacks on the systems I manage are from elsewhere in continental Europe.

And furthermore, these things or not legally grey, like some of the things the media industry is pursuing. But pitch black, as far as I know.

So why does it take such incredible effort to get back at these people?

This is an early prototype kind of thing. Here’s what it looks like:

You will find the code and some quick instructions are hosted at Github:
http://github.com/stephank/libvirtweb/

The rest of this post is some background.

Our office is a battlefield split in two

On the one side, there’s Linux on the desktop, Mac OS X on the desktop, Linux machines powering all primary services we sell. And then on the other, Microsoft Exchange, and Windows administrators. Sarcastic retorts and counter retorts go back and forth.

So far, this is holding up fairly okayish. There’s just one piece of iron powering our office, and a Windows virtual machine gobbling up about half of it’s resources.

We’ve tried a couple of alternative virtualization technologies, but surprisingly, an Ubuntu Server install with KVM was the most stable of all! Now comes the problem of colleagues wanting to manage the thing from their Windows desktops.

Turns out it was a good choice not to make Windows the host. When we doubled up the processor, we found our Windows license goes up to just 4 cores. Score one for the *nix side.

Face to face with libvirt

No doubt one of nastiest obstacles in creating this was libvirt. When you look at it’s goals from a high level (a very very high level), it sounds nice: a single API to manage virtual machines of any kind.

In practice, that has become a single API to manage local and remote iron over various types of connections and tunnels, and running various virtualization technologies. All that in a blocking fashion. Hell, I can’t even imagine how to do all that in a nice asynchronous API.

It looks like the only useful thing the libvirt API can be used for is to script some things in Python. And the Python bindings are very limited. Scripting things in bash works just as well using virsh.

If I had a say in it, the libvirtd daemon would not be optional, and would simply expose a DBus API. Rather like NetworkManager.

Mixing Twisted and CherryPy

The above is written in Twisted and CherryPy. However, when dealing with SSH tunnels, I had to find a way to take the tunnel out of the equation for the browser. The approach I took is a proxy for the RFB protocol, which is what is used to view consoles in KVM and QEMU.

Adding an RFB server to a CherryPy server is not really one of CherryPy’s use cases. However, CherryPy can be treated like a WSGI application. This allowed me to host CherryPy in a twisted.web.wsgi.WSGIResource, and mix the rest of my Twisted components with the webapp.

Using CherryPy as a WSGI application, and specifically in Twisted, is not really documented well anywhere. So here’s a snippit of code to do it. This is based on CherryPy 3.1.2 and Twisted 8.2.0.

#!/usr/bin/env python

import cherrypy
from twisted.internet import reactor
from twisted.web import wsgi, server
from twisted.python import threadpool

# Here's our hello world CherryPy application
class Root(object):
   @cherrypy.expose
   def index(self):
      return "Hello world!"

# Use the 'embedded' configuration template
cherrypy.config.update({'environment': 'embedded'})
# We need to unsubscribe the CherryPy server to prevent a port conflict
cherrypy.server.unsubscribe()
# Start CherryPy internals
cherrypy.engine.start()
# Make sure we shut down CherryPy when we're done
reactor.addSystemEventTrigger('after', 'shutdown', cherrypy.engine.exit)

# Create a WSGI callable from our application
app = cherrypy.Application(Root())
# Twisted needs a threadpool to run WSGI applications
threads = threadpool.ThreadPool()
threads.start()
# Make sure we shut this down too
reactor.addSystemEventTrigger('after', 'shutdown', threads.stop)

# Setup the twisted.web factory, and listen on 8080
resource = wsgi.WSGIResource(reactor, threads, app)
factory = server.Site(resource, 'server.log')
reactor.listenTCP(8080, factory)

# The main loop
reactor.run()

I went beyond that, and here is bit of work I’d like to call Whiite-OE. It’s an integration of the Whiite Linux kernel patches for Nintendo Wii support, with the OpenEmbedded cross compile environment.

Find it at: http://stephan.kochen.nl/proj/wii-oe/

Eoin Hennessy has taken over GstRaopSink development. He actually got it up and running, and set up a git repository with the code.

I was also requested to publish the source code of the Nurikabe generator. The co-author, Vladimir Panteleev, was okay with it, so I uploaded the source code of the generator and half-done GUI with a GPLv3 license.

My brother is getting up to speed performing as a DJ again, going by the stage name Eric Preston. If you’re into minimal techno, you’ll definitely want to tune in to his monthly livemixes at ericprestonmusic.com. The first of which is now available for free. I think you’ll be pleasantly surprised.

First part is a GDB script that automates the process of breaking on a gtk_paint_* operation and dumping the necessary information. This was written with The Widget Factory in mind, so if your application needs parameters, or is written in Python, it needs adjustment. I’ve also uploaded some gzipped sample output of The Widget Factory.

Second part is a Python script that reads the redirected GDB output and visualizes it like so: (click the thumbnail for the full version)

It doesn’t draw images, text or polygons, but it shows most of the original. I found it helpful in figuring out what kind of calls widgets make when drawing, but also seeing from the theme engine point of view how the methods are invoked.

Here’s the Mist engine with a translucent background:

The big problem is, and Cimi’s blog post does mention this: it needs application support. The theme engine can easily detect and perform drawing with translucency, but it’s up to the application to actually enable it. One way to do this is using a snippit of code like this, after gtk_init and before any windows are created:

 GdkDisplay* display = gdk_display_get_default ();
 gint nscreens = gdk_display_get_n_screens (display);
 for (gint i = 0; i < nscreens; i++)
 {
     GdkScreen* screen = gdk_display_get_screen (display, i);
     GdkColormap* rgbamap = gdk_screen_get_rgba_colormap (screen);
     if (rgbamap != NULL)
         gdk_screen_set_default_colormap (screen, rgbamap);
 }

Another way to do this, is to patch GTK+ itself to prefer RGBA surfaces over the trusty old RGB surfaces. Trusty, because RGBA still tends to break some applications, and probably some GTK+ custom widgets as well.

So to get translucency, either all applications need to be patched, or GTK+ needs to break compatibility. Patching all applications is a rather stupid option, if we have to do that anyways once GTK+ breaks compatibility. I say it’s about time for GTK+ 3.0.

Though I think it’s overhyped, I guess that it will end up to be a good thing. Maybe we will finally get a push for more of this great bling in GTK+.

By the way, here’s the patch against gtk-engines trunk for the above screenshot of translucent Mist:
http://stephan.kochen.nl/proj/gtk-translucent-mist.patch

I’m normally a bit skeptical of anything Microsoft produces, but I’ll say C# is a rather nice language in itself; a bit more so than Java. There’s nothing spectacular about it, but it does offer some things that would make me prefer it over Java. I’m thinking small things like events, readonly fields, and no silly restriction of one class per unit.

That’s just the language, though. I don’t feel like judging the standard libraries. But I can say that I’ve had no significant dissappointment there either.

What this post is really about, however, and what gets me quite excited, is MonoDevelop. I dare say that this is shaping up to be the integrated Rapid Application Development environment for GNOME. It’s a bit buggy, I still can’t find any debugger integration (but saw screenshots of work in progress months ago, so I guess it’s in the pipeline), but altogether I really wouldn’t be able to point out any other IDE quite like it. And this comes from someone who’s otherwise quite firmly in the Eclipse camp when it comes to Java, Python, C/C++ and D.

The builtin GUI builder Stetic works well, writing signal handling code is a snap, other features I expect from an IDE are there as well: refactoring, class browser, compiler error parsing, code completion, etc. Perhaps the only thing really missing is the debugger integration.

Even tough the project I’m working on has both a Windows Forms and GTK# front-end, development primarily takes place in MonoDevelop. Updating the Windows Forms front-end is often postponed until right before a release.

So that’s it. I just wanted to dedicate a post to this great emerging platform.