(2006-09-30) Mccallister Web Scripting Platform

Brian McCallister seeks an architecture providing a robust general web server designed for writing WebApp-s.

The real goal was to find a good way to run Rails and Django and TurboGears and whatnot on Apache HTTPD. Should be trivial, right? Sadly, it is not. Nor is it trivial to do it on LightTPD, nor much else. The only easy ways to run rails are on webrick or Mongrel. Seriously. Neither of these is a good option as a primary web server, which is unfortunate. The situation is similar for Python, though I don't do much python work anymore, so I'll keep my trap shut on that front ;-) There is an obvious option for a robust general Web Server designed for writing web apps (Apache). To maximize it though, you need to use an mpm other than prefork (Pre-Forking). The main reason you need to avoid prefork, for ruby and python apps, is the massive memory footprint engendered by loading a ruby or python vm in each httpd process. They both have fairly hefty memory requirements for their VMs, so having lots of processes (which can work fine for small VM systems, like PHP) gets painful disturbingly quickly. You need prefork to run them in process because they both have, basically, broken behavior with regard to posix threads. The alternative to having a VM per httpd process is to have a pool of processes and play games with various protocols to hand off the request to the the scripting VM. The aforementioned FastCGI is a popular option, just proxying (Proxy Server) the HTTP is another popular one, AJP is yet another, as is SCGI. This is what we all wind up doing not just in Ruby and Python, but also in Java (where AJP dominates). This certainly works, but it would be nice to not add the network (or domain socket) hop if we could help it.

The basic problem, in my opinion, is that Ruby (and Python) both like to be the language. You extend them, you don't embed them. You use ruby to drive C, not C to drive ruby. Extending ruby is fun and easy. Extending httpd with ruby is pain... So, in my copious free time I have been trying to find a good way that works with httpd rather than against it. I really do believe that Scripting Language-s are the best way to glue most webapps together, so I want to keep that. In order to work nicely with httpd the language has to be fairly agnostic with regard to running in a multithreaded server (or even a single thread for everything server like lighttpd (I think)). So, I go looking for a scripting language that does play nicely therein (Ruby doesn't - I love ruby, I use ruby daily, but it is crap for embedding in an embarrassingly concurrent, multithreaded server).

It is surprising how bad the state of robust, thread-friendly scripting languages is. I found three (and a half) so far that look like they'll fit my needs. It is a surprising list until you think about it. They are Lua, JavaScript (Spider Monkey), and TCL. The half is Ficl. I am not sure it works yet, but it should. It should also give me lots of headaches being Forth based :-)

The shocking part of this list is that there is not a single Scheme on it... There is a big benefit to working with something that sees itself as the first class citizen - they tend to have better^w more extensive libraries. For libraries, Perl > PHP > Python > Ruby > TCL > Lua > Spider Monkey. That said, C and Java have them all beaten, and hooking C into Lua, at least, is even more trivial than with Ruby or PHP. It just means embracing C, which is something people have been working to escape. C is really good at slinging bits around on unixy systems though. Really, really good. Pretty much better than anything else, even if it is not as nice on developers as other languages/environments.

So, where does this ramble lead? I have been playing around a fair bit with Lua + C libraries + httpd (modules) based webapp development. I really like it.

Mar'2008 update: Nathan Youngman notes that this work became mod_wombat, and Matthew Burke is looking for students to work on mod_wombat as part of the 2008 Google Summer of Code.