Scredit Crunch

Bye bye Perl (the story of HTTP chunked encoding with Django)

Posted in Hacks, Night of Code by setok on March 10, 2009

A week or so ago the last remnants of our old Perl version were banished from Scred. Last year, during summer, we spent quite some time porting Scred completely from Perl to Python and Django. Users did not really notice anything: the new version looked and acted identically to the old one. The reason for this switch was simple: our old self-baked framework was beginning to fray at at the edges and development was slow and painful.

We did evaluate moving Scred to Catalyst. However, Perl is a language which definitely splits opinions, both within Scred and beyond, and we needed to find something everyone would be at least semi-comfortable with. Now, as many know, my own personal favourite language is Tcl, for its wonderful simplicity and ultimate power. In fact it’s probably the most underrated language out there. For that reason we took a look at OpenACS which is reckoned to be extremely mature and scalable, but the learning curve at the time also felt very severe. It would also require running AOLWebServer. This has been built to be an extremely scalable web server, with Tcl at its scripting core, but everyone in the team had had at least some experience with Apache. Moving away from that did not feel appealing at all. There are Apache modules for Tcl on Apache, but nothing in the way of a full web framework (something for a summer project, perhaps?).

So Django it was, despite the pre-1.0 status, and overall it has mostly managed to do the job.

Everything was then ported, except one crucial part: the internal API used by the mobile Scred client. This was working perfectly well as it was so was never high up on the priority list to port. More importantly, Django/WSGI/mod_python have frankly no support for chunked HTTP encoding. This is an utterly inexplicable oversight as the HTTP standard quite clearly says chunked encoding MUST be supported, and one that has still not be satisfactorily remedied. Often browsers do not use chunked encoding but our friends the mobile platforms commonly do (both MIDP and the iPhone). Finally in January it was decided that this port really had to be done, as it was slowing the pace of our development to have to keep checking that the Perl based API would continue to work with new database changes.

After some amount of Googling we found a solution that was based on using Apache’s mod_proxy for dechunking. It was a hack, no doubt about it, but it seemed to be working right up until we tested with the MIDP emulator (ancient 1.0.3 as there’s nothing later for Macs and MIDP 1 was our target platform). API calls would regularly end up in an explicable “Network error” on the client. After some amount of debugging we could not find any reason for this. Once more the update to Python was put on the back burner to make way for several other important features which had to be pushed out.

Finally we dug up some time to continue to solve this issue. The confusing thing was that it worked on an actual device, but we could not be sure if that was just luck without pulling out tcpdump, netcat and similar tools. To cut a long story short we found that mod_proxy was closing the connection right after sending a response to each HTTP request. Now, this is perfectly fine, although usually web servers today keep connections open, at least for a time. Disconnecting after response was the HTTP 1.0 way. Unfortunately it seems like there is a bug in the 1.0.3 MIDP runtime. It did not react to this disconnect by then reconnecting with the next request. Instead it would just cause an IOException once a new request was sent (the details of the actual underlying socket are somewhat hidden in J2ME). That means that every other request works fine, every other one not.

At the end of the day we decided to push the update out. We tested on some phones and did not detect any problems so, with any luck, it’s only the emulator environment which does not handle this correctly. If you stumble across problems, do let us know!

Moral of the story: if you want to handle chunked encoding with Django, set up mod_proxy and be aware that the 1.0.3 MIDP environment has a potential bug in it!

About these ads

4 Responses

Subscribe to comments with RSS.

  1. Job Wales said, on March 10, 2009 at 10:12 pm

    Wow sounds like Django really solved a lot of your problems, saved you time and money.

    Actually you should read your post again to get the full feeling for how much time you wasted porting an application that probably didn’t need it.

  2. Mike Hedge said, on March 10, 2009 at 10:57 pm

    wow. awesome news. exciting

  3. setok said, on March 11, 2009 at 2:35 pm

    @Job whether it was worth it is something that time will tell and depends a lot on our capability to further advance Scred. It was a risk and took more time than we had wanted (and it was done after we were incorporated, with the view that the previous work was prototyping). Development with the Perl version was getting slow and messy.

    For the record it wasn’t as if we were working on that port since summer. It was just that the API port was not highly critical to get ported so we focused on actual development work instead, after we got the rest of the core ported. MiniCorps, for instance, have been built only in Django.

  4. LostONe said, on September 24, 2010 at 9:02 pm

    If you’re searching for something for apache.. try mod_websh it’s FULLY SCALABLE AND WORKS. It’s always under modification, even if it’s not downloadable, you can use the svn or cvs.

    I’ve used it for a project, i had a rough start, but once you get the hang of it you won’t regret it:D.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

%d bloggers like this: