Scredit Crunch

A week or so ago the last remnants of our old Perl version were banished from Scred. Last year, during summer, we spent quite some time porting Scred completely from Perl to Python and Django. Users did not really notice anything: the new version looked and acted identically to the old one. The reason for this switch was simple: our old self-baked framework was beginning to fray at at the edges and development was slow and painful.

We did evaluate moving Scred to Catalyst. However, Perl is a language which definitely splits opinions, both within Scred and beyond, and we needed to find something everyone would be at least semi-comfortable with. Now, as many know, my own personal favourite language is Tcl, for its wonderful simplicity and ultimate power. In fact it’s probably the most underrated language out there. For that reason we took a look at OpenACS which is reckoned to be extremely mature and scalable, but the learning curve at the time also felt very severe. It would also require running AOLWebServer. This has been built to be an extremely scalable web server, with Tcl at its scripting core, but everyone in the team had had at least some experience with Apache. Moving away from that did not feel appealing at all. There are Apache modules for Tcl on Apache, but nothing in the way of a full web framework (something for a summer project, perhaps?).

So Django it was, despite the pre-1.0 status, and overall it has mostly managed to do the job.

Everything was then ported, except one crucial part: the internal API used by the mobile Scred client. This was working perfectly well as it was so was never high up on the priority list to port. More importantly, Django/WSGI/mod_python have frankly no support for chunked HTTP encoding. This is an utterly inexplicable oversight as the HTTP standard quite clearly says chunked encoding MUST be supported, and one that has still not be satisfactorily remedied. Often browsers do not use chunked encoding but our friends the mobile platforms commonly do (both MIDP and the iPhone). Finally in January it was decided that this port really had to be done, as it was slowing the pace of our development to have to keep checking that the Perl based API would continue to work with new database changes.

After some amount of Googling we found a solution that was based on using Apache’s mod_proxy for dechunking. It was a hack, no doubt about it, but it seemed to be working right up until we tested with the MIDP emulator (ancient 1.0.3 as there’s nothing later for Macs and MIDP 1 was our target platform). API calls would regularly end up in an explicable “Network error” on the client. After some amount of debugging we could not find any reason for this. Once more the update to Python was put on the back burner to make way for several other important features which had to be pushed out.

Finally we dug up some time to continue to solve this issue. The confusing thing was that it worked on an actual device, but we could not be sure if that was just luck without pulling out tcpdump, netcat and similar tools. To cut a long story short we found that mod_proxy was closing the connection right after sending a response to each HTTP request. Now, this is perfectly fine, although usually web servers today keep connections open, at least for a time. Disconnecting after response was the HTTP 1.0 way. Unfortunately it seems like there is a bug in the 1.0.3 MIDP runtime. It did not react to this disconnect by then reconnecting with the next request. Instead it would just cause an IOException once a new request was sent (the details of the actual underlying socket are somewhat hidden in J2ME). That means that every other request works fine, every other one not.

At the end of the day we decided to push the update out. We tested on some phones and did not detect any problems so, with any luck, it’s only the emulator environment which does not handle this correctly. If you stumble across problems, do let us know!

Moral of the story: if you want to handle chunked encoding with Django, set up mod_proxy and be aware that the 1.0.3 MIDP environment has a potential bug in it!