Plot M12

Moved: http://whelkography.blogspot.co.uk/2015/09/plot-m12.html

Python: Django on IIS

Because I only have easy access to IIS servers, installing open source software is often an exercise in total masochism. PHP is invariably an unnecessary amount of pain. Anything that then uses PHP then adds more pain (MySQL is surprisingly pain free actually - it just works, even more so than SQL Server).

because I am now used to the level of pain that PHP on IIS creates, I decided to up the ante and try Django out.

(The trick with PHP is FastCGI. I had lots of legacy PHP sites using the deprecated ISAPI extension; with FastCGI everything just works.)

Back to Django. Frankly, I don't know anything about it. However, I have been playing with Python, and really liking it, so I thought I needed a web framework as well. And a challenge.

The technology / versions I am using are:

• Windows Server 2003 / IIS 6.
• Python 2.6 (more on this later).
• PyISAPIe 1.1.0-rc4 as the ISAPI magic connect to Django doobrie.
• Django 1.2.3.
• SQL Server 2005 (this is not yet set up with Django - that will be a later post).
• Test site installed as new web site NOT a virtual directory in IIS (set up on a new port rather than with a host header, but that shouldn't matter)

Most of the installation can be done by following the PyISAPIe instructions, but there are a few gotchas, and a few things that glosses over, so here they are, hopefully in a Google friendly fashion, because I found nothing on most of them.

1. Using Python 2.7 with PyISAPIe

You can't. At least not with the available binaries, as far as I can tell. When I installed Python, I chose the 2.x branch as the most widely supported, and 2.7 as the most recent version. After following the steps in the link above, up to the bit about running Info.py as a test, I went to the website and got the error Specified module could not be loaded.

To cut a long story short, this is a Windows error; it cannot load a DLL referenced by the ISAPI filter. This is because the RC4 release of PyISAPIe looks for python26.dll, and Python 2.7 has, you guessed it, python27.dll.

The solution for this is to install Python 2.6 instead; after that Info.py works perfectly.

There are a couple of posts out there that allude to using PyISAPIe with Python 2.7; unless I am missing something then you cannot do this with the binaries; you would have to download and compile the Visual Studio project, which was more than I was willing to do for a test environment.

2. Installing PyISAPIe

There is an import omission in the instructions, that if you do not know about it will result in the test script working fine, but every Django request later returning Virtual Directory listing denied.

In the section Setting Up Files, it says "Go to c:\pyisapie\source\PyISAPIe\Python\ and copy the entire Http folder to c:\python25\lib\site-packages". What you ACTUALLY want to do is either MOVE the folder, or copy it then rename the original; the reason being is that the Python module Http supplied will be loaded by preference from the same folder as the PyISAPIe DLL, meaning that if you have copied everything nicely into the Python directory and made configuration changes there, sod all will happen.

3. Installing Django

This is pretty straightforward; a few things to mention that aren't immediately apparent if you are used to the Microsoft way:

• The tarred Django directory should be placed in a sensible place BEFORE you run python setup.py install; you still need it after the install script has run - it is not analogous to an .msi file.
• The PyISAPIe instructions set up a test project in C:\test - it might bear mentioning that this will become the Django project directory so I put mine in a more sensible place, with all the other web applications on the server. Note however that you want to be careful with permissions here - you don't want the web server to have direct access to the source files, so don't put it in a subdirectory of an Inetpub or similar folder.

4. Getting Django working with PyISAPIe

First of all, ensure that you have followed the PyISAPIE instructions, and also that the original Http folder in the PyISAPIe folder is renamed / deleted.

The next thing to do is ignore the Examples\Django folder completely; this is implied, but I don't think it was implied strongly enough. Or maybe I'm just dense. What you want is the Examples\WSGI folder; copy Isapi.py out of that into the Http folder in your Python install.

Next, add the sys.path related lines and alter the os.environ["DJANGO_SETTINGS_MODULE"] as in the instructions. Note that the sys.path line points to the parent folder of your Django project, NOT actually the project folder.

This then gave various weird errors:

Could not initialize interpreter

Traceback (most recent call last):
  File "C:\Python26\Lib\site-packages\Http\Isapi.py", line 29, in 
    from md5 import md5
  File "C:\Python26\Lib\md5.py", line 8, in 
    DeprecationWarning, 2)
  File "C:\Python26\Lib\warnings.py", line 29, in _show_warning
    file.write(formatwarning(message, category, filename, lineno, line))
Exception: Not currently processing a request

So far, I have fixed this by commented out the from md5 import md5 line; I am not using the RunScript stuff, so it is fine.

This then gave me:

Could not initialize interpreter

Traceback (most recent call last):
  File "C:\Python26\Lib\site-packages\Http\Isapi.py", line 54, in 
    from trac.web.main import dispatch_request as TracHandler
ImportError: No module named trac.web.main

I'm definitely not using Trac, so I commented out the Trac related lines.

This then gave me Hello World! and a working Django install.

Now I just have to figure out what the hell it is, what the hell it does, and how to make it do it. Oh, and connect it to SQL Server.

Just to recap, here are the major fuzzy bits I found in the PyISAPIe documentation:

• Make sure your PyISAPIe version matches your Python version.
• If you copy the Http folder into your Python install as suggested, make sure you rename the original.
• Make sure you use the PyISAPIe Isapi.py from Examples\WSGI, NOT Examples\Django.
• Remove references in Isapi.py to anything you don't have, and either fix or comment out the MD5 import so you don't get the deprecation warning.

Or I suppose you could run it on Apache, but where's the fun in that?

Javascript: JQuery DatePicker, Chrome and MAXLENGTH

Having just spent 2 hours swearing violently at my computer, I thought I'd share this little bug tracking story...

I am writing a simple data entry page that contains a couple of date fields, for which I'm using the very nice JQuery UI datepickers.

I have used them all over the place with no problems at all, however on this page I was getting some weird behaviour when the submit button was clicked:

1. If the date fields were empty, the form submitted fine.
2. But if a date had been selected, when you clicked submit, the date picker for the first populated date field popped up and the form did not submit.

Some testing revealed this only happened in Chrome - my version of which is the v6 beta release.

An update to the JQuery UI did not fix the problem.

Some debugging code revealed that on submit, the datepicker fields were being focussed - so as far as the datepicker code was concerned, it was working correctly.

However, the object inspector in Chrome showed no unwanted event listeners that could be doing this.

Some more swearing, a minimal test case and a comparison with a working page later revealed the problem - the MAXLENGTH attribute had been set, in a copy/paste debacle, to 8 - clearly too short for a dd/mm/yyyy date.

So, presumably, Chrome allows you to set content longer than the MAXLENGTH from script, but then will not let you submit the form, instead focussing the offending field - it makes perfect sense when you think about it, really.

Firefox and IE both allow you to submit the form with the longer data - although you can make an argument for that making sense too. I don't know what the W3C say on this issue.

C#: Doing stuff with Google using OAuth - an idiot's guide

I have been playing around with OAuth and connecting to Google apps. After wading through the OAuth standards and Google API docs I decided a library was the way forward, so downloaded DotNetOpenAuth.

As often seems to be the case with freely provided software, it is awesome and well documented at the reference level, but not very good at handholding you through a basic scenario; although that may be due to my preferred way of starting up a blank project and attempting to get a simple application working rather than starting up a sample project and modifying it :o)

I eventually found a good example of what I wanted to do in Samples\OAuthConsumerWpf, but having done hardly any WPF and being too old and grumpy to use lambda functions, it still took a while to get my head round it and get a simple Windows Forms app going.

This is what I did:

1. Created a new Windows Forms application in VS2008
2. Added a reference to the DotNetOpenAuth assembly.
3. Copied the files GoogleConsumer.cs and Util.cs from \Samples\DotNetOpenAuth.ApplicationBlock to my application.

4. You also need an object implementing IConsumerTokenManager. The samples use the InMemoryTokenManager, which is fine for testing - obviously you will need to re-authorise the application on each run, and a useful application would have some kind of backing store.

   The InMemoryTokenManager is located in \Samples\DotNetOpenAuth.ApplicationBlock\InMemoryTokenManager.cs but is disabled outside the samples using compiler directives; I copied it into my new project and removed the compiler directives because I'm too lazy to change the DEFINEs - and again, I prefer to start with nothing and work up to a working sample app in order to get my head round things.

Once you have all these files, you can talk to Google from a Windows Forms app with a tiny amount of code, most of which is palmed off to the GoogleConsumer object in GoogleConsumer.cs.

If you, like me, are new to OAuth programming, the sequence is as follows:

1. Create a DesktopConsumer that will talk to Google, using the GoogleConsumer.ServiceDescription and a new IConsumerTokenManager for the consumer key anonymous and secret anonymous.

   The consumer key and secret identify your application - it will not be registered with Google, so you use anonymous - see http://code.google.com/apis/accounts/docs/OAuth_ref.html#SigningOAuth.

2. Call GoogleConsumer.RequestAuthorization with this consumer and flags specifying the Google apps you want access to. This returns the URL of a page that will ask the user to authorise your application, and a request token which is used later in the process.
3. So far, you have just told Google you want to access application data; at this point, no specific user has been mentioned. You now need to start up a web browser to the URL returned by GoogleConsumer.RequestAuthorization. This will allow a user to grant access to an application, and give them a verification key. The easiest way to achieve this is to start off the browser with System.Diagnostics.Process.Start(url) and show a modal window asking for the verification key to be pasted when it is retrieved.
4. You now use the ProcessUserAuthorization of the DeskTopConsumer instance to get an access token, passing in the request token returned in step 2 and the verification code pasted in by the user.
5. Once you have the access token, you can use that and the consumer to start interacting with Google apps!

Assuming all of the required DotNetOpenAuth files have been referenced or copied in as above, here is the (very simple) code to post a blog entry on Google:

// You will need the following using directives:
using DotNetOpenAuth.OpenId.Extensions.OAuth;
using DotNetOpenAuth.OAuth;
using DotNetOpenAuth.OAuth.ChannelElements;
using DotNetOpenAuth.OAuth.Messages;

// Needed for GoogleConsumer
using DotNetOpenAuth.ApplicationBlock;

// Set up consumer
DesktopConsumer consumer = new DesktopConsumer(GoogleConsumer.ServiceDescription, new InMemoryTokenManager("anonymous", "anonymous"));

// Get the request token and user access authorisation URL
String requesttoken;
Uri authuri = GoogleConsumer.RequestAuthorization(consumer, GoogleConsumer.Applications.Blogger, out requesttoken);

// Start up a browser for the user to grant access and be given the verification key
System.Diagnostics.Process.Start(authuri.AbsoluteUri);

// I have not documented my FormGetKey, but it is a simple form with one textbox and an Ok button on 
// it that just sits there being modal until the user pastes the verification key in and clicks Ok.
FormGetKey form = new FormGetKey();
form.ShowDialog();

// Oh yes, and the form has a public property which is simply { get { return textboxVerificationKey.Text; } } :o)
String vercode = form.VerificationCode;
//

// Get the access token given the request token and the verification code
AuthorizedTokenResponse grantedAccess = consumer.ProcessUserAuthorization(requesttoken, vercode);
String accesstoken = grantedAccess.AccessToken;
//

// Now we are authorized, we have an access token, and we can do stuff
GoogleConsumer.PostBlogEntry(consumer, accesstoken, "http://mylovelyblog.blogspot.com", "OAuth test", XElement.Parse("<p>OAuth test content.</p>"));
//

The next step in the process is providing a more useful implementation of IConsumerTokenManager with a backing store and using the tokens from that. I will post a similarly lift-up-flaps level post on that when I have tried it out, and again with a web app based system.

Looking at the code now, it must be just the fact that it's Friday that I found this complicated...

J2ME: Quickie release of "lightmeter" code

I don't seem to be ever getting around to doing this and now a Java developer with a Nokia has asked me for it, so here is the full source code to the phone EXIFy lightmeter.

Well, I say "full"; it's very simple! If you have seen Java before and have the J2ME SDK you should be able to do something with it.

I will try to do a more informative post on it soon. You will obviously need Java installed and also the J2ME 3.0 SDK.

Quickly, though: /_support contains the Javascript files I ported the EXIF reader from (see this post), /src/akme/mobile/util contains maths routines missing from the mobile edition (check the source for URLs and credits), /src/expinfo contains the hastily hacked together and very horrible code for the MIDlet itself, /src/rext/graphics contains a routine for generating thumbnails that seems to exist across many blog and forum posts, and /src/rext/simpleexif is the main codebase, containing the EXIF reader port and the code for moving around EV values.

I stopped developing the code as soon as it worked for the first time, so it is a fairly parlous state :o)

ExposureInfo.zip

If you have found this post and are not a developer but just want to try the app, the Flickr group has more useful info.

XSL: xsl:element and short tags

I have had problems with XSL and short tags before when using identity transforms, but it can also be an issue doing simple element creation. Obviously as far as a parser is concerned, it makes no difference whatsoever whether you have short tags (e.g. <fish/>) or the complete form (<fish></fish>) for you empty elements, but it makes it much fatter and far less human readable - especially as the .NET System.Xml.Xsl.XslCompiledTransform at least puts closing tags on a new line for empty elements as well as ones with children.

So, if you generate a completely empty element with <xsl:element>, then the short form is used:

<xsl:template match="fish">
 <xsl:element name="{@type}"/>
</xsl:template>

However, if you add any attributes, then the complete form will be used, even though there are still no child elements:

<xsl:template match="fish">
 <xsl:element name="{@type}">
  <xsl:attribute name="type">Fish</xsl:attribute>
 </xsl:element>
</xsl:template>

The trick is to generate your element into an <xsl:variable> and then use <xsl:copy-of> to output it, which results in the short form being maintained:

<xsl:template match="fish">
 <xsl:variable name="el">
  <xsl:element name="{@type}">
   <xsl:attribute name="type">Fish</xsl:attribute>
  </xsl:element>
 </xsl:variable>
 <xsl:copy-of select="$el"/>
</xsl:template>

Easy when you know how. I couldn't find this anywhere on Google, so it's either an obvious thing that everyone but me already knows, my searching sucks, or I'm the first person to have this problem.

Following a comment by Martin Honnen on my original MSDN query about this, it seems that it is a bit more complicated; the basic problem is that XSLT does not define an output format for these situations, so it is entirely up to the processor / serialization combination.

The processor I was using to run these tests was on v2.0 of the framework, and was the deprecated System.Xml.Xsl.XslTransform object, which requires the workaround. The newer System.Xml.Xsl.XslCompiledTransform uses the short form for empty tags and so does not need the workaround.

Annoyingly, the target application uses XslCompiledTransform and therefore would have worked fine, it is only the scratchpad I use for knocking up stylesheets that uses the legacy code that caused the issue! And a quick change of typename for one object and removal of one parameter that was null anyway, and it's bloody working there too. Hurrah for wasted time.

Ah well, if someone is still using the deprecated code then maybe this will help.

J2ME: EXIF and lightmetering Part 1

I am a bit of a camera obsessive. I got into photography with a digital P&S, progressed to a DSLR, and then got into film. Since then, my camera collection has taken over several surfaces in the flat. I don't have that many, but then I don't have that many surfaces.

Most of the collection consists of old cameras; my theory is that if you want to take crisp "perfect" images, a DSLR is the way to go. This is a theory based mainly on laziness, but it works for me.

In general, the film cameras have been chosen for looks at least as much as quality. The Rolleiflex has both in spades, of course.

Most of the old cameras do not have a light meter, and many with the old selenium cell meters simply do not work any more.

I do own a couple of lightmeters, but I find they are a pain to use, and new ones are basically extremely expensive flash looking versions with exactly the same functionality.

What I have been doing it using the dSLR to take a preview shot, then transferring the settings over to whatever camera I am currently playing with. However, this becomes a bit tedious after a while, and also having, say, the Rolleiflex and the dSLR round my neck results in severe neck strain.

So, ideally, I want a very small camera than can take a preview shot with the ability to tell it the ISO and, usually, the aperture I want. I can then check the preview looks okay, and transfer the settings over with any modifications required. However, P&S cameras with these manual settings are not cheap and often have big limitations on the ISO and f/stop they allow you to set, so an alternative solution is to take the preview with whatever settings the camera wants, then allow me to easily convert them into ones I can use.

This is where my mobile comes in. It is extremely small, I always have it on me, and it has a camera with a full auto mode that takes JPEGs stuffed full with the EXIF data for the exposure. Perfect. Except that, on the phone, there is no way of actually viewing the EXIF.

However, the phone, a Sony Ericsson C903, has mobile Java, and I've seen some Java before, so I decided to give writing an app for it a shot.

It was an interesting exercise, and along the way I discovered several fun things. For example, mobile Java is one of the most frustratingly NON-device independent environments it's been my misfortune to use, the certification requirements are beyond absurd, and my phone has one of the most annoying bugs ever to have existed - when your app works perfectly and is extremely useful, having the phone randomly lose and/or corrupt the memory card it's sitting on is rather irritating.

The first bit of the code, which we have finally come to, is something that will read EXIF data out of JPEGs. I had a fish round for some libraries and found lots of great code that was either extremely big, extremely non J2ME compatible, or both.

Then I discovered this Javascript library, which I have to say, is brilliant. It's a use for client side Javascript and AJAX that simply would never have occurred to me in a million years. It is perfect for my uses because being entirely client side Javascript it uses very basic language features and is very small, ideal for conversion to Java by someone who hasn't used Java for about 8 years :o)

The code was written in the J2ME 3.0 SDK editor, which seems to have mangled the indentation for any other editor, but source code formatters are not exactly hard to find. The basic EXIF when compiled has not J2ME specific imports, so can be linked into any Java project; this is very handy for testing, obviously.

I made a couple of changes; I added X_FNumberAsString as a new data item for laziness, and created a Fraction class to help with maintaining the normal form of shutter speeds when manipulating them (a later post).

The ByteArrayWrapper class duplicates the required functionality from binaryajax.js.

EXIFReader contains the main functionality, and EXIFTags the tag names / descriptions and offsets, and also the list constants for various tags.

The classes live in the package rext.simpleexif.

It is pretty easy to use; you simply load in an image file into a byte[], then create the reader:

EXIFReader reader = new EXIFReader();
  
boolean ok = false;

try
{
 ok = reader.readEXIFFromByteArry(data);
}
catch(Exception e)
{
}

The reader object has three Hashtables for the various tags, EXIF, TIFF and GPS, and a Vector for debug messages called LOG.

Print out all EXIF tags to the console:

Enumeration exifk = reader.EXIF.keys();

while(exifk.hasMoreElements())
{
 String s = (String)exifk.nextElement();
 System.out.println(s + ": " + String.valueOf(reader.EXIF.get(s)));
}

You can get the code from the link below, including a small test console app that will print off the debug info then the EXIF data for a JPEG file.

exif.zip