Getting to know MEF - Meet the DefaultDependencyResolver

When I started working for my current employer, the system we were working on was an ASP.NET MVC application. I was new to C#, new to ASP.NET, and never heard of the MVC framework ("MVC? You mean the architecture of...?").

8 months later, I wouldn't say I got it all, but one thing I did get to know which was also interesting was the MEF Dependency Injection framework. MEF 2 is just coming out and it's too early for me to discuss even the first version. What I want to write about today is Dependency Injection. I will review the Dependency Resolver that ships with MVC by default in order to illustrate how simple (read: useless) it is. This will be a good starting point on why MEF is useful.

So what is dependency injection?

The best description for Dependency Injection (DI) I've found so far appears on the Google Guice project description:

Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice's @Inject as the new new. You will still need to write factories in some cases, but your code will not depend directly on them. Your code will be easier to change, unit test and reuse in other contexts.

Guice's Motivation page elaborates on what DI is and how an API for it would look. Whether you'll be using Guice, Ninject, Autofac, Castle Windsor or MEF - basically it's the same thing all over again. (Sorry, rival developers of different DI implementations!)

Martin Fowler writes about Dependency Injection too, and gives an example of moving an object's dependencies into the container, then using some "container" (more on those later) to supply the dependencies and construct the object.

So, when do we program without using new? In ASP.NET MVC, the most common use-case is when our application is online and someone enters some controller's action. When an incoming request is handled, a new constructor instance must be created. We're not the one writing the code for it - MVC does it "by itself". And if we're complicating our controller's constructor by adding dependencies for it (such as a database context, a provider or a repository class), MVC still has to create it somehow...

MVC creates the controllers?

Yep. And I can prove it to you.

Start Visual Studio (2012 in my case), and start a new C# Project from the "ASP.NET MVC 3 Web Application" template. You'll get the example project with a HomeController. Add a default constructor (that is, a parameterless constructor) to the controller, put a breakpoint on its end and debug the application. When the breakpoint is reached (quite immediately), examine the Call Stack. If you're only seeing 2 calls, right click inside the Call Stack window and enable the "Show External Code" feature.

I'm posting the entire call stack to make one clear point here: although it's looks like lots and lots of details, it can be decyphered. It took me a long time to stop seeing this as "details" and start to actually make things out of it. This was accomplished thanks to two people: my boss, and the author of JustDecompile, the C# Decompiler that everybody love-love-loves.

Below is what you will see. The colors were, of course, added by me:

> MvcApplication2.dll!MvcApplication2.Controllers.HomeController.HomeController() Line 15 C#
  [Native to Managed Transition] 
  [Managed to Native Transition] 
  mscorlib.dll!System.RuntimeType.CreateInstanceSlow(bool publicOnly, bool skipCheckThis, bool fillCache, ref System.Threading.StackCrawlMark stackMark) + 0x72 bytes 
  mscorlib.dll!System.Activator.CreateInstance(System.Type type, bool nonPublic) + 0x54 bytes 
  mscorlib.dll!System.Activator.CreateInstance(System.Type type) + 0x7 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver.GetService(System.Type serviceType) + 0x1c bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.DefaultControllerActivator.Create(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x2a bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x51 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController(System.Web.Routing.RequestContext requestContext, string controllerName) + 0x4b bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.ProcessRequestInit(System.Web.HttpContextBase httpContext, out System.Web.Mvc.IController controller, out System.Web.Mvc.IControllerFactory factory) + 0xc5 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest.AnonymousMethod__2() + 0x32 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust.AnonymousMethod__a() + 0xe bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.GetCallInAppTrustThunk.AnonymousMethod__0(System.Action f) + 0x8 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(System.Action action) + 0x17 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(System.Func func) + 0x59 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase httpContext, System.AsyncCallback callback, object state) + 0x63 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext httpContext, System.AsyncCallback callback, object state) + 0x33 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, object extraData) + 0x11 bytes 
  System.Web.dll!System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() + 0x227 bytes 
  System.Web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, ref bool completedSynchronously) + 0x9c bytes 
  System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error) + 0x342 bytes 
  System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb) + 0x60 bytes 
  System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context) + 0xbb bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f3 bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes 
  [Native to Managed Transition] 
  [Managed to Native Transition] 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x350 bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes 
  [Appdomain Transition]

In this stack you can see all the crazy stuff that your incoming request triggers. This is a lot of code to actually go through, even with a decompiler at hand, so we'll only be roughly speaking about what happens. Below are the most interesting steps. remember, the top of the stack is the last thing to happen, so the items in this list appear in the opposite order to their presentation above.

1) System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController (System.Web.Routing.RequestContext requestContext, string controllerName)

Upon the incoming request (to /Home/Index), MVC's DefaultControllerFactory is requested to supply the appropriate controller (HomeController in this case). Note that the requestContext, out of which the controllerName was obtained, are the parameters. The controllerName ("Home") is parsed by MVC from the request URL (more here - old but still quite accurate). The next steps would link this name to the respective controller's classname (really MvcApplication2.Controllers.HomeController, since my dummy project is named MvcApplication2), and instantiate the class.
2) System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver .GetService(System.Type serviceType)

After the controller's Type is determined from its string name (details left to the reader), the CreateController method tried to create it. And who's job is it? Finally, the DependencyResolver! And not any DependencyResolver, but the DefaultDependencyResolver. Decompilation reveals that the DefaultDependencyResolver is an inner class of the DependencyResolver.

How does the DefaultDependencyResolver does this? Namely, how does the GetService implementation looks like? Let's decompile and see:

Looks like the DefaultDependencyResolver just calls Activator.CreateInstance(Type type) . So what does Activator.CreateInstance do? (pssst! It calls Activator.CreateInstance(Type type, bool nonPublic) with the same type and false for a boolean. Below is this second method's code:)
Voila!

Now we've got it! The DefaultDependencyResolver will invoke methods that will finally CreateInstanceDefaultCtor - that is, can only use parameterless constructors!!!

Let's go through the last 2 steps, just to make sure this is really happenning:

3) mscorlib.dll!System.RuntimeType.CreateInstanceSlow(...parameters...)
The depdendency resolver makes the CLR core (we're in mscorlib.dll now...) create the instance, with all of its weird parameters...

4) MvcApplication2.dll!MvcApplication2.Controllers.HomeController
      .HomeController() Line 15 C#
...and finally, the constructor of the right type (our HomeController) is invoked, and this is where our brakepoint was placed (line 15 in the HomeController.cs file).

Dependency Injection in MVC

The conclusion of this journey within the assemblies shows that dependency injection is built into MVC. By default, the actual construction of controllers is separated from the code the requires these controllers. The responsibility for obtaining the controller instance is delegated to the DefaultDependencyResolver. This is exactly what the DI pattern aims at. Too bad the DefaultDependencyResolver is so weak...

Show it to me again, but in the simplest possible way!

Here goes. This is the sole class in new, empty C# project:

So what happens in this single line of code?

The DependencyResolver class' Current public property is invoked, and once the current DependencyResolver is obtained, its method GetService<T> is called (with Foo being T). This is really the DefaultDepepdencyResolver we've seen earlier.

But what if Foo wouldn't have a default constructor? In that case, the CreateInstanceDefaultCtor method we've seen before would throw an exception: System.MissingMethodException: No parameterless constructor defined for this object. Add a new constructor to the controller that has some parameters (even an int would suffice) and you'll see it happening in an MVC application as well.

Conclusion: So why using the DefaultDependencyResolver?

If you're writing ASP.NET MVC applications, the (Default)DependencyResolver is a given fact. Sure, you can replace the default controller factory, but a) Why would you do that? and b) there's a chance even your custom controller factory will use a DI of some sort.

DefaultDependencyResolver implements the IDependencyResolver interface, which is also a part of the MVC assembly. Here's what it can do:

And since the DependencyResolver (not an implementor of the interface, by the way!) has a public SetResolver method, what we see here really is an extension point.

See, the MVC people know their DependencyResolver isn't what you need! It just serves as a placeholder for something more exciting and capable. Something like MEF, Castle Windsor, Ninject, etc. To start understanding MEF, one should first understand how life without MEF would look like. What MEF has come to replace.



Download JustDecompile from Telerik: http://www.telerik.com/products/decompiler.aspx

For the code snippet appearing above I've used Alex Gorbatchev's Syntax Highlighter, which I incorporated into Blogger by following Tahir Hassan's instructions. I won't be moving to gists since I don't have anything important enough to be forked etc.

Enumerating Objects in an ActiveDirectory OU via C#

The goal.

My corporation employes Microsoft Active Directory to control (for lack of better word) its users and machines.

I have several computer labs under my control, and I want to list all of the machines in these labs, defined as Organizational Units in my domain's Active Directory.

I want this done via C#.

The straightforward solution.

Use this code:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19

Which requires an OuDn parameter, which you can find (for your OU of interest) using this tool:
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

Voila!

And now for the back story.

I often intend to post stuff in this blog, but then give up. I got to implement some cool stuff lately. But sadly, all of these things were already done better by others, or will be made easier by the framework in the future. One example is branding (that is, matching the language and culture, texts and CSS for users entering your site from different domains). The second is a synchronous service bus. We originally intended to use it until we'll need something "real" such as RabbitMQ; but now it seems we'll resort to using some readymade library anyway.

So this post is not about anything cool, but about something that is 1) terribly annoying and 2) will probably not be improved in its future versions. Why? Because it's delving into the Active Directory of my workplace and extracting some information out. It's Microsoft, it's Active Directory (which is legacy-code all over AFAIK), and it's not intuitive in any way.

Let's make things clear - I am not an expert on Active Directory, nor do I want to be. From my point of view, as a (somewhat privileged) user of this system but not its maintainer here in any way - this is just the odd cloud of Microsoft nonsense that hovers over the network and make authentication work. I was exposed to some of its nice features (such as group policy), but I came to deal with it for a particular reason (more below) and I couldn't care less about its internals until after I get this thing working. With some other technology I would gladly read some documentation or open a book. But in a jungle-o-spaghetti such as Active Directory - no way!

So let's break it down.

After some Googling and executing failing code from different sources, I got to this example - "Howto: (Almost) Everything In Active Directory via C#" by thund3rstruck:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19

This article from 2008 is still relevant today - more than 5 years later. Note that according to this article, most of the MSDN examples are irrelevant or incorrect. Surprising...

For this code to execute, one must instantiate a DirectoryEntry object:

new DirectoryEntry("LDAP://" + OuDn);

with the OuDn parameter defined like this:

The parameter OuDn is the Organizational Unit distinguishedName such as OU=Users,dc=myDomain,dc=com




As a user I have access the Active Directory Users and Computers tool installed on our server which queries the corporate Active Directory server. I know exactly what I want - an Organizational Unit called "Labs", which contains several OUs - each representing a computer lab, and each of the labs contents - the machines in this lab. But what are the OuDn's for this?

Luckily, some good guys in the SysInternals team came out with this tool - Active Directory Explorer v1.44:http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

This tool allows you to browse through the Active Directory tree - but it gives you the OuDn's for everything you go through!

Using the OuDn with thund3rstruck's code, I could obtain the list of machines I wanted. And then I could pass them to some PowerShell script which did some maintenance over them. Nice!

As for the code - again, do enter this link:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
I used the code exactly as he posted it - I only got the OuDn using the explorer tool.
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

Credit for thund3rstruck and to Bryce Cogswell and Mark Russinovich of Microsoft. I just patched things together.

Notes on the django 1.5 tutorial01: defining mysql as your database

I'm referring to the instructions found here:
https://docs.djangoproject.com/en/1.5/intro/tutorial01/

If you'd like to use MySQL as your database backend, there are some steps you must do beforehand (besides installing MySQL itself).

1. Add your database's settings to the settings.py file. While this step is listed on the tutorial, note that you should notice the name, username and password, as you will need to re-use them in the following steps.
   

   DATABASES = {
       'default':{
           'ENGINE': 'django.db.backends.mysql',
           'NAME':'mysite',
           'USER':'django',
           'PASSWORD':'django',
           'HOST':'',
           'PORT':''
       },
       'original_default': {#I've left their example here
           'ENGINE': 'django.db.backends.', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
           'NAME': '',                      # Or path to database file if using sqlite3.
           # The following settings are not used with sqlite3:
           'USER': '',
           'PASSWORD': '',
           'HOST': '',                      # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
           'PORT': '',                      # Set to empty string for default.
       }
   }
   

2. Install the Python MySQL support:
   
   #http://stackoverflow.com/questions/2952187/getting-error-loading-mysqldb-module-no-module-named-mysqldb-have-tried-pre
   sudo apt-get install python-mysqldb
   sudo easy_install mysql-python
3. Create a user/pasword pair, same as you'd put in your site's settings.py file:
   
   #http://dev.mysql.com/doc/refman/5.1/en/create-user.html
    mysql -uroot -p
   #and in the mysql login:
   #create user '<username>'@'localhost' identified by '<password in plaintext>';
   #for instance:

   CREATE USER 'jeffrey'@'localhost' IDENTIFIED BY 'mypass';

4. Create a database with the same name you specified in the settings.py file and grant your user the necessary permissions:
   #http://dev.mysql.com/doc/refman/5.1/en/grant.html
   

   CREATE USER 'jeffrey'@'localhost' IDENTIFIED BY 'mypass';
   GRANT ALL ON db1.* TO 'jeffrey'@'localhost';

As they say in walkthroughs, this will get you as far as the models definition...

I'll post more updates if I'll come across more inconsistencies.

Integrating LimeSurvey with Amazon Mechanical Turk (or anything that requires parsing URL GET params with JavaScript)

I resolved from writing my own suverys framework into using LimeSurvey which seemed to be the best freely available hosting. It is written in PHP and JQuery and requires a server for a host - which is something I could supply.


This solution was tested with:

• WampServer 2.2;
• LimeSurvey 1.9.2;
• Browser/OS:
• On Windows 7: Internet Explorer 9; FireFox 13; Google Chrome 20.0.1132.57 m.

Installing, configuring and start using LimeSurvey

I built it from scratch.

Thus, for WAMP I used WampServer: http://www.wampserver.com/en/

Downloaded Livesurvey from here: http://www.limesurvey.org/en/stable-release

Unzipped, moved the limesurvey directory to the WampServer www directory.

Browsed to http://localhost/limesurvey . Continued with the installation from there.

When LimeSurvey is installed, login as an admin. The l/p values are stored in the config.php file in LimeSurveuy's root directory. The defaults are admin/password.

Now. Why are we here?

I would like LimeSurvey to

1. Read the assignmentId parameter of the URL. This is a part of the URL added by Mechanical Turk to any HIT page it refers to.
2. Store the assignmentId as one of the answers to the questionnaire. (for further inspection before I actually pay people).
3. Refer workers who finished the questionnaire to a confirmation link, which adds them to the list of people I should (consider) paying to.

The Steps

1) Change LimeSurvey's Global Settings \ Security \ Filter HTML for XSS to No.

2) Create a new Survey, QuestionsGroup, Question. Go to the edit question screen. The question type should be set to "Short free text" (unless you need something different). Pick a Question Code now. It can be anything not used again in your survey. In the example below, the Question Code is 'assignmentId' (without the quotes of course).

3) Click on the Source button. This will disable rich-text editing, and will take anything you'd print in as
raw code to be added into the questionnaire.

4) After you clicked on the Source button, paste the following code:

<script type="text/javascript">
// Adapted by guy4261 from: http://papermashup.com/read-url-get-variables-withjavascript/
function getUrlVars() {
var vars = new Array();
var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi,
function(m,key,value) {
        vars[key] = value;
    });
    return vars;
}

//$(document).ready(function() {
  $('#{SGQ}').val(getUrlVars()["assignmentId"]);
//});
</script>
<style type="text/css">
#question{assignmentId.qid}{
display: none;
}
</style>

5) Go to the Survey's Edit survey text elements page. Scroll down to the End URL field. Enter the following URL:
(For Debug:) http://www.google.com/search?q={assignmentId}
(For Release:) http://workersandbox.mturk.com/mturk/externalSubmit?finished=true&assignmentId={assignmentId}

Further Customization Notes

4.1) Note that you need a specific value. In my case it's assignmentId (assigned by Mechanical Turk); in a different system that could've been a completely different parameter(s).

4.2) After the script tag is closed, comes the style tag. By choosing #question1 and making it invisible, we prevent the user from changing the assigned value. The original tutorials suggest hiding a question by tampering with the template.css file, but at least in Chrome this works just as well. However, the selector should be #questionQQ where QQ is the question ID. In the example shown here, while the SGQA (http://docs.limesurvey.org/SGQA+identifier) is 22456X1X1, the question's ID is simply 1. Supposedly, question codes can replace this values starting with LimeSurvey 1.92, but this is what I have installed and I just couldn't make it work.

5) In the End URL, use the macro {questionCode} (in our case: {assignmentId}) to include the collected assignmentId in the reference URL to Mechanical Turk's confirmation page. There is probably a nicer way to make it work with JavaScript auto-forwarding etc. but that's beyond the scope of this tutorial. This also means that the assignmentId is saved along with the survey's results. However, from what I understood, this is useful to avoid paying people who did not feel the questionnaire properly.

Explanations of the Solutions' Steps

4.1) You should be aware that LimeSurvey uses lots of JQuery, hence the $ selectors in this code.
4.2) Note that in the original suggestions across the LimeSurvey forum, the original suggestion was to run
the code as part of a custom document.ready function.
4.3) The function that parses the URL variables can be replaced; I took this one by papermashup: http://papermashup.com/read-url-get-variables-withjavascript/
4.4) {SGQ} is a the LimeSurvey macro for the current question. $('#{SGQ}').val('FOO') simply populates the answer with FOO.

Further Reading

JavaScript in LimeSurvey examples: http://docs.limesurvey.org/Workarounds%3A+Manipulating+a+survey+at+runtime+using+Javascript&structure=English+Instructions+for+LimeSurvey#How_to_use_Script_eg._JavaScript_etc._in_LimeSurvey_

Solving the powernow-k8: transmission frequency failed in Xen , Ubuntu 12.04 , AMD

Did this occur to you? It occured to me on Ubuntu Server 12.04 , on a machine with AMD processors, while installing a Xen Hypervisor dom0 :

powernow-k8: fid trans failed, fid 0x2, curr 0x0
powernow-k8: transmission frequency failed

If so, you might've read the posts here:
https://bugzilla.redhat.com/show_bug.cgi?id=739159
http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1789

and like me, did not understand what you should do.

What I did was to read here:
http://ubuntuguide.org/wiki/Ubuntu:Precise#Disable_CPU_Frequency_scaling

And follow their instructions:

sudo apt-get install rcconf dialog
sudo rcconf

Running rcconf displays a screen somewhat like this: (it's different on each machine, depending on it's available rc scripts)

(image taken from http://www.intelligrape.com/blog/tag/rcconf/ , direct link: http://www.intelligrape.com/blog/wp-content/uploads/2010/07/rcconf.png , used without permission.)

Removing acpi-support solved that for me. xen used to start spitting the powernow-k8 error less than a minute after I logged in. As of now, it's 10 minutes and everything seems to be working just fine...

sudo reboot when you are done.

The good part is that, well, it's working.

The bad part is that we don't have ACPI support. But since this bug is known since last year and I couldn't find any decent solution, I gave up. Hope I made the right choice...

Customizing Google Docs Forms - Part 3

(This is the third part. Part 1. Part 2.)

Time For Some Code

Save the Google Docs Form page (viewform) to a local file on c:\wamp\www\viewform.php .

Changed the form action to point to a local page. viewpost.php:

<form action="viewpost.php?assignmentId=dog" method="POST" id="ss-form">

When I browse to http://localhost/viewform.php , I get a page looking exactly like my Google Docs Form, thanks to the fact that the CSS files was saved with the page. I created a page viewpost.php with the following code:


<?php
print("<pre>");
print_r($_POST);
print_r($_GET);
print("</pre>");
?>

Finally, I entered the form on localhost and submitted it. Here is the output:

Array
(
    [entry_68_single_yodawg] => 
    [entry_63_single] => 
    [pageNumber] => 0
    [backupCache] => 
    [submit] => Submit
)

Array
(
    [assignmentId] => dog
)

What can we learn from this?

1. PHP makes accessing POST or GET (URL) parameters very easy. As easy as $_POST['param-name'].
2. For every input element in the original form, it's name attribute is used as the parameter name.
3. For radio button groups in which no selection was made, the matching parameter does not appear in the $_POST variable.
4. And after some validations against the code - dots are not allowed in the parameter names, so they are converted to underscores in the $_POST array. "entry_63_single" in $_POST is originally "entry.63.single" in the HTML.

So what now?

First, we need to define a protocol for describing a questionnaire. In Google Docs, you get a WYSIWYG editor and things are very easy. In this project's case, I am not interested in creating such an editor. My employers did not allocate such amounts of money for this project - and something simpler would suffice. I am thinking about an Excel spreadsheet which they'd write according to my guidelines. I will convert the spreadsheet into a TSV file which Python will handle.


We'll use this protocol to generate the questionnaire. Again, that means two php pages:

One just for displaying the form. That's not so hard, we've sort of got it done.

The second is for validating the form, insert the data into the SQL server if it's good or display it with error boxes if necessary. Both should have their HTML elements tagged in accordance with Google Docs' CSS file so they would look good. (The distances I'd walk to avoid styling on my own are immense, I assure you.)

Another thing we'll need to generate is SQL code for defining the table properly. Creating the database could also be nice but that's less important for me right now. Defining the table, though, is crucial.

The display, validation , insertion and table definition are all dependant on the protocol. Therefore, my solution is to iterate over the protocol-based input file (probably a TSV adapted from excel) and for each row, parse it and generate PHP code for each of these targets.

On that note, creating an interface for vieweing or downloading the results can be nice. wamp server comes with phpmyadmin installed, but something more friendly for the researchers is always a pleasure.

Viewing

The viewer was the easiest part to make and you can find it here: https://gist.github.com/3049801

It's mostly based on this post: http://www.anyexample.com/programming/php/php_mysql_example__display_table_as_html.xml

This php script

• Pulls the table data as a TSV, independantly of the table's schema.
• Makes your browser automatically download it.
• Can get dbname and table as GET parameters (in the URL).

It works when the connection details (server address, username, password) are correctly set; and when the mysql server itself has permissions for that user over the requested table.

Since it's already 02:25, I think I'll go to sleep now... Later this week I'll be working on creating prepared SQL statements and binding them with the incoming POST parameters.

Happy hacking!

Customizing Google Docs Forms - Part 2

(This is the second part. Part 1.)

Google Docs Forms - Exploring the Front-End

Let's recap. What we are about to do in this part is to save a Google Docs Form entirely (HTML+CSS), write PHP scripts which will replace Google's own back-end, and set the form to work with these scripts.

This is the form in question:
https://docs.google.com/spreadsheet/viewform?formkey=dDJ5VDB3aURJSVN3NThjdmE2M1pLbGc6MQ#gid=0

Note that it's actually a link to https://docs.google.com/spreadsheet/viewform with two parameters in its URL - formkey and gid. Navigating directly to the viewform script returns an error; we can't really read this from Google. Which is why I decided to use PHP for this job - it just reminded me of how PHP behaves, always responding to a GET request with generated HTML.

This is a questionnaire with 50 questions. Working with Chrome, you can hit the F12 key to show the developer window. From the Console tab, you can enter this code to check all radio buttons to 1:

_radios = document.getElementsByClassName('ss-q-radio');for(i = 0; i < _radios.length; i++){ _radio = _radios[i]; if (_radio.value == "1") { _radio.checked = true; }}

(this post is written after I examined the page's source-code, which is how I got this code written.)

Nevertheless, let's not do that right away; go to the bottom of the page and submit the form as it is. You'll get to an error screen:

https://docs.google.com/spreadsheet/formResponse?formkey=dDJ5VDB3aURJSVN3NThjdmE2M1pLbGc6MQ&ifq

which is the same screen you'll get when you submit the form correctly.

So now, it's clear the two scripts are working here: viewform and formResponse. The first displays the form initially; the second validates the input, and either displays the form with the necessary error marks or simply displays a success message. Google being our model for good design, this is the template we are about to follow.

A third and unseen factor here is the SQL table. In Google Docs, creating a form automatically creates the linked Google Docs Spreadsheet which contains all the entries of people who filled the form. This is a very user-friendly view of the data. Unfortunately for our researcher, I will not supply such a nice interface. I can use the Google Docs spreadsheet, though

Into the Form's Source

Viewing the source-code of our form, a couple of points pop out.

First of all, for an automatically-generated page, the code is very need. It contains a single HTML <form>, refers to only one CSS stylesheet, and contains just one snippet of Javascript code, which is here below.

      (function() {var divs = document.getElementById('ss-form').getElementsByTagName('div');var numDivs = divs.length;for (var j = 0; j < numDivs; j++) {if (divs[j].className == 'errorbox-bad') {divs[j].lastChild.firstChild.lastChild.focus();return;}}for (var i = 0; i < numDivs; i++) {var div = divs[i];if (div.className == 'ss-form-entry' &&div.firstChild &&div.firstChild.className == 'ss-q-title') {div.lastChild.focus();return;}}})();

 What this snippet does, is to focus the page on either the title of the Form or on the highest errorbox (returned by formResponse when a required field is missing). It is placed near the end of the page's <body> and is designated to run as soon as the rest of the page has been parsed.

These kind of things are exactly the reason why I chose to scrape everything from Google Docs. I've done a couple of relatively small projects regarding web design and have come very quickly to the conclusion, that it's very hard to write HTML and CSS which display well on all browsers and on all platforms. I take this code from Google since I trust them. If anyone get do it right - I believe it is them. When I go to the Form page from Chrome, I believe I get the same page I'd get when I'd browse from either IE or Firefox; and it will display just right on all three.

To sum everything up, the design will should be as follows:

• viewform - will display the initial form. While Google's version takes the formkey parameter to identify the form, we can serve static HTML. The <form> element's action should be our formResponse script.
• formResponse - given the form data by the POST method, this page should:
• If all required fields have been filled: save the data to the SQL. Display a link or re-direct to Turk's assignment approval URL...
• Otherwise, if some inputs are missing: display the questionnaire again. If we choose to highlight the problematic parts (just as Google Docs does), that means some tampering with the returned HTML is required - we can't just serve static content. The effort is very close to generating the entire page content dynamically according to the questionnaire's structure. Perhaps giving up on this question-specific error highlighting is the wise thing to do right now...

A small note I forgot to mention on the previous post. I've mentioned the mandatory reason for writing our own code is to hook up with Turk. But now that we have control over the form using Javascript et al., we can do some other cool things. For instance - a Youtube video is being showed to our test subjects. We can use the Youtube Javascript API to make the form's submit button active only after the movie has been displayed. Also, we can verify the time it took our test subjects to complete the form, which is useful for making sure they didn't do it too fast. That's a nice benefit.

So what's next?

1. Write a PHP page which returns a form, which redirects its data to a second PHP page.
2. Write the second PHP page which can observe the data sent by the first and decided if it's OK or not.
3. If data is missing, have the second PHP page display an error message.
4. If the data is OK, write it to an SQL table and display a link to Turk's confirmation URL.

So this is it - this is the point where I should really start writing some code. The default location for WampServer's index page is C:\wamp\www and this is exactly where I am heading.

Wait for the next post for some code examples.