Tuesday, October 22, 2013

JSON Search Results via Bing

Hello again. Still working on the same stuff, still needing to get some search results off search engines, in JSON format, automatically...

If you thought getting these results out of Google was a nightmare - Bing is 10x times the circus that Google is. For crying out loud, it seems the the Bing Search V2.0 is the old, deprecated version, while the currently available version is called... Bing Search API. Thanks for nothing. As usually with Microsoft, it's some posts in StackOverflow and some trials and errors that eventually supplied the solution.


Goto the Bing Search API page, now part of the "Azure Datamarket", since nothing in Microsoft can't go by a simple, humble name*:

To sign up for the free 5000 transactions/month program, you'll might have to sign up to the Windows Azure Marketplace. Yes, you are going to have a Microsoft account.

After signing up to this program, goto your account details:

You'll see that your Primary Account Key is listed. Keep the Primary Account Key.

And now for the somewhat annoying (IMHO) part. Unlike in the Google Search API, you can't use that key in the URL. When Bing was all about AppIDs, this was a query parameter as well. But that thing's gone. Now, you'll have to add authentication to your HTTPS GET request. This can be done by requests.

Code example:
import requests
url = "https://api.datamarket.azure.com/Bing/Search/Web?$format=json&Query=%27Xbox%27"
key = 
r = requests.get(url, auth=(key,key))
print r.status_code
print r.headers['content-type']
print r.json()

There you have it. "The basic principles of time travel are there".

Other useful links:
If you want to search for other stuff such as images etc, use the general search API:
If you want to have even more fun with their APIs, go for the whole family:
Review your datasets - how much searches are left for each of the APIs:

* DreamSpark. I rest my case.

Monday, October 21, 2013

Retrieve Google Search results in JSON format

Let's say you want to...

Write a Python script which searches a specific site for a query in Google and returns the results [in JSON format].

Sounds easy, huh? Use requests to, err, make the HTTPS requests, then beautifulsoup to parse the links out. (and what would be the easiest way to grab those links? By getting the XPath from the browser...).

Well, that's what I've thought. Wrong.

Google don't want you to do that. And when Google doesn't want you to do it, you just can't. You'll find yourself blocked from making automatic requests pretty darn fast. You'll be required to solve CAPTCHAs in order to search. Not ideal for an automatic crawler...

The impolite solution would be to write a very sophisticated crawler which uses multiple proxies to cover the search-space without seeming to suspicious. The advantage is, you'll get it done. The obvious disadvantage - it's complicated!

The polite solution would be to use Google's own tool for this job - the Custom Search. You get 100 queries per day for free, and can pay for more (or utilize more accounts - sure easier than utilizing more proxies...).

I've written this guide today while performing these actions, so I can guarentee it's working - for now. Doing this will get you to far, strange places in Google like I've never seen before. Their documentation is quite lousy, which is why I've written this guide. I hope it won't get outdated too soon. Their official documentation can be found here: https://developers.google.com/custom-search/docs/overview

The short explanation

Goto Google Cloud Console and login.
https://cloud.google.com/console
Create a new project.
Under APIs and Auth > APIs, enable the Custom Search API.
Under APIs and Auth > Registered Apps, register a new web application.
Keep the Browser Key API KEY.

Goto Google Custom Search and login.
https://www.google.com/cse/
Add a site you would like to search.
Under Edit search engine > Setup, click on the Search engine ID button. Keep it as well.

Construct the URL as follows:
https://www.googleapis.com/customsearch/v1?q=<query>&cx=<Search engine ID>&key=<API KEY>

For instance:
https://www.googleapis.com/customsearch/v1?q=ooad&cx=011540632070411059229:cvop8g3tl2q&key=AIzaSyBNND2nT0vVb4M0875cxkhcUwzepeIkXng

Appearently, this imports via code as well. (requests is required):

import requests, json
url = "https://www.googleapis.com/customsearch/v1?q=ooad&cx=011540632070411059229:cvop8g3tl2q&key=AIzaSyBNND2nT0vVb4M0875cxkhcUwzepeIkXng"    
r = requests.get(url)
data = r.json()
with open('data.json','wb') as f:
    json.dump(data,f)

The results will look like this:

{
 "kind": "customsearch#search",
 "url": {
  "type": "application/json",
  "template": "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&cref={cref?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json"
 },
 "queries": {
  "request": [
   {
    "title": "Google Custom Search - ooad",
    "totalResults": "3",
    "searchTerms": "ooad",
    "count": 3,
    "startIndex": 1,
    "inputEncoding": "utf8",
    "outputEncoding": "utf8",
    "safe": "off",
    "cx": "011540632070411059229:cvop8g3tl2q"
   }
  ]
 },
 "context": {
  "title": "Cs.bgu.ac.il"
 },
 "searchInformation": {
  "searchTime": 0.247919,
  "formattedSearchTime": "0.25",
  "totalResults": "3",
  "formattedTotalResults": "3"
 },
 "items": [
  {
   "kind": "customsearch#result",
   "title": "ADSS122 (Spring 2012)",
   "htmlTitle": "ADSS122 (Spring 2012)",
   "link": "http://www.cs.bgu.ac.il/~guyrap/adss122/",
   "displayLink": "www.cs.bgu.ac.il",
   "snippet": "OOAD Winter 2012's Moed Aleph Solution: PDF, pptx. (10.07.2012) New OCL   Examples: PDF, docx. Preparation for Moed Bet, ADSS Spring 2012: PDF, pptx.",
   "htmlSnippet": "\u003cb\u003eOOAD\u003c/b\u003e Winter 2012's Moed Aleph Solution: PDF, pptx. (10.07.2012) New OCL \u003cbr\u003e  Examples: PDF, docx. Preparation for Moed Bet, ADSS Spring 2012: PDF, pptx.",
   "cacheId": "WC8B134UUl0J",
   "formattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/",
   "htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/"
  },
  {
   "kind": "customsearch#result",
   "title": "Guy Rapaport - CS Webpage",
   "htmlTitle": "Guy Rapaport - CS Webpage",
   "link": "http://www.cs.bgu.ac.il/~guyrap/",
   "displayLink": "www.cs.bgu.ac.il",
   "snippet": "Mar 21, 2013 ... [HighLearn]; Teach-assist the OOAD and ADSS courses (under Dr. Arnon Sturm).   (HighLearn , additional resources) (2011-2012, 4 semesters).",
   "htmlSnippet": "Mar 21, 2013 \u003cb\u003e...\u003c/b\u003e [HighLearn]; Teach-assist the \u003cb\u003eOOAD\u003c/b\u003e and ADSS courses (under Dr. Arnon Sturm). \u003cbr\u003e  (HighLearn , additional resources) (2011-2012, 4 semesters).",
   "cacheId": "YXe3atmPI-8J",
   "formattedUrl": "www.cs.bgu.ac.il/~guyrap/",
   "htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/"
  },
  {
   "kind": "customsearch#result",
   "title": "הכנה למועד ב'",
   "htmlTitle": "הכנה למועד ב'",
   "link": "http://www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx",
   "displayLink": "www.cs.bgu.ac.il",
   "snippet": "OOAD – סתיו 2012. במערכת ניהול הסטודנטים לתואר שני (תקרא מעתה המערכת) נדרש   לספק תמיכה לכלל הנוגעים לטיפול בסטודנטים הלומדים לתואר שני באוניברסיטה. אלו כוללים   את ...",
   "htmlSnippet": "\u003cb\u003eOOAD\u003c/b\u003e – סתיו 2012. במערכת ניהול הסטודנטים לתואר שני (תקרא מעתה המערכת) נדרש \u003cbr\u003e  לספק תמיכה לכלל הנוגעים לטיפול בסטודנטים הלומדים לתואר שני באוניברסיטה. אלו כוללים \u003cbr\u003e  את \u003cb\u003e...\u003c/b\u003e",
   "cacheId": "rcFEb9K_Sq4J",
   "mime": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
   "fileFormat": "Microsoft Powerpoint",
   "formattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx",
   "htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx"
  }
 ]
}


The long explanation (with pictures)
1. Goto Google Cloud Console and login. https://cloud.google.com/console
2. Create Project. I used the default settings.
3. This is the newly created project overview page. Click on APIs & auth > APIs on the left-hand sidebar.
4. Go through the list of APIs and turn on the Custom Search API.
5. Go to the Registered apps. Click on the "REGISTER APP" button.
6. Register your new app. I used "Web Application".
7. This is the newly created app's overview page. Open up the Browser Key tab and keep the API KEY.
The first part -obtaining an API KEY - is done.
8. Goto the Google Custom Search page. "Create  a custom search engine". You should already be logged in after your activity in the Google Cloud Console.

9. In the CSE creation page, enter the sites you would like to custom-search. "Create".
10. For example:
(this is my page on BGU's CS dept. web server.)
11. "You've successfully created your Custom search engine". Why the capital C on custom??
Click on "Edit search engine" in the left side-bar.
12. Under details, click on the "Search engine ID". Keep the Search engine ID

As I've said in the short explanation (see above) -construct your search query URL and enjoy.

Sunday, October 20, 2013

Ignoring lid close on Xubuntu 13.10 / xfce

Yesterday I've installed Xubuntu 13.10 (the low-resources version of Ubuntu which fits older, weaker machines) on my 10" laptop. I intend to work with the laptop plugged into a large screen. Naturally, after I connected the laptop to the screen I closed the lid. Whoops! The computer became suspended...

To make a long story short, it's not the BIOS fault, and trying to solve this via xfce's Settings Manager did not work. Or at least, did not do enough. Any suggestions I've read online on how to solve it (dconf editing, gconf editing, writing acpid handlers...) did not do the trick.

The solution came from the arch linux forums:
https://bbs.archlinux.org/viewtopic.php?id=154914

Edit the logind.conf file (I found it on /etc/systemd/logind.conf), uncomment the HandleLidSwitch=suspend line and change it to HandleLidSwitch=ignore.

sudo nano /etc/systemd/logind.conf
-#HandleLidSwitch=suspend
+HandleLidSwitch=ignore

Done! I've rebooted the system and everything worked like magic.

Thursday, July 18, 2013

Getting to know MEF - Meet the DefaultDependencyResolver

When I started working for my current employer, the system we were working on was an ASP.NET MVC application. I was new to C#, new to ASP.NET, and never heard of the MVC framework ("MVC? You mean the architecture of...?").

8 months later, I wouldn't say I got it all, but one thing I did get to know which was also interesting was the MEF Dependency Injection framework. MEF 2 is just coming out and it's too early for me to discuss even the first version. What I want to write about today is Dependency Injection. I will review the Dependency Resolver that ships with MVC by default in order to illustrate how simple (read: useless) it is. This will be a good starting point on why MEF is useful.

So what is dependency injection?

The best description for Dependency Injection (DI) I've found so far appears on the Google Guice project description:
Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice's @Inject as the new new. You will still need to write factories in some cases, but your code will not depend directly on them. Your code will be easier to change, unit test and reuse in other contexts.
Guice's Motivation page elaborates on what DI is and how an API for it would look. Whether you'll be using Guice, Ninject, Autofac, Castle Windsor or MEF - basically it's the same thing all over again. (Sorry, rival developers of different DI implementations!)

Martin Fowler writes about Dependency Injection too, and gives an example of moving an object's dependencies into the container, then using some "container" (more on those later) to supply the dependencies and construct the object.

So, when do we program without using new? In ASP.NET MVC, the most common use-case is when our application is online and someone enters some controller's action. When an incoming request is handled, a new constructor instance must be created. We're not the one writing the code for it - MVC does it "by itself". And if we're complicating our controller's constructor by adding dependencies for it (such as a database context, a provider or a repository class), MVC still has to create it somehow...

MVC creates the controllers?

Yep. And I can prove it to you.

Start Visual Studio (2012 in my case), and start a new C# Project from the "ASP.NET MVC 3 Web Application" template. You'll get the example project with a HomeController. Add a default constructor (that is, a parameterless constructor) to the controller, put a breakpoint on its end and debug the application. When the breakpoint is reached (quite immediately), examine the Call Stack. If you're only seeing 2 calls, right click inside the Call Stack window and enable the "Show External Code" feature.

I'm posting the entire call stack to make one clear point here: although it's looks like lots and lots of details, it can be decyphered. It took me a long time to stop seeing this as "details" and start to actually make things out of it. This was accomplished thanks to two people: my boss, and the author of JustDecompile, the C# Decompiler that everybody love-love-loves.

Below is what you will see. The colors were, of course, added by me:

> MvcApplication2.dll!MvcApplication2.Controllers.HomeController.HomeController() Line 15 C#
  [Native to Managed Transition] 
  [Managed to Native Transition] 
  mscorlib.dll!System.RuntimeType.CreateInstanceSlow(bool publicOnly, bool skipCheckThis, bool fillCache, ref System.Threading.StackCrawlMark stackMark) + 0x72 bytes 
  mscorlib.dll!System.Activator.CreateInstance(System.Type type, bool nonPublic) + 0x54 bytes 
  mscorlib.dll!System.Activator.CreateInstance(System.Type type) + 0x7 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver.GetService(System.Type serviceType) + 0x1c bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.DefaultControllerActivator.Create(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x2a bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x51 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController(System.Web.Routing.RequestContext requestContext, string controllerName) + 0x4b bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.ProcessRequestInit(System.Web.HttpContextBase httpContext, out System.Web.Mvc.IController controller, out System.Web.Mvc.IControllerFactory factory) + 0xc5 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest.AnonymousMethod__2() + 0x32 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust.AnonymousMethod__a() + 0xe bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.GetCallInAppTrustThunk.AnonymousMethod__0(System.Action f) + 0x8 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(System.Action action) + 0x17 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(System.Func func) + 0x59 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase httpContext, System.AsyncCallback callback, object state) + 0x63 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext httpContext, System.AsyncCallback callback, object state) + 0x33 bytes 
  System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, object extraData) + 0x11 bytes 
  System.Web.dll!System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() + 0x227 bytes 
  System.Web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, ref bool completedSynchronously) + 0x9c bytes 
  System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error) + 0x342 bytes 
  System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb) + 0x60 bytes 
  System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context) + 0xbb bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f3 bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes 
  [Native to Managed Transition] 
  [Managed to Native Transition] 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x350 bytes 
  System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes 
  [Appdomain Transition]

In this stack you can see all the crazy stuff that your incoming request triggers. This is a lot of code to actually go through, even with a decompiler at hand, so we'll only be roughly speaking about what happens. Below are the most interesting steps. remember, the top of the stack is the last thing to happen, so the items in this list appear in the opposite order to their presentation above.

1) System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController (System.Web.Routing.RequestContext requestContext, string controllerName)

Upon the incoming request (to /Home/Index), MVC's DefaultControllerFactory is requested to supply the appropriate controller (HomeController in this case). Note that the requestContext, out of which the controllerName was obtained, are the parameters. The controllerName ("Home") is parsed by MVC from the request URL (more here - old but still quite accurate). The next steps would link this name to the respective controller's classname (really MvcApplication2.Controllers.HomeController, since my dummy project is named MvcApplication2), and instantiate the class.
2) System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver .GetService(System.Type serviceType)

After the controller's Type is determined from its string name (details left to the reader), the CreateController method tried to create it. And who's job is it? Finally, the DependencyResolver! And not any DependencyResolver, but the DefaultDependencyResolver. Decompilation reveals that the DefaultDependencyResolver is an inner class of the DependencyResolver.

How does the DefaultDependencyResolver does this? Namely, how does the GetService implementation looks like? Let's decompile and see:



Looks like the DefaultDependencyResolver just calls Activator.CreateInstance(Type type) . So what does Activator.CreateInstance do? (pssst! It calls Activator.CreateInstance(Type type, bool nonPublic) with the same type and false for a boolean. Below is this second method's code:)
Voila!

Now we've got it! The DefaultDependencyResolver will invoke methods that will finally CreateInstanceDefaultCtor - that is, can only use parameterless constructors!!!

Let's go through the last 2 steps, just to make sure this is really happenning:

3) mscorlib.dll!System.RuntimeType.CreateInstanceSlow(...parameters...)
The depdendency resolver makes the CLR core (we're in mscorlib.dll now...) create the instance, with all of its weird parameters...

4) MvcApplication2.dll!MvcApplication2.Controllers.HomeController
      .HomeController() Line 15 C#
...and finally, the constructor of the right type (our HomeController) is invoked, and this is where our brakepoint was placed (line 15 in the HomeController.cs file).

Dependency Injection in MVC

The conclusion of this journey within the assemblies shows that dependency injection is built into MVC. By default, the actual construction of controllers is separated from the code the requires these controllers. The responsibility for obtaining the controller instance is delegated to the DefaultDependencyResolver. This is exactly what the DI pattern aims at. Too bad the DefaultDependencyResolver is so weak...

Show it to me again, but in the simplest possible way!

Here goes. This is the sole class in new, empty C# project:

So what happens in this single line of code?

The DependencyResolver class' Current public property is invoked, and once the current DependencyResolver is obtained, its method GetService<T> is called (with Foo being T). This is really the DefaultDepepdencyResolver we've seen earlier.

But what if Foo wouldn't have a default constructor? In that case, the CreateInstanceDefaultCtor method we've seen before would throw an exception: System.MissingMethodException: No parameterless constructor defined for this object. Add a new constructor to the controller that has some parameters (even an int would suffice) and you'll see it happening in an MVC application as well.

Conclusion: So why using the DefaultDependencyResolver?

If you're writing ASP.NET MVC applications, the (Default)DependencyResolver is a given fact. Sure, you can replace the default controller factory, but a) Why would you do that? and b) there's a chance even your custom controller factory will use a DI of some sort.

DefaultDependencyResolver implements the IDependencyResolver interface, which is also a part of the MVC assembly. Here's what it can do:

And since the DependencyResolver (not an implementor of the interface, by the way!) has a public SetResolver method, what we see here really is an extension point.

See, the MVC people know their DependencyResolver isn't what you need! It just serves as a placeholder for something more exciting and capable. Something like MEF, Castle Windsor, Ninject, etc. To start understanding MEF, one should first understand how life without MEF would look like. What MEF has come to replace.



Download JustDecompile from Telerik: http://www.telerik.com/products/decompiler.aspx

For the code snippet appearing above I've used Alex Gorbatchev's Syntax Highlighter, which I incorporated into Blogger by following Tahir Hassan's instructions. I won't be moving to gists since I don't have anything important enough to be forked etc.

Sunday, July 7, 2013

Enumerating Objects in an ActiveDirectory OU via C#

The goal.

My corporation employes Microsoft Active Directory to control (for lack of better word) its users and machines.

I have several computer labs under my control, and I want to list all of the machines in these labs, defined as Organizational Units in my domain's Active Directory.

I want this done via C#.

The straightforward solution.

Use this code:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19

Which requires an OuDn parameter, which you can find (for your OU of interest) using this tool:
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

Voila!

And now for the back story.

I often intend to post stuff in this blog, but then give up. I got to implement some cool stuff lately. But sadly, all of these things were already done better by others, or will be made easier by the framework in the future. One example is branding (that is, matching the language and culture, texts and CSS for users entering your site from different domains). The second is a synchronous service bus. We originally intended to use it until we'll need something "real" such as RabbitMQ; but now it seems we'll resort to using some readymade library anyway.

So this post is not about anything cool, but about something that is 1) terribly annoying and 2) will probably not be improved in its future versions. Why? Because it's delving into the Active Directory of my workplace and extracting some information out. It's Microsoft, it's Active Directory (which is legacy-code all over AFAIK), and it's not intuitive in any way.

Let's make things clear - I am not an expert on Active Directory, nor do I want to be. From my point of view, as a (somewhat privileged) user of this system but not its maintainer here in any way - this is just the odd cloud of Microsoft nonsense that hovers over the network and make authentication work. I was exposed to some of its nice features (such as group policy), but I came to deal with it for a particular reason (more below) and I couldn't care less about its internals until after I get this thing working. With some other technology I would gladly read some documentation or open a book. But in a jungle-o-spaghetti such as Active Directory - no way!

So let's break it down.

After some Googling and executing failing code from different sources, I got to this example - "Howto: (Almost) Everything In Active Directory via C#" by thund3rstruck:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19

This article from 2008 is still relevant today - more than 5 years later. Note that according to this article, most of the MSDN examples are irrelevant or incorrect. Surprising...

For this code to execute, one must instantiate a DirectoryEntry object:
new DirectoryEntry("LDAP://" + OuDn);

with the OuDn parameter defined like this:

The parameter OuDn is the Organizational Unit distinguishedName such as OU=Users,dc=myDomain,dc=com

As a user I have access the Active Directory Users and Computers tool installed on our server which queries the corporate Active Directory server. I know exactly what I want - an Organizational Unit called "Labs", which contains several OUs - each representing a computer lab, and each of the labs contents - the machines in this lab. But what are the OuDn's for this?

Luckily, some good guys in the SysInternals team came out with this tool - Active Directory Explorer v1.44:http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

This tool allows you to browse through the Active Directory tree - but it gives you the OuDn's for everything you go through!

Using the OuDn with thund3rstruck's code, I could obtain the list of machines I wanted. And then I could pass them to some PowerShell script which did some maintenance over them. Nice!

As for the code - again, do enter this link:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
I used the code exactly as he posted it - I only got the OuDn using the explorer tool.
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx

Credit for thund3rstruck and to Bryce Cogswell and Mark Russinovich of Microsoft. I just patched things together.

Tuesday, March 12, 2013

Notes on the django 1.5 tutorial01: defining mysql as your database

I'm referring to the instructions found here:
https://docs.djangoproject.com/en/1.5/intro/tutorial01/

If you'd like to use MySQL as your database backend, there are some steps you must do beforehand (besides installing MySQL itself).
  1. Add your database's settings to the settings.py file. While this step is listed on the tutorial, note that you should notice the name, username and password, as you will need to re-use them in the following steps.
    DATABASES = {
        'default':{
            'ENGINE': 'django.db.backends.mysql',
            'NAME':'mysite',
            'USER':'django',
            'PASSWORD':'django',
            'HOST':'',
            'PORT':''
        },
        'original_default': {#I've left their example here
            'ENGINE': 'django.db.backends.', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
            'NAME': '',                      # Or path to database file if using sqlite3.
            # The following settings are not used with sqlite3:
            'USER': '',
            'PASSWORD': '',
            'HOST': '',                      # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
            'PORT': '',                      # Set to empty string for default.
        }
    }
    
  2. Install the Python MySQL support:

    #http://stackoverflow.com/questions/2952187/getting-error-loading-mysqldb-module-no-module-named-mysqldb-have-tried-pre
    sudo apt-get install python-mysqldb
    sudo easy_install mysql-python
  3. Create a user/pasword pair, same as you'd put in your site's settings.py file:

    #http://dev.mysql.com/doc/refman/5.1/en/create-user.html
     mysql -uroot -p
    #and in the mysql login:
    #create user '<username>'@'localhost' identified by '<password in plaintext>';
    #for instance:
    CREATE USER 'jeffrey'@'localhost' IDENTIFIED BY 'mypass';
  4. Create a database with the same name you specified in the settings.py file and grant your user the necessary permissions:
    #http://dev.mysql.com/doc/refman/5.1/en/grant.html
    CREATE USER 'jeffrey'@'localhost' IDENTIFIED BY 'mypass';
    GRANT ALL ON db1.* TO 'jeffrey'@'localhost';
As they say in walkthroughs, this will get you as far as the models definition...

I'll post more updates if I'll come across more inconsistencies.

Sunday, July 29, 2012

Integrating LimeSurvey with Amazon Mechanical Turk (or anything that requires parsing URL GET params with JavaScript)


I resolved from writing my own suverys framework into using LimeSurvey which seemed to be the best freely available hosting. It is written in PHP and JQuery and requires a server for a host - which is something I could supply.


This solution was tested with:

  • WampServer 2.2;
  • LimeSurvey 1.9.2;
  • Browser/OS:
    • On Windows 7: Internet Explorer 9; FireFox 13; Google Chrome 20.0.1132.57 m.

Installing, configuring and start using LimeSurvey

I built it from scratch.

Thus, for WAMP I used WampServer: http://www.wampserver.com/en/

Downloaded Livesurvey from here: http://www.limesurvey.org/en/stable-release

Unzipped, moved the limesurvey directory to the WampServer www directory.

Browsed to http://localhost/limesurvey . Continued with the installation from there.

When LimeSurvey is installed, login as an admin. The l/p values are stored in the config.php file in LimeSurveuy's root directory. The defaults are admin/password.

Now. Why are we here?

I would like LimeSurvey to

  1. Read the assignmentId parameter of the URL. This is a part of the URL added by Mechanical Turk to any HIT page it refers to.
  2. Store the assignmentId as one of the answers to the questionnaire. (for further inspection before I actually pay people).
  3. Refer workers who finished the questionnaire to a confirmation link, which adds them to the list of people I should (consider) paying to.

The Steps

1) Change LimeSurvey's Global Settings \ Security \ Filter HTML for XSS to No.


2) Create a new Survey, QuestionsGroup, Question. Go to the edit question screen. The question type should be set to "Short free text" (unless you need something different). Pick a Question Code now. It can be anything not used again in your survey. In the example below, the Question Code is 'assignmentId' (without the quotes of course).

3) Click on the Source button. This will disable rich-text editing, and will take anything you'd print in as
raw code to be added into the questionnaire.

4) After you clicked on the Source button, paste the following code:

<script type="text/javascript">
// Adapted by guy4261 from: http://papermashup.com/read-url-get-variables-withjavascript/
function getUrlVars() {
var vars = new Array();
var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi,
function(m,key,value) {
        vars[key] = value;
    });
    return vars;
}

//$(document).ready(function() {
  $('#{SGQ}').val(getUrlVars()["assignmentId"]);
//});
</script>
<style type="text/css">
#question{assignmentId.qid}{
display: none;
}
</style>

5) Go to the Survey's Edit survey text elements page. Scroll down to the End URL field. Enter the following URL:
(For Debug:) http://www.google.com/search?q={assignmentId}
(For Release:) http://workersandbox.mturk.com/mturk/externalSubmit?finished=true&assignmentId={assignmentId}


Further Customization Notes

4.1) Note that you need a specific value. In my case it's assignmentId (assigned by Mechanical Turk); in a different system that could've been a completely different parameter(s).

4.2) After the script tag is closed, comes the style tag. By choosing #question1 and making it invisible, we prevent the user from changing the assigned value. The original tutorials suggest hiding a question by tampering with the template.css file, but at least in Chrome this works just as well. However, the selector should be #questionQQ where QQ is the question ID. In the example shown here, while the SGQA (http://docs.limesurvey.org/SGQA+identifier) is 22456X1X1, the question's ID is simply 1. Supposedly, question codes can replace this values starting with LimeSurvey 1.92, but this is what I have installed and I just couldn't make it work.

5) In the End URL, use the macro {questionCode} (in our case: {assignmentId}) to include the collected assignmentId in the reference URL to Mechanical Turk's confirmation page. There is probably a nicer way to make it work with JavaScript auto-forwarding etc. but that's beyond the scope of this tutorial. This also means that the assignmentId is saved along with the survey's results. However, from what I understood, this is useful to avoid paying people who did not feel the questionnaire properly.

Explanations of the Solutions' Steps

4.1) You should be aware that LimeSurvey uses lots of JQuery, hence the $ selectors in this code.
4.2) Note that in the original suggestions across the LimeSurvey forum, the original suggestion was to run
the code as part of a custom document.ready function.
4.3) The function that parses the URL variables can be replaced; I took this one by papermashup: http://papermashup.com/read-url-get-variables-withjavascript/
4.4) {SGQ} is a the LimeSurvey macro for the current question. $('#{SGQ}').val('FOO') simply populates the answer with FOO.

Further Reading

JavaScript in LimeSurvey examples: http://docs.limesurvey.org/Workarounds%3A+Manipulating+a+survey+at+runtime+using+Javascript&structure=English+Instructions+for+LimeSurvey#How_to_use_Script_eg._JavaScript_etc._in_LimeSurvey_