Shameless self-promotion: this is a (Hebrew) article about the new Hadoop system we built in BGU.
http://in.bgu.ac.il/Pages/news/Hadoop.aspx
Couldn't do it without reading posts by Michael G. Noll. Start with his posts about installing Hadoop if you ever want to get it done right.
http://www.michael-noll.com/
Yay!
Tuesday, January 14, 2014
Wednesday, December 25, 2013
python + pylint + sublime text = pylinton
I write Python with Notepad++ or Geany, and got sick of getting NameErrors which could have been foreseen. The simplest example is
#foo.py
def foo():
print x
foo()
But imagine something more nasty happening because of your own stupidity which pops out in a function that's been running for a good hour's time.
So, yeah, tests and everything... But can't there be a way to pay a bit less for such a stupid error?
Introducing: pylint. Like jslint which I've already met, pylint checks your Python code for stupid mistakes such as this. You can get it here: http://www.pylint.org/ or using pip install pylint.
You can test your pylint installation using the example from above: pylint -E foo.py will output only the most important errors, and will of course exit with a nonzero status code.
Nice! But how would I add this to my editors?
Introducing(2): Sublime Text. Word on the street (or on the train back home with Micky who told me about it) is that it's the coolest new lightweight editor. We don't want to write Python on Eclipse and PyDev, now, do we...?
You can get it here: http://www.sublimetext.com
Now apparently Sublime Text can build your code using all sorts of builders, including predefined! While the language for defining such builds is not very sophisticated (i.e. no conditionals), you can write a little bash script to go with it so that whenever you build your code, first pylint will go over it and won't let it run if it's buggy.
Say you put the following code in /home/you/pylinton :
#!/bin/bash
pylint -E $1 2>/dev/null
if [ $? -eq 0 ]
then
python $1
fi
Then, from Sublime Text, go to Tools >> Build System >> New Build System and use this definition:
{
"cmd": ["/home/you/pylinton" , "$file"]
}
Then voila! Ctrl+B builds your currently open Python source code. You'll get pylint's errors if there are any; otherwise, Python will execute. That won't replace your tests but it can save you some time and heartaches when you're running ahead.
tl;dr
(1/4) pylint is jslint for Python. http://www.pylint.org/ or using pip install pylint
(2/4) sublime editor is your new cross-platform notepad++ . http://www.sublimetext.com
That was a bit long as well imho!!!
#foo.py
def foo():
print x
foo()
But imagine something more nasty happening because of your own stupidity which pops out in a function that's been running for a good hour's time.
So, yeah, tests and everything... But can't there be a way to pay a bit less for such a stupid error?
Introducing: pylint. Like jslint which I've already met, pylint checks your Python code for stupid mistakes such as this. You can get it here: http://www.pylint.org/ or using pip install pylint.
You can test your pylint installation using the example from above: pylint -E foo.py will output only the most important errors, and will of course exit with a nonzero status code.
Nice! But how would I add this to my editors?
Introducing(2): Sublime Text. Word on the street (or on the train back home with Micky who told me about it) is that it's the coolest new lightweight editor. We don't want to write Python on Eclipse and PyDev, now, do we...?
You can get it here: http://www.sublimetext.com
Now apparently Sublime Text can build your code using all sorts of builders, including predefined! While the language for defining such builds is not very sophisticated (i.e. no conditionals), you can write a little bash script to go with it so that whenever you build your code, first pylint will go over it and won't let it run if it's buggy.
Say you put the following code in /home/you/pylinton :
#!/bin/bash
pylint -E $1 2>/dev/null
if [ $? -eq 0 ]
then
python $1
fi
{
"cmd": ["/home/you/pylinton" , "$file"]
}
Then voila! Ctrl+B builds your currently open Python source code. You'll get pylint's errors if there are any; otherwise, Python will execute. That won't replace your tests but it can save you some time and heartaches when you're running ahead.
tl;dr
(1/4) pylint is jslint for Python. http://www.pylint.org/ or using pip install pylint
(3/4) Put this in /home/you/pylinton...
(4/4) ...and use this as a new build system definition in sublime text:
#!/bin/bash
pylint -E $1 2>/dev/null
if [ $? -eq 0 ]
then
python $1
fi
pylint -E $1 2>/dev/null
if [ $? -eq 0 ]
then
python $1
fi
{
"cmd": ["/home/guyrap/pylinton" , "$file"]
}
Tuesday, October 22, 2013
JSON Search Results via Bing
Hello again. Still working on the same stuff, still needing to get some search results off search engines, in JSON format, automatically...
If you thought getting these results out of Google was a nightmare - Bing is 10x times the circus that Google is. For crying out loud, it seems the the Bing Search V2.0 is the old, deprecated version, while the currently available version is called... Bing Search API. Thanks for nothing. As usually with Microsoft, it's some posts in StackOverflow and some trials and errors that eventually supplied the solution.
Goto the Bing Search API page, now part of the "Azure Datamarket", since nothing in Microsoft can't go by a simple, humble name*:
There you have it. "The basic principles of time travel are there".
Other useful links:
* DreamSpark. I rest my case.
If you thought getting these results out of Google was a nightmare - Bing is 10x times the circus that Google is. For crying out loud, it seems the the Bing Search V2.0 is the old, deprecated version, while the currently available version is called... Bing Search API. Thanks for nothing. As usually with Microsoft, it's some posts in StackOverflow and some trials and errors that eventually supplied the solution.
Goto the Bing Search API page, now part of the "Azure Datamarket", since nothing in Microsoft can't go by a simple, humble name*:
To sign up for the free 5000 transactions/month program, you'll might have to sign up to the Windows Azure Marketplace. Yes, you are going to have a Microsoft account.
After signing up to this program, goto your account details:
You'll see that your Primary Account Key is listed. Keep the Primary Account Key.
And now for the somewhat annoying (IMHO) part. Unlike in the Google Search API, you can't use that key in the URL. When Bing was all about AppIDs, this was a query parameter as well. But that thing's gone. Now, you'll have to add authentication to your HTTPS GET request. This can be done by requests.
Code example:
import requests url = "https://api.datamarket.azure.com/Bing/Search/Web?$format=json&Query=%27Xbox%27" key = |
There you have it. "The basic principles of time travel are there".
Other useful links:
If you want to search for other stuff such as images etc, use the general search API:
If you want to have even more fun with their APIs, go for the whole family:
Review your datasets - how much searches are left for each of the APIs:
* DreamSpark. I rest my case.
Monday, October 21, 2013
Retrieve Google Search results in JSON format
Let's say you want to...
Write a Python script which searches a specific site for a query in Google and returns the results [in JSON format].
Sounds easy, huh? Use requests to, err, make the HTTPS requests, then beautifulsoup to parse the links out. (and what would be the easiest way to grab those links? By getting the XPath from the browser...).
Well, that's what I've thought. Wrong.
Google don't want you to do that. And when Google doesn't want you to do it, you just can't. You'll find yourself blocked from making automatic requests pretty darn fast. You'll be required to solve CAPTCHAs in order to search. Not ideal for an automatic crawler...
The impolite solution would be to write a very sophisticated crawler which uses multiple proxies to cover the search-space without seeming to suspicious. The advantage is, you'll get it done. The obvious disadvantage - it's complicated!
The polite solution would be to use Google's own tool for this job - the Custom Search. You get 100 queries per day for free, and can pay for more (or utilize more accounts - sure easier than utilizing more proxies...).
I've written this guide today while performing these actions, so I can guarentee it's working - for now. Doing this will get you to far, strange places in Google like I've never seen before. Their documentation is quite lousy, which is why I've written this guide. I hope it won't get outdated too soon. Their official documentation can be found here: https://developers.google.com/custom-search/docs/overview
The short explanation
Goto Google Cloud Console and login.
https://cloud.google.com/console
Create a new project.
Under APIs and Auth > APIs, enable the Custom Search API.
Under APIs and Auth > Registered Apps, register a new web application.
Keep the Browser Key API KEY.
Goto Google Custom Search and login.
https://www.google.com/cse/
Add a site you would like to search.
Under Edit search engine > Setup, click on the Search engine ID button. Keep it as well.
Construct the URL as follows:
https://www.googleapis.com/customsearch/v1?q=<query>&cx=<Search engine ID>&key=<API KEY>
For instance:
https://www.googleapis.com/customsearch/v1?q=ooad&cx=011540632070411059229:cvop8g3tl2q&key=AIzaSyBNND2nT0vVb4M0875cxkhcUwzepeIkXng
Appearently, this imports via code as well. (requests is required):
The results will look like this:
The long explanation (with pictures)
1. Goto Google Cloud Console and login. https://cloud.google.com/console
7. This is the newly created app's overview page. Open up the Browser Key tab and keep the API KEY.
Write a Python script which searches a specific site for a query in Google and returns the results [in JSON format].
Sounds easy, huh? Use requests to, err, make the HTTPS requests, then beautifulsoup to parse the links out. (and what would be the easiest way to grab those links? By getting the XPath from the browser...).
Well, that's what I've thought. Wrong.
Google don't want you to do that. And when Google doesn't want you to do it, you just can't. You'll find yourself blocked from making automatic requests pretty darn fast. You'll be required to solve CAPTCHAs in order to search. Not ideal for an automatic crawler...
The impolite solution would be to write a very sophisticated crawler which uses multiple proxies to cover the search-space without seeming to suspicious. The advantage is, you'll get it done. The obvious disadvantage - it's complicated!
The polite solution would be to use Google's own tool for this job - the Custom Search. You get 100 queries per day for free, and can pay for more (or utilize more accounts - sure easier than utilizing more proxies...).
I've written this guide today while performing these actions, so I can guarentee it's working - for now. Doing this will get you to far, strange places in Google like I've never seen before. Their documentation is quite lousy, which is why I've written this guide. I hope it won't get outdated too soon. Their official documentation can be found here: https://developers.google.com/custom-search/docs/overview
The short explanation
Goto Google Cloud Console and login.
https://cloud.google.com/console
Create a new project.
Under APIs and Auth > APIs, enable the Custom Search API.
Under APIs and Auth > Registered Apps, register a new web application.
Keep the Browser Key API KEY.
Goto Google Custom Search and login.
https://www.google.com/cse/
Add a site you would like to search.
Under Edit search engine > Setup, click on the Search engine ID button. Keep it as well.
Construct the URL as follows:
https://www.googleapis.com/customsearch/v1?q=<query>&cx=<Search engine ID>&key=<API KEY>
For instance:
https://www.googleapis.com/customsearch/v1?q=ooad&cx=011540632070411059229:cvop8g3tl2q&key=AIzaSyBNND2nT0vVb4M0875cxkhcUwzepeIkXng
Appearently, this imports via code as well. (requests is required):
import requests, json
url = "https://www.googleapis.com/customsearch/v1?q=ooad&cx=011540632070411059229:cvop8g3tl2q&key=AIzaSyBNND2nT0vVb4M0875cxkhcUwzepeIkXng"
r = requests.get(url)
data = r.json()
with open('data.json','wb') as f:
json.dump(data,f)
|
The results will look like this:
{
"kind": "customsearch#search",
"url": {
"type": "application/json",
"template": "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&cref={cref?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json"
},
"queries": {
"request": [
{
"title": "Google Custom Search - ooad",
"totalResults": "3",
"searchTerms": "ooad",
"count": 3,
"startIndex": 1,
"inputEncoding": "utf8",
"outputEncoding": "utf8",
"safe": "off",
"cx": "011540632070411059229:cvop8g3tl2q"
}
]
},
"context": {
"title": "Cs.bgu.ac.il"
},
"searchInformation": {
"searchTime": 0.247919,
"formattedSearchTime": "0.25",
"totalResults": "3",
"formattedTotalResults": "3"
},
"items": [
{
"kind": "customsearch#result",
"title": "ADSS122 (Spring 2012)",
"htmlTitle": "ADSS122 (Spring 2012)",
"link": "http://www.cs.bgu.ac.il/~guyrap/adss122/",
"displayLink": "www.cs.bgu.ac.il",
"snippet": "OOAD Winter 2012's Moed Aleph Solution: PDF, pptx. (10.07.2012) New OCL Examples: PDF, docx. Preparation for Moed Bet, ADSS Spring 2012: PDF, pptx.",
"htmlSnippet": "\u003cb\u003eOOAD\u003c/b\u003e Winter 2012's Moed Aleph Solution: PDF, pptx. (10.07.2012) New OCL \u003cbr\u003e Examples: PDF, docx. Preparation for Moed Bet, ADSS Spring 2012: PDF, pptx.",
"cacheId": "WC8B134UUl0J",
"formattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/",
"htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/"
},
{
"kind": "customsearch#result",
"title": "Guy Rapaport - CS Webpage",
"htmlTitle": "Guy Rapaport - CS Webpage",
"link": "http://www.cs.bgu.ac.il/~guyrap/",
"displayLink": "www.cs.bgu.ac.il",
"snippet": "Mar 21, 2013 ... [HighLearn]; Teach-assist the OOAD and ADSS courses (under Dr. Arnon Sturm). (HighLearn , additional resources) (2011-2012, 4 semesters).",
"htmlSnippet": "Mar 21, 2013 \u003cb\u003e...\u003c/b\u003e [HighLearn]; Teach-assist the \u003cb\u003eOOAD\u003c/b\u003e and ADSS courses (under Dr. Arnon Sturm). \u003cbr\u003e (HighLearn , additional resources) (2011-2012, 4 semesters).",
"cacheId": "YXe3atmPI-8J",
"formattedUrl": "www.cs.bgu.ac.il/~guyrap/",
"htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/"
},
{
"kind": "customsearch#result",
"title": "הכנה למועד ב'",
"htmlTitle": "הכנה למועד ב'",
"link": "http://www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx",
"displayLink": "www.cs.bgu.ac.il",
"snippet": "OOAD – סתיו 2012. במערכת ניהול הסטודנטים לתואר שני (תקרא מעתה המערכת) נדרש לספק תמיכה לכלל הנוגעים לטיפול בסטודנטים הלומדים לתואר שני באוניברסיטה. אלו כוללים את ...",
"htmlSnippet": "\u003cb\u003eOOAD\u003c/b\u003e – סתיו 2012. במערכת ניהול הסטודנטים לתואר שני (תקרא מעתה המערכת) נדרש \u003cbr\u003e לספק תמיכה לכלל הנוגעים לטיפול בסטודנטים הלומדים לתואר שני באוניברסיטה. אלו כוללים \u003cbr\u003e את \u003cb\u003e...\u003c/b\u003e",
"cacheId": "rcFEb9K_Sq4J",
"mime": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
"fileFormat": "Microsoft Powerpoint",
"formattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx",
"htmlFormattedUrl": "www.cs.bgu.ac.il/~guyrap/adss122/2012WinterPreMoedBet.pptx"
}
]
}
|
The long explanation (with pictures)
1. Goto Google Cloud Console and login. https://cloud.google.com/console
2. Create Project. I used the default settings.
3. This is the newly created project overview page. Click on APIs & auth > APIs on the left-hand sidebar.
4. Go through the list of APIs and turn on the Custom Search API.
5. Go to the Registered apps. Click on the "REGISTER APP" button.
6. Register your new app. I used "Web Application".
The first part -obtaining an API KEY - is done.
8. Goto the Google Custom Search page. "Create a custom search engine". You should already be logged in after your activity in the Google Cloud Console.
9. In the CSE creation page, enter the sites you would like to custom-search. "Create".
10. For example:
(this is my page on BGU's CS dept. web server.)
11. "You've successfully created your Custom search engine". Why the capital C on custom??
Click on "Edit search engine" in the left side-bar.
12. Under details, click on the "Search engine ID". Keep the Search engine ID.
As I've said in the short explanation (see above) -construct your search query URL and enjoy.
Sunday, October 20, 2013
Ignoring lid close on Xubuntu 13.10 / xfce
Yesterday I've installed Xubuntu 13.10 (the low-resources version of Ubuntu which fits older, weaker machines) on my 10" laptop. I intend to work with the laptop plugged into a large screen. Naturally, after I connected the laptop to the screen I closed the lid. Whoops! The computer became suspended...
To make a long story short, it's not the BIOS fault, and trying to solve this via xfce's Settings Manager did not work. Or at least, did not do enough. Any suggestions I've read online on how to solve it (dconf editing, gconf editing, writing acpid handlers...) did not do the trick.
The solution came from the arch linux forums:
https://bbs.archlinux.org/viewtopic.php?id=154914
Edit the logind.conf file (I found it on /etc/systemd/logind.conf), uncomment the HandleLidSwitch=suspend line and change it to HandleLidSwitch=ignore.
sudo nano /etc/systemd/logind.conf
-#HandleLidSwitch=suspend
+HandleLidSwitch=ignore
Done! I've rebooted the system and everything worked like magic.
To make a long story short, it's not the BIOS fault, and trying to solve this via xfce's Settings Manager did not work. Or at least, did not do enough. Any suggestions I've read online on how to solve it (dconf editing, gconf editing, writing acpid handlers...) did not do the trick.
The solution came from the arch linux forums:
https://bbs.archlinux.org/viewtopic.php?id=154914
Edit the logind.conf file (I found it on /etc/systemd/logind.conf), uncomment the HandleLidSwitch=suspend line and change it to HandleLidSwitch=ignore.
sudo nano /etc/systemd/logind.conf
-#HandleLidSwitch=suspend
+HandleLidSwitch=ignore
Done! I've rebooted the system and everything worked like magic.
Thursday, July 18, 2013
Getting to know MEF - Meet the DefaultDependencyResolver
When I started working for my current employer, the system we were working on was an ASP.NET MVC application. I was new to C#, new to ASP.NET, and never heard of the MVC framework ("MVC? You mean the architecture of...?").
8 months later, I wouldn't say I got it all, but one thing I did get to know which was also interesting was the MEF Dependency Injection framework. MEF 2 is just coming out and it's too early for me to discuss even the first version. What I want to write about today is Dependency Injection. I will review the Dependency Resolver that ships with MVC by default in order to illustrate how simple (read: useless) it is. This will be a good starting point on why MEF is useful.
Martin Fowler writes about Dependency Injection too, and gives an example of moving an object's dependencies into the container, then using some "container" (more on those later) to supply the dependencies and construct the object.
So, when do we program without using new? In ASP.NET MVC, the most common use-case is when our application is online and someone enters some controller's action. When an incoming request is handled, a new constructor instance must be created. We're not the one writing the code for it - MVC does it "by itself". And if we're complicating our controller's constructor by adding dependencies for it (such as a database context, a provider or a repository class), MVC still has to create it somehow...
Start Visual Studio (2012 in my case), and start a new C# Project from the "ASP.NET MVC 3 Web Application" template. You'll get the example project with a HomeController. Add a default constructor (that is, a parameterless constructor) to the controller, put a breakpoint on its end and debug the application. When the breakpoint is reached (quite immediately), examine the Call Stack. If you're only seeing 2 calls, right click inside the Call Stack window and enable the "Show External Code" feature.
I'm posting the entire call stack to make one clear point here: although it's looks like lots and lots of details, it can be decyphered. It took me a long time to stop seeing this as "details" and start to actually make things out of it. This was accomplished thanks to two people: my boss, and the author of JustDecompile, the C# Decompiler that everybody love-love-loves.
Below is what you will see. The colors were, of course, added by me:
In this stack you can see all the crazy stuff that your incoming request triggers. This is a lot of code to actually go through, even with a decompiler at hand, so we'll only be roughly speaking about what happens. Below are the most interesting steps. remember, the top of the stack is the last thing to happen, so the items in this list appear in the opposite order to their presentation above.
1) System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController (System.Web.Routing.RequestContext requestContext, string controllerName)
Upon the incoming request (to /Home/Index), MVC's DefaultControllerFactory is requested to supply the appropriate controller (HomeController in this case). Note that the requestContext, out of which the controllerName was obtained, are the parameters. The controllerName ("Home") is parsed by MVC from the request URL (more here - old but still quite accurate). The next steps would link this name to the respective controller's classname (really MvcApplication2.Controllers.HomeController, since my dummy project is named MvcApplication2), and instantiate the class.
2) System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver .GetService(System.Type serviceType)
After the controller's Type is determined from its string name (details left to the reader), the CreateController method tried to create it. And who's job is it? Finally, the DependencyResolver! And not any DependencyResolver, but the DefaultDependencyResolver. Decompilation reveals that the DefaultDependencyResolver is an inner class of the DependencyResolver.
3) mscorlib.dll!System.RuntimeType.CreateInstanceSlow(...parameters...)
The depdendency resolver makes the CLR core (we're in mscorlib.dll now...) create the instance, with all of its weird parameters...
4) MvcApplication2.dll!MvcApplication2.Controllers.HomeController
.HomeController() Line 15 C#
...and finally, the constructor of the right type (our HomeController) is invoked, and this is where our brakepoint was placed (line 15 in the HomeController.cs file).
So what happens in this single line of code?
The DependencyResolver class' Current public property is invoked, and once the current DependencyResolver is obtained, its method GetService<T> is called (with Foo being T). This is really the DefaultDepepdencyResolver we've seen earlier.
But what if Foo wouldn't have a default constructor? In that case, the CreateInstanceDefaultCtor method we've seen before would throw an exception: System.MissingMethodException: No parameterless constructor defined for this object. Add a new constructor to the controller that has some parameters (even an int would suffice) and you'll see it happening in an MVC application as well.
DefaultDependencyResolver implements the IDependencyResolver interface, which is also a part of the MVC assembly. Here's what it can do:
And since the DependencyResolver (not an implementor of the interface, by the way!) has a public SetResolver method, what we see here really is an extension point.
See, the MVC people know their DependencyResolver isn't what you need! It just serves as a placeholder for something more exciting and capable. Something like MEF, Castle Windsor, Ninject, etc. To start understanding MEF, one should first understand how life without MEF would look like. What MEF has come to replace.
Download JustDecompile from Telerik: http://www.telerik.com/products/decompiler.aspx
For the code snippet appearing above I've used Alex Gorbatchev's Syntax Highlighter, which I incorporated into Blogger by following Tahir Hassan's instructions. I won't be moving to gists since I don't have anything important enough to be forked etc.
8 months later, I wouldn't say I got it all, but one thing I did get to know which was also interesting was the MEF Dependency Injection framework. MEF 2 is just coming out and it's too early for me to discuss even the first version. What I want to write about today is Dependency Injection. I will review the Dependency Resolver that ships with MVC by default in order to illustrate how simple (read: useless) it is. This will be a good starting point on why MEF is useful.
So what is dependency injection?
The best description for Dependency Injection (DI) I've found so far appears on the Google Guice project description:Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice's @Inject as the new new. You will still need to write factories in some cases, but your code will not depend directly on them. Your code will be easier to change, unit test and reuse in other contexts.Guice's Motivation page elaborates on what DI is and how an API for it would look. Whether you'll be using Guice, Ninject, Autofac, Castle Windsor or MEF - basically it's the same thing all over again. (Sorry, rival developers of different DI implementations!)
Martin Fowler writes about Dependency Injection too, and gives an example of moving an object's dependencies into the container, then using some "container" (more on those later) to supply the dependencies and construct the object.
So, when do we program without using new? In ASP.NET MVC, the most common use-case is when our application is online and someone enters some controller's action. When an incoming request is handled, a new constructor instance must be created. We're not the one writing the code for it - MVC does it "by itself". And if we're complicating our controller's constructor by adding dependencies for it (such as a database context, a provider or a repository class), MVC still has to create it somehow...
MVC creates the controllers?
Yep. And I can prove it to you.Start Visual Studio (2012 in my case), and start a new C# Project from the "ASP.NET MVC 3 Web Application" template. You'll get the example project with a HomeController. Add a default constructor (that is, a parameterless constructor) to the controller, put a breakpoint on its end and debug the application. When the breakpoint is reached (quite immediately), examine the Call Stack. If you're only seeing 2 calls, right click inside the Call Stack window and enable the "Show External Code" feature.
I'm posting the entire call stack to make one clear point here: although it's looks like lots and lots of details, it can be decyphered. It took me a long time to stop seeing this as "details" and start to actually make things out of it. This was accomplished thanks to two people: my boss, and the author of JustDecompile, the C# Decompiler that everybody love-love-loves.
Below is what you will see. The colors were, of course, added by me:
> MvcApplication2.dll!MvcApplication2.Controllers.HomeController.HomeController() Line 15 C# [Native to Managed Transition] [Managed to Native Transition] mscorlib.dll!System.RuntimeType.CreateInstanceSlow(bool publicOnly, bool skipCheckThis, bool fillCache, ref System.Threading.StackCrawlMark stackMark) + 0x72 bytes mscorlib.dll!System.Activator.CreateInstance(System.Type type, bool nonPublic) + 0x54 bytes mscorlib.dll!System.Activator.CreateInstance(System.Type type) + 0x7 bytes System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver.GetService(System.Type serviceType) + 0x1c bytes System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.DefaultControllerActivator.Create(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x2a bytes System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(System.Web.Routing.RequestContext requestContext, System.Type controllerType) + 0x51 bytes System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController(System.Web.Routing.RequestContext requestContext, string controllerName) + 0x4b bytes System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.ProcessRequestInit(System.Web.HttpContextBase httpContext, out System.Web.Mvc.IController controller, out System.Web.Mvc.IControllerFactory factory) + 0xc5 bytes System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest.AnonymousMethod__2() + 0x32 bytes System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust.AnonymousMethod__a() + 0xe bytes System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.GetCallInAppTrustThunk.AnonymousMethod__0(System.Action f) + 0x8 bytes System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust(System.Action action) + 0x17 bytes System.Web.Mvc.dll!System.Web.Mvc.SecurityUtil.ProcessInApplicationTrust (System.Func func) + 0x59 bytes System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase httpContext, System.AsyncCallback callback, object state) + 0x63 bytes System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContext httpContext, System.AsyncCallback callback, object state) + 0x33 bytes System.Web.Mvc.dll!System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, object extraData) + 0x11 bytes System.Web.dll!System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() + 0x227 bytes System.Web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, ref bool completedSynchronously) + 0x9c bytes System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error) + 0x342 bytes System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb) + 0x60 bytes System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context) + 0xbb bytes System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f3 bytes System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes [Native to Managed Transition] [Managed to Native Transition] System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x350 bytes System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) + 0x1f bytes [Appdomain Transition]
In this stack you can see all the crazy stuff that your incoming request triggers. This is a lot of code to actually go through, even with a decompiler at hand, so we'll only be roughly speaking about what happens. Below are the most interesting steps. remember, the top of the stack is the last thing to happen, so the items in this list appear in the opposite order to their presentation above.
1) System.Web.Mvc.dll!System.Web.Mvc.DefaultControllerFactory.CreateController (System.Web.Routing.RequestContext requestContext, string controllerName)
Upon the incoming request (to /Home/Index), MVC's DefaultControllerFactory is requested to supply the appropriate controller (HomeController in this case). Note that the requestContext, out of which the controllerName was obtained, are the parameters. The controllerName ("Home") is parsed by MVC from the request URL (more here - old but still quite accurate). The next steps would link this name to the respective controller's classname (really MvcApplication2.Controllers.HomeController, since my dummy project is named MvcApplication2), and instantiate the class.
2) System.Web.Mvc.dll!System.Web.Mvc.DependencyResolver.DefaultDependencyResolver .GetService(System.Type serviceType)
After the controller's Type is determined from its string name (details left to the reader), the CreateController method tried to create it. And who's job is it? Finally, the DependencyResolver! And not any DependencyResolver, but the DefaultDependencyResolver. Decompilation reveals that the DefaultDependencyResolver is an inner class of the DependencyResolver.
How does the DefaultDependencyResolver does this? Namely, how does the GetService implementation looks like? Let's decompile and see:
Looks like the DefaultDependencyResolver just calls Activator.CreateInstance(Type type) . So what does Activator.CreateInstance do? (pssst! It calls Activator.CreateInstance(Type type, bool nonPublic) with the same type and false for a boolean. Below is this second method's code:)
Voila!
Now we've got it! The DefaultDependencyResolver will invoke methods that will finally CreateInstanceDefaultCtor - that is, can only use parameterless constructors!!!
Let's go through the last 2 steps, just to make sure this is really happenning:
Voila!
Now we've got it! The DefaultDependencyResolver will invoke methods that will finally CreateInstanceDefaultCtor - that is, can only use parameterless constructors!!!
Let's go through the last 2 steps, just to make sure this is really happenning:
3) mscorlib.dll!System.RuntimeType.CreateInstanceSlow(...parameters...)
The depdendency resolver makes the CLR core (we're in mscorlib.dll now...) create the instance, with all of its weird parameters...
4) MvcApplication2.dll!MvcApplication2.Controllers.HomeController
.HomeController() Line 15 C#
...and finally, the constructor of the right type (our HomeController) is invoked, and this is where our brakepoint was placed (line 15 in the HomeController.cs file).
Dependency Injection in MVC
The conclusion of this journey within the assemblies shows that dependency injection is built into MVC. By default, the actual construction of controllers is separated from the code the requires these controllers. The responsibility for obtaining the controller instance is delegated to the DefaultDependencyResolver. This is exactly what the DI pattern aims at. Too bad the DefaultDependencyResolver is so weak...
Show it to me again, but in the simplest possible way!
Here goes. This is the sole class in new, empty C# project:So what happens in this single line of code?
The DependencyResolver class' Current public property is invoked, and once the current DependencyResolver is obtained, its method GetService<T> is called (with Foo being T). This is really the DefaultDepepdencyResolver we've seen earlier.
But what if Foo wouldn't have a default constructor? In that case, the CreateInstanceDefaultCtor method we've seen before would throw an exception: System.MissingMethodException: No parameterless constructor defined for this object. Add a new constructor to the controller that has some parameters (even an int would suffice) and you'll see it happening in an MVC application as well.
Conclusion: So why using the DefaultDependencyResolver?
If you're writing ASP.NET MVC applications, the (Default)DependencyResolver is a given fact. Sure, you can replace the default controller factory, but a) Why would you do that? and b) there's a chance even your custom controller factory will use a DI of some sort.DefaultDependencyResolver implements the IDependencyResolver interface, which is also a part of the MVC assembly. Here's what it can do:
And since the DependencyResolver (not an implementor of the interface, by the way!) has a public SetResolver method, what we see here really is an extension point.
See, the MVC people know their DependencyResolver isn't what you need! It just serves as a placeholder for something more exciting and capable. Something like MEF, Castle Windsor, Ninject, etc. To start understanding MEF, one should first understand how life without MEF would look like. What MEF has come to replace.
Download JustDecompile from Telerik: http://www.telerik.com/products/decompiler.aspx
For the code snippet appearing above I've used Alex Gorbatchev's Syntax Highlighter, which I incorporated into Blogger by following Tahir Hassan's instructions. I won't be moving to gists since I don't have anything important enough to be forked etc.
Sunday, July 7, 2013
Enumerating Objects in an ActiveDirectory OU via C#
The goal.
My corporation employes Microsoft Active Directory to control (for lack of better word) its users and machines.
I have several computer labs under my control, and I want to list all of the machines in these labs, defined as Organizational Units in my domain's Active Directory.
I want this done via C#.
The straightforward solution.
Use this code:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
Which requires an OuDn parameter, which you can find (for your OU of interest) using this tool:
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx
Voila!
And now for the back story.
I often intend to post stuff in this blog, but then give up. I got to implement some cool stuff lately. But sadly, all of these things were already done better by others, or will be made easier by the framework in the future. One example is branding (that is, matching the language and culture, texts and CSS for users entering your site from different domains). The second is a synchronous service bus. We originally intended to use it until we'll need something "real" such as RabbitMQ; but now it seems we'll resort to using some readymade library anyway.
So this post is not about anything cool, but about something that is 1) terribly annoying and 2) will probably not be improved in its future versions. Why? Because it's delving into the Active Directory of my workplace and extracting some information out. It's Microsoft, it's Active Directory (which is legacy-code all over AFAIK), and it's not intuitive in any way.
Let's make things clear - I am not an expert on Active Directory, nor do I want to be. From my point of view, as a (somewhat privileged) user of this system but not its maintainer here in any way - this is just the odd cloud of Microsoft nonsense that hovers over the network and make authentication work. I was exposed to some of its nice features (such as group policy), but I came to deal with it for a particular reason (more below) and I couldn't care less about its internals until after I get this thing working. With some other technology I would gladly read some documentation or open a book. But in a jungle-o-spaghetti such as Active Directory - no way!
So let's break it down.
After some Googling and executing failing code from different sources, I got to this example - "Howto: (Almost) Everything In Active Directory via C#" by thund3rstruck:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
This article from 2008 is still relevant today - more than 5 years later. Note that according to this article, most of the MSDN examples are irrelevant or incorrect. Surprising...
For this code to execute, one must instantiate a DirectoryEntry object:
with the OuDn parameter defined like this:
As a user I have access the Active Directory Users and Computers tool installed on our server which queries the corporate Active Directory server. I know exactly what I want - an Organizational Unit called "Labs", which contains several OUs - each representing a computer lab, and each of the labs contents - the machines in this lab. But what are the OuDn's for this?
My corporation employes Microsoft Active Directory to control (for lack of better word) its users and machines.
I have several computer labs under my control, and I want to list all of the machines in these labs, defined as Organizational Units in my domain's Active Directory.
I want this done via C#.
The straightforward solution.
Use this code:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
Which requires an OuDn parameter, which you can find (for your OU of interest) using this tool:
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx
Voila!
And now for the back story.
I often intend to post stuff in this blog, but then give up. I got to implement some cool stuff lately. But sadly, all of these things were already done better by others, or will be made easier by the framework in the future. One example is branding (that is, matching the language and culture, texts and CSS for users entering your site from different domains). The second is a synchronous service bus. We originally intended to use it until we'll need something "real" such as RabbitMQ; but now it seems we'll resort to using some readymade library anyway.
So this post is not about anything cool, but about something that is 1) terribly annoying and 2) will probably not be improved in its future versions. Why? Because it's delving into the Active Directory of my workplace and extracting some information out. It's Microsoft, it's Active Directory (which is legacy-code all over AFAIK), and it's not intuitive in any way.
Let's make things clear - I am not an expert on Active Directory, nor do I want to be. From my point of view, as a (somewhat privileged) user of this system but not its maintainer here in any way - this is just the odd cloud of Microsoft nonsense that hovers over the network and make authentication work. I was exposed to some of its nice features (such as group policy), but I came to deal with it for a particular reason (more below) and I couldn't care less about its internals until after I get this thing working. With some other technology I would gladly read some documentation or open a book. But in a jungle-o-spaghetti such as Active Directory - no way!
So let's break it down.
After some Googling and executing failing code from different sources, I got to this example - "Howto: (Almost) Everything In Active Directory via C#" by thund3rstruck:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
This article from 2008 is still relevant today - more than 5 years later. Note that according to this article, most of the MSDN examples are irrelevant or incorrect. Surprising...
For this code to execute, one must instantiate a DirectoryEntry object:
new DirectoryEntry("LDAP://" + OuDn);
with the OuDn parameter defined like this:
The parameter
OuDn is the Organizational Unit distinguishedName such as OU=Users,dc=myDomain,dc=com
Luckily, some good guys in the SysInternals team came out with this tool - Active Directory Explorer v1.44:http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx
This tool allows you to browse through the Active Directory tree - but it gives you the OuDn's for everything you go through!
Using the OuDn with thund3rstruck's code, I could obtain the list of machines I wanted. And then I could pass them to some PowerShell script which did some maintenance over them. Nice!
As for the code - again, do enter this link:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
I used the code exactly as he posted it - I only got the OuDn using the explorer tool.
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx
Credit for thund3rstruck and to Bryce Cogswell and Mark Russinovich of Microsoft. I just patched things together.
This tool allows you to browse through the Active Directory tree - but it gives you the OuDn's for everything you go through!
Using the OuDn with thund3rstruck's code, I could obtain the list of machines I wanted. And then I could pass them to some PowerShell script which did some maintenance over them. Nice!
As for the code - again, do enter this link:
http://www.codeproject.com/Articles/18102/Howto-Almost-Everything-In-Active-Directory-via-C#19
I used the code exactly as he posted it - I only got the OuDn using the explorer tool.
http://technet.microsoft.com/en-us/sysinternals/bb963907.aspx
Credit for thund3rstruck and to Bryce Cogswell and Mark Russinovich of Microsoft. I just patched things together.
Subscribe to:
Posts (Atom)