Tuesday, October 22, 2013

JSON Search Results via Bing

Hello again. Still working on the same stuff, still needing to get some search results off search engines, in JSON format, automatically...

If you thought getting these results out of Google was a nightmare - Bing is 10x times the circus that Google is. For crying out loud, it seems the the Bing Search V2.0 is the old, deprecated version, while the currently available version is called... Bing Search API. Thanks for nothing. As usually with Microsoft, it's some posts in StackOverflow and some trials and errors that eventually supplied the solution.


Goto the Bing Search API page, now part of the "Azure Datamarket", since nothing in Microsoft can't go by a simple, humble name*:

To sign up for the free 5000 transactions/month program, you'll might have to sign up to the Windows Azure Marketplace. Yes, you are going to have a Microsoft account.

After signing up to this program, goto your account details:

You'll see that your Primary Account Key is listed. Keep the Primary Account Key.

And now for the somewhat annoying (IMHO) part. Unlike in the Google Search API, you can't use that key in the URL. When Bing was all about AppIDs, this was a query parameter as well. But that thing's gone. Now, you'll have to add authentication to your HTTPS GET request. This can be done by requests.

Code example:
import requests
url = "https://api.datamarket.azure.com/Bing/Search/Web?$format=json&Query=%27Xbox%27"
key = 
r = requests.get(url, auth=(key,key))
print r.status_code
print r.headers['content-type']
print r.json()

There you have it. "The basic principles of time travel are there".

Other useful links:
If you want to search for other stuff such as images etc, use the general search API:
If you want to have even more fun with their APIs, go for the whole family:
Review your datasets - how much searches are left for each of the APIs:

* DreamSpark. I rest my case.