A few weeks ago, I asked for pointers to something that would translate 400-odd partial paper citations (usually just author names and paper titles) into full bibliography entries. So far, none of the proposed solutions have worked out:
- Bibsonomy’s search only found 2 of the 10 papers I tried it on manually.
- JabRef did no better.
- CrossRef found none.
- The “possibly working Python script” for querying Google Scholar, and my home-grown attempts to do likewise, produced nothing but “your agent isn’t allowed access” messages. (Yes, I’ve included all the same headers that Firefox sends when I do searches manually—doesn’t make a difference. Google, they smart.)
- I didn’t get cb2bib to run on Mac OS X. Admittedly, I only tried for ten minutes, but this isn’t a critical path project for me.
*sigh*
Sorry to hear that your attempts have not produced much so far… That’s an interesting project!
With regards to cb2bib, I’d like to point out that it is available via Fink (at least in the unstable branch).
OK, about fooling google with the User-Agent setting (if you’ve tested your script against a local web server you’ll notice that the User-Agent header is still set to urllib-blah-blah) you’ll need to subclass urllib.URLOpener and set the class attribute version to your desired header.
like this: http://paste.pocoo.org/show/156877/
Maybe this Python script can help you a bit: http://gist.github.com/255743
It searches for a given phrase on Bibsonomy and reports entries found. In case if it finds very few matches (as you mention in the blog post) try making the search phrase shorter / simpler.
Have you considered reverse engineering one of the Mycroft project Firefox search plugins? They include many searches of Google scholar that use various proxies.
Surely you could build a python script with the hints from one of these.
// Tony
Have you looked at Mendeley? It’s a pretty nice solution for managing a library of academic papers. One of it’s features is integration with Google Scholar. I’ve found it works pretty well but it does require some manual intervention.
http://www.mendeley.com/
The problem with any script that attempts to scrape Google Scholar is that Scholar detects spikes in activity from your IP (for some value of ‘spikes’) and will send you a Forbidden until you revalidate via CAPTCHA as a human. Scraping and scripts are against the Google ToS.
AFAIK, this is because Google gives scholar so much less bandwidth than the main search engine and they defend it much more than the regular search engine.