Comments on: Stymied http://third-bit.com/blog/archives/3307.html Data is ones and zeroes | Software is ones and zeroes and hard work. Thu, 24 May 2012 15:30:31 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Jim Graham http://third-bit.com/blog/archives/3307.html#comment-3304 Jim Graham Mon, 21 Dec 2009 21:50:15 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3304 The problem with any script that attempts to scrape Google Scholar is that Scholar detects spikes in activity from your IP (for some value of 'spikes') and will send you a Forbidden until you revalidate via CAPTCHA as a human. Scraping and scripts are against the Google ToS. AFAIK, this is because Google gives scholar so much less bandwidth than the main search engine and they defend it much more than the regular search engine. The problem with any script that attempts to scrape Google Scholar is that Scholar detects spikes in activity from your IP (for some value of ‘spikes’) and will send you a Forbidden until you revalidate via CAPTCHA as a human. Scraping and scripts are against the Google ToS.

AFAIK, this is because Google gives scholar so much less bandwidth than the main search engine and they defend it much more than the regular search engine.

]]> By: Nathan http://third-bit.com/blog/archives/3307.html#comment-3303 Nathan Sun, 20 Dec 2009 18:39:42 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3303 Have you looked at Mendeley? It's a pretty nice solution for managing a library of academic papers. One of it's features is integration with Google Scholar. I've found it works pretty well but it does require some manual intervention. http://www.mendeley.com/ Have you looked at Mendeley? It’s a pretty nice solution for managing a library of academic papers. One of it’s features is integration with Google Scholar. I’ve found it works pretty well but it does require some manual intervention.

http://www.mendeley.com/

]]> By: Tony Wiliams http://third-bit.com/blog/archives/3307.html#comment-3302 Tony Wiliams Mon, 14 Dec 2009 05:29:22 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3302 Have you considered reverse engineering one of the Mycroft project Firefox search plugins? They include many searches of Google scholar that use various proxies. Surely you could build a python script with the hints from one of these. // Tony Have you considered reverse engineering one of the Mycroft project Firefox search plugins? They include many searches of Google scholar that use various proxies.

Surely you could build a python script with the hints from one of these.

// Tony

]]> By: Uldis Bojars http://third-bit.com/blog/archives/3307.html#comment-3301 Uldis Bojars Mon, 14 Dec 2009 03:16:48 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3301 Maybe this Python script can help you a bit: http://gist.github.com/255743 It searches for a given phrase on Bibsonomy and reports entries found. In case if it finds very few matches (as you mention in the blog post) try making the search phrase shorter / simpler. Maybe this Python script can help you a bit: http://gist.github.com/255743

It searches for a given phrase on Bibsonomy and reports entries found. In case if it finds very few matches (as you mention in the blog post) try making the search phrase shorter / simpler.

]]> By: Aldo Chan http://third-bit.com/blog/archives/3307.html#comment-3300 Aldo Chan Mon, 14 Dec 2009 02:46:31 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3300 OK, about fooling google with the User-Agent setting (if you've tested your script against a local web server you'll notice that the User-Agent header is still set to urllib-blah-blah) you'll need to subclass urllib.URLOpener and set the class attribute version to your desired header. like this: http://paste.pocoo.org/show/156877/ OK, about fooling google with the User-Agent setting (if you’ve tested your script against a local web server you’ll notice that the User-Agent header is still set to urllib-blah-blah) you’ll need to subclass urllib.URLOpener and set the class attribute version to your desired header.
like this: http://paste.pocoo.org/show/156877/

]]> By: Eric O. LEBIGOT (EOL) http://third-bit.com/blog/archives/3307.html#comment-3299 Eric O. LEBIGOT (EOL) Sun, 13 Dec 2009 21:30:32 +0000 http://pyre.third-bit.com/blog/?p=3307#comment-3299 Sorry to hear that your attempts have not produced much so far… That's an interesting project! With regards to cb2bib, I'd like to point out that it is available via Fink (at least in the unstable branch). Sorry to hear that your attempts have not produced much so far… That’s an interesting project!

With regards to cb2bib, I’d like to point out that it is available via Fink (at least in the unstable branch).

]]>