When scraping a website, typically the majority of time is spent waiting for the data to download. So to be efficient I work on multiple scraping projects simultaneously.
Blog
-
Web Scraping User Interface
Crawling Business Web2py December 29, 2012
-
Why web2py?
Web2py May 03, 2010
In a previous post I mentioned that web2py is my weapon of choice for building web applications. Before web2py I had learnt a variety of approaches to building dynamic websites (raw PHP, Python CGI, Turbogears, Symfony, Rails, Django), but find myself most productive with web2py.
This is because web2py: