Sometimes when scraping a website I need my script to login in order to access the data of interest. Usually reverse engineering the login form is straightforward, however some websites makes this difficult. For example if login requires passing a CAPTCHA. Or if the website only allows one simultaneous login session per account. For difficult cases such as these I have an alternative solution - manually login to the website of interest in a web browser and then have my script load and reuse the login session.
I have now packaged this solution as an open source python module. Here is some example usage:
If you have a bitbucket account and are logged in in a supported browser then you should see your account name printed here. Currently Firefox (Linux/OSX/Windows) and Chrome (Linux/OSX) are supported and I will add more platforms if get the chance to test.
The latest version of the Apple Apps database now contains 978,698 apps, which is 200,000 more than the previous version. If you have purchased this database previously you can login to your account to download the latest version for free.
I often use Google’s geocoding API to find details about a location like this:
The drawback of this approach is the Google API limits each user to 2500 requsts per 24 hours. So if I want to geocode 1 million locations then I would need to rent a lot of proxies or else the API calls will take over a year to complete (1,000,000 / 2,500 = 400 days). To meet this use case I built a module to reverse geocode a latitude / longitude coordinate using a list of known locations from geonames.
Here is some example usage:
Internally the module uses a k-d tree to efficiently find the nearest neighbour of each given coordinate. On my netbook I find building the tree takes ~2.5 seconds and then each location query just ~1.5 ms.
The module is licensed under the LGPL on bitbucket: https://bitbucket.org/richardpenman/reverse_geocode
A few friends asked me what web services I use to run my business so I am writing this to point people in future.
To manage email I originally used Google Apps. When I signed up this was free for 10 users but now costs $50 per user for new accounts. I eventually found email not structured enough so built my own basic ticket system on Google App Engine. Then I came across Zendesk, which did everything my custom system supported and was better designed so migrated there.
For invoicing I originally just sent an email with the price but quickly discovered companies require something more official looking. For version 2 I found a Google Docs template and then modified this with the client details for each project. This was cumbersome so I built a Zendesk extension to automatically generate the invoices from ticket details. For someone non-technical I would recommend Freshbooks.
For crawling I originally used my home computer. When that became too slow I rented an Amazon EC2 server but I found this overly expensive because they charge per IO operation and bandwidth. Then I tried several dedicated hosting providers and am currently on Digital Ocean.
For hosting this website I have used WebFaction for over 5 years now and several times they have increased the server specs and features while keeping price constant. My only complaint is their changing SMTP policy caused me to miss some client emails.
For sharing files I use Google Storage. This is a similar service to S3 but I was given a year free hosting and it comes with a nice command line tool and web interface, so I have never considered migrating.
Basically I would recommend to start simple and invest in a better tool when you know you need it. Your business will probably fail anyway! So don’t spend your time configuring a helpdesk system before you have any clients.
For the next year I am going to be working onsite at a web scraping focussed startup in New York. Looking forward to the experience! I found that working in the US is straightforward for Australian’s because of the fantastic E3 visa. I just took my job offer letter to the US consulate along with some documentation, paid a few hundred dollars, and within a fortnight I had a 2 year work visa that can be extended indefinitely.
During my time in New York I will keep working with existing projects but limit the new clients I take on to make room for the day job.
There is a nice website screenshots.com that hosts historic screenshots for many websites. This post will show how to generate our own historic screenshots with python.
The source of historic webpages will be the Wayback Machine, which has been caching webpages since 1996. I found that a new beta version is being tested, available here. Unfortunately the default version does not include any new caches since mid 2011 so this script will need to depend on the beta.
The Wayback Machine has a convenient URL structure based on timestamp:
http://web-beta.archive.org/web/YYYYMMDDHHMMSS/website http://web-beta.archive.org/web/20121030031942/http://www.amazon.com/ http://web-beta.archive.org/web/20121123023424/http://www.amazon.com/
If just a partial timestamp is sent then the server will redirect to the nearest match. For example 2011 redirects to December 31, 2011:
http://web-beta.archive.org/web/2011/http://www.amazon.com/ -> http://web-beta.archive.org/web/20111231235154/http://www.amazon.com/
The first step is finding how far back the history is available for a webpage. At the top of the cached page for amazon.com is the message: http://amazon.com has been crawled 10,271 times going all the way back to December 12, 1998.
The link here contains a timestamp for the earliest date, which will be parsed into a datetime object:
Next need to iterate from this earliest timestamp to the present and download each of the cached pages in-between. The below snippet uses increments of a year between each timestamp download:
Webkit is then used to generate the screenshot of each downloaded webpage. To make the results prettier the Wayback toolbar is removed before rendering:
This script includes a feature to generate a webpage with the screenshots. Here is an example history from Yahoo (follow the image link to view the complete history):
This is interesting - from 1996 to 2004 the Yahoo homepage became progressively more complex, then 2006 - 2009 drastically simpler, and then 2010 onwards returns to the traditional portal site. They must have been trying to compete with Google’s simple approach but that didn’t work so went back to being a media company.
These are among the better results I have generated so far. Often I found the Wayback Machine crawler was blocked or returned tacky domain squatter websites. If you find a website with an interesting history let me know.