Automating webkit

I have received some inquiries about using webkit for web scraping, so here is an example using the webscraping module:

from webscraping import webkit
w = webkit.WebkitBrowser(gui=True) 
# load webpage
# fill search textbox 
w.fill('input[id=search_form_input_homepage]', 'sitescraper')
# take screenshot of browser
# click search button'input[id=search_button_homepage]')
# wait on results page
# take another screenshot

Here are the screenshots saved:

I often use webkit when working with websites that rely heavily on JavaScript.

Source code is available on bitbucket.

blog comments powered by Disqus