Automating webkit
Posted 14 Feb 2012 in example, python, qt, and webkit

I have received some inquiries about using webkit for web scraping, so here is an example using the webscraping module:

from webscraping import webkit
w = webkit.WebkitBrowser(gui=True) 
# load webpage
w.get('http://duckduckgo.com')
# fill search textbox 
w.fill('input[id=search_form_input_homepage]', 'sitescraper')
# take screenshot of browser
w.screenshot('duckduckgo_search.jpg')
# click search button 
w.click('input[id=search_button_homepage]')
# wait on results page
w.wait(10)
# take another screenshot
w.screenshot('duckduckgo_results.jpg')

Here are the screenshots saved:

I often use webkit when working with websites that rely heavily on JavaScript.

Source code is available on bitbucket.

blog comments powered by Disqus