Python and other scripting languages are sometimes dismissed because of their inefficiency compared to compiled languages like C. For example here are implementations of the fibonacci sequence in C and Python:
And here are the execution times:
$ time ./fib 3.099s $ time python fib.py 16.655s
As expected C has a much faster execution time - 5x faster in this case.
In the context of web scraping, executing instructions is less important because the bottleneck is I/O - downloading the webpages. But I use Python in other contexts too so let's see if we can do better.
First install psyco. On Linux this is just:
sudo apt-get install python-psyco
Then modify the Python script to call psyco:
import psyco psyco.full() def fib(n): if n < 2: return n else: return fib(n - 1) + fib(n - 2) fib(40)
And here is the updated execution time:
$ time python fib.py 3.190s
Just 3 seconds - with psyco the execution time is now equivalent to the C example! Psyco achieves this by compiling code on the fly to avoid interpreting each line.
I now add the below snippet to most of my Python scripts to take advantage of psyco when installed:
try: import psyco psyco.full() except ImportError: pass # psyco not installed so continue as usual