How to make python faster
Posted 01 Feb 2012 in efficiency and python

Python and other scripting languages are sometimes dismissed because of their inefficiency compared to compiled languages like C. For example here are implementations of the fibonacci sequence in C and Python:

int fib(int n){
   if (n < 2)
     return n;
   else
     return fib(n - 1) + fib(n - 2);
}

int main() {
    fib(40);
    return 0;
}
def fib(n):
  if n < 2:
     return n
  else:
     return fib(n - 1) + fib(n - 2)
fib(40)

And here are the execution times:

$ time ./fib
3.099s
$ time python fib.py
16.655s

As expected C has a much faster execution time - 5x faster in this case.

In the context of web scraping, executing instructions is less important because the bottleneck is I/O - downloading the webpages. But I use Python in other contexts too so let's see if we can do better.

First install psyco. On Linux this is just:

sudo apt-get install python-psyco

Then modify the Python script to call psyco:

import psyco
psyco.full()

def fib(n):
  if n < 2:
     return n
  else:
     return fib(n - 1) + fib(n - 2)
fib(40)

And here is the updated execution time:

$ time python fib.py
3.190s

Just 3 seconds - with psyco the execution time is now equivalent to the C example! Psyco achieves this by compiling code on the fly to avoid interpreting each line.

I now add the below snippet to most of my Python scripts to take advantage of psyco when installed:

try:
    import psyco
    psyco.full()
except ImportError:
    pass # psyco not installed so continue as usual

blog comments powered by Disqus