$ sudo python setup.py install
But there is no uninstall option and the manual removal of the files seems the only way.
Today I stumbled on the checkinstall tool. Here's what the manual say:
checkinstall is a program that monitors an installation procedure (such as make install, install.sh), and creates a standard package for your distribution (currently deb, rpm and tgz packages are supported) that you can install through your distribution's package management system (dpkg, rpm or installpkg).
Good thing it can also be used on python source packages, and it's really easy:
$ sudo checkinstall python setup.py install
This will prompt user for some answers and best thing is to use the default ones (in other words, just press enter). After this checkinstall will create a standard package file, and install it, in my case a .deb package. The module can now be uninstalled easily with a package management program.