Search code examples
pythonwebkitweb-crawlersquid

Does webkit crawler need to use squid proxy?

I am writing a crawler using webkit, does webkit cache stuffs? Do I need to use squid as a proxy for my webkit based crawler?

Solution

  • Are you using QWebKit? PyQt/PySide doesn't use disk cache by default. You have to set a QNetworkDiskCache object to QNetworkManager to enable caching.

    See the source code of webkit.py module in webscraping library, it shows how to enable caching in QWebkit.