I am making an web scraping app by using Rails 4. I chose capybara-webkit for browser engine. It looks the scraping app is running well and I can parse the data what I need.
The problem is memory. The process called "webkit-server" is taking lots of memories while my app is performing its scraping process and I am assuming that capybra-webkit is using the process by somehow.
Since, it tends to occupy about 1GB after scraping few hundreds pages, I want to limit/clear the memory occupation.Is there any way to do that?
It looks Selenium is free from the memory issue (keeps occupying not more than 300mb around), but it runs slower than webkit.
You need to run reset! occasionally on the driver instance.
The driver is written with automated test suites in mind, and it doesn't perform some kinds of cleanup until reset! is called. It's normally invoked between each test.
If the process is holding onto an unusual amount of memory after calling reset!, you've found a bug in the driver.