I wanna disable robots.txt checking in Nutch and crawl everything from websites.
Disable means before fetching or parsing any website, skip checking robot.txt.
Is this possible?
as far as i understand we are not able to disable robots.txt in nutch.