I've been using Perl for years to crawl and scrape for various different purposes, one thing that's always bugged me is while there are tons of great CPAN modules for small-scale scraping and crawling, such as LWP, WWW::Mechanize, Web::Scraper, AnyEvent::HTTP, and now Mojo::UserAgent, there don't appear to be any crawling frameworks in the same way as there are for other languages.
For example Apache Nutch (/Droids) & Scrapy (Python).
Anybody know of any projects at all in Perl that are equivalent?
You probably need to take a look at modules such as HTML::Robot::Scraper or HTTP::UserAgentString::Robot and I think there are a few more with robot in their name.