Search code examples
amazon-web-servicessolrnutch

Nutch 2.1 (HBase, SOLR) with Amazon Web Services

I experienced Nutch 2.1 locally without any difficulty. I have also tried on a 3 machine distributed cluster. We're now discussing whether to run it with Amazon Web Services or not. I do not have much experience with AWS. My question is that, is it possible and neccessary to try Nutch2.1 crawling and indexing parts on the cloud. What possible advantages and disadvantages we will have?

Thanks.

Solution

  • If you have a cluster with same capacity as that of a AWS cluster (that you plan to invest in) then there is no advantage except for #1 below.

    Here are several factors that you should think about before switching to AWS:

    1. Locality of hosts crawled: If you are sitting in Europe and the websites that you want to crawl are hosted far away ... say Australia. If you buy AWS nodes located in Australia, it would be much faster for crawling that data rather than crawling from Europe.

    2. Cost: For using AWS machines, you need to pay then on hourly basis. Can you afford that ? If not better use your own machines

    3. Current cluster capacity : does your current cluster has ample capacity and space to handle the amount of crawled data ? I think there wont be problem in terms of computational speed as Nutch runs on Hadoop which was designed to run on commodity hardware. Can your cluster accommodate entire data that is being fetched by the crawler.

    4. Volume of data : What is a rough estimate of the data that is being crawled ? If its less, then it makes no sense to have an AWS cluster.

    5. Time constraints : Is there any time bound for completion for the crawl ?

    If you are doing this for a professional project, then these factors must be given a thought.

    If you are doing it for fun/hobby/learning, go ahead and use free tier nodes of AWS. Those are low capacity nodes given free by Amazon. Its fun to learn new things :)

    Advantages of AWS:

    1. No need to buy machines for setting up a cluster. get started without having any hardware except a terminal PC.
    2. Locality
    3. No need to look after machines. If a node crashes badly, leave it (its not your problem :P). Buy a new one, add it to the cluster and go ahead.

    Disadvantages of AWS:

    1. Costly.
    2. Copying data to any machine outside AWS cluster is charged.
    3. Your data is NOT persisted when u give up the procured AWS nodes. If u want to persist it, pay them and use the S3 storage service.