avoid duplicated documents in Solr

Question

while indexing DB documents using SolrJ, I was able to find there were duplicated documents in Solr(5.2.1). I want to avoid duplications and rewrite the documents based on "id" field. with my googling, "dedupe" is useful for duplications, so I applied it to solrconfig.xml but sadly it didn't work .

if there are two same documents then rewrite with latest one. for example,
   "id" = 750000 "title" = here I am
   "id" = 750000 "title" = here you are 
hence, final result would be "id" =750000 "title" = here you are

    //here is my part of schema.xml

    <field name="id" type="long" indexed="true" stored="true" required="true"/>
    <field name="title" type="string" indexed="true" stored="true" required="true" />
    <field name="unique_id" type="string" multiValued="false" indexed="true" required="false" stored="true"
    <uniqueKey>unique_id</uniqueKey>

    //below code is solrconfig.xml   

    <updateRequestProcessorChain name="dedupe">

    <processor class="solr.processor.SignatureUpdateProcessorFactory">
         <bool name="enabled">true</bool>
         <str name="signatureField">id</str>
         <bool name="overwriteDupes">true</bool>
         <str name="fields">id</str>
         <str name="signatureClass">solr.processor.TextProfileSignature</str>
       </processor>

       <processor class="solr.LogUpdateProcessorFactory" />
       <processor class="solr.RunUpdateProcessorFactory" />
     </updateRequestProcessorChain>

need your kind advice.

below code is core parts of my indexing programe with SolrJ (edited on 2015.05.08)

 SolrClient solr = new HttpSolrClient(urlArray[i]); //localhost:8983/solr/#/core_name[i] 
      String id;
      SolrInputDocument doc = new SolrInputDocument();
      UpdateResponse response;
      String[] array;

      for (Map.Entry<String,Object> entry : list.get(i).entrySet()) { // get my DB values such as id, title ,description...

        array = String.valueOf(entry.getValue()).split(","); // split DB values depend on ","
        id = entry.getKey();
        doc.addField("id", entry.getKey()); // unique id
        doc.addField("title", array[1]);

        doc.addField("link", array[2]);
        doc.addField("description", array[3]);
        response = solr.add(doc);

        doc.clear();

      }

      solr.commit();
      solr.close();

Answer 1

make sure to change your update handlers (the ones that you use in SolrJ) to use the defined chain (in your case "dedupe")

<requestHandler name="/update" class="solr.UpdateRequestHandler" >
  <lst name="defaults">
    <str name="update.chain">dedupe</str>
  </lst>
...
</requestHandler>

Have a look in this url https://cwiki.apache.org/confluence/display/solr/De-Duplication