Nutch Dedup Question

Hi,

I am little confused about what exactly dedup does?

a. Does dedup delete duplicate documents from Index and
Segments?

b. Is there a way that we could delete duplicated documents
for two
segments? 

Let me know. Thanks.

-- 
View this message in context: http://www.nabble.com/Nutch-Dedup-Question-tf
4488321.html#a12799680
Sent from the Nutch - User mailing list archive at
Nabble.com.

karthik085 wrote:
> Hi,
> 
> I am little confused about what exactly dedup does?
> 
> a. Does dedup delete duplicate documents from Index and
Segments?

Only from the index.

> 
> b. Is there a way that we could delete duplicated
documents for two
> segments? 

bin/nutch mergesegs


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||/|  Information Retrieval, Semantic Web
___|||__||  |  ||  |  Embedded Unix, System Integration
http://www.sigram.com 
Contact: info at sigram dot com

Thanks - that's much clearer.


Andrzej Bialecki wrote:
> 
> karthik085 wrote:
>> Hi,
>> 
>> I am little confused about what exactly dedup
does?
>> 
>> a. Does dedup delete duplicate documents from Index
and Segments?
> 
> Only from the index.
> 
>> 
>> b. Is there a way that we could delete duplicated
documents for two
>> segments? 
> 
> bin/nutch mergesegs
> 
> 
> -- 
> Best regards,
> Andrzej Bialecki     <><
>   ___. ___ ___ ___ _ _  
__________________________________
> [__ || __|__/|__||/|  Information Retrieval, Semantic
Web
> ___|||__||  |  ||  |  Embedded Unix, System
Integration
> http://www.sigram.com 
Contact: info at sigram dot com
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Nutch-Dedup-Question-tf
4488321.html#a12801358
Sent from the Nutch - User mailing list archive at
Nabble.com.