org.apache.nutch.crawl

## Class DeduplicationJob

- java.lang.Object
- - org.apache.hadoop.conf.Configured
  - - org.apache.nutch.util.NutchTool
    - - org.apache.nutch.crawl.DeduplicationJob

- All Implemented Interfaces:
    Configurable, Tool

      public class DeduplicationJob
      extends NutchTool
      implements Tool

  Generic deduplicator which groups fetched URLs with the same digest and marks all of them as duplicate except the one with the highest score (based on the score in the crawldb, which is not necessarily the same as the score indexed). If two (or more) documents have the same score, then the document with the latest timestamp is kept. If the documents have the same timestamp then the one with the shortest URL is kept. The documents marked as duplicate can then be deleted with the command CleaningJob.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `DeduplicationJob.DBFilter`
    `static class` | `DeduplicationJob.DedupReducer`
    `static class` | `DeduplicationJob.StatusUpdateReducer` Combine multiple new entries for a url.

  - ### Field Summary

    - ### Fields inherited from class org.apache.nutch.util.NutchTool

      `currentJob, currentJobNum, numJobs, results, status`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `DeduplicationJob()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static void` | `main(String[] args)`
    `Map<String,Object>` | `run(Map<String,Object> args, String crawlId)` Runs the tool, using a map of arguments.
    `int` | `run(String[] args)`

    - ### Methods inherited from class org.apache.nutch.util.NutchTool

      `getProgress, getStatus, killJob, stopJob`

    - ### Methods inherited from class org.apache.hadoop.conf.Configured

      `getConf, setConf`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.hadoop.conf.Configurable

      `getConf, setConf`

- - ### Constructor Detail

    - #### DeduplicationJob

          public DeduplicationJob()

  - ### Method Detail

    - #### run

          public int run(String[] args)
                  throws IOException

      Specified by:
        `run` in interface `Tool`
      Throws:
        `IOException`

    - #### main

          public static void main(String[] args)
                           throws Exception

      Throws:
        `Exception`

    - #### run

          public Map<String,Object> run(Map<String,Object> args,
                                        String crawlId)
                                 throws Exception

      Description copied from class: `NutchTool`
      Runs the tool, using a map of arguments. May return results, or null.
      Specified by:
        `run` in class `NutchTool`
      Throws:
        `Exception`

Copyright © 2021 The Apache Software Foundation