org.apache.nutch.scoring.opic

## Class OPICScoringFilter

- java.lang.Object
- - org.apache.nutch.scoring.opic.OPICScoringFilter

- All Implemented Interfaces:
    Configurable, Pluggable, ScoringFilter

      public class OPICScoringFilter
      extends Object
      implements ScoringFilter

  This plugin implements a variant of an Online Page Importance Computation (OPIC) score, described in this paper: Abiteboul, Serge and Preda, Mihai and Cobena, Gregory (2003), Adaptive On-Line Page Importance Computation.
  Author:
    Andrzej Bialecki

- - ### Field Summary

    - ### Fields inherited from interface org.apache.nutch.scoring.ScoringFilter

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `OPICScoringFilter()`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `CrawlDatum` | `distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount)` Get a float value from Fetcher.SCORE_KEY, divide it by the number of outlinks and apply.
    `float` | `generatorSortValue(Text url, CrawlDatum datum, float initSort)` Use `CrawlDatum.getScore()`.
    `Configuration` | `getConf()`
    `float` | `indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)` Dampen the boost value by scorePower.
    `void` | `initialScore(Text url, CrawlDatum datum)` Set to 0.0f (unknown value) - inlink contributions will bring it to a correct level.
    `void` | `injectedScore(Text url, CrawlDatum datum)` Set an initial score for newly injected pages.
    `void` | `passScoreAfterParsing(Text url, Content content, Parse parse)` Copy the value from Content metadata under Fetcher.SCORE_KEY to parseData.
    `void` | `passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)` Store a float value of CrawlDatum.getScore() under Fetcher.SCORE_KEY.
    `void` | `setConf(Configuration conf)`
    `void` | `updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked)` Increase the score by a sum of inlinked scores.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface org.apache.nutch.scoring.ScoringFilter

      `orphanedScore`

- - ### Constructor Detail

    - #### OPICScoringFilter

          public OPICScoringFilter()

  - ### Method Detail

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`

    - #### injectedScore

          public void injectedScore(Text url,
                                    CrawlDatum datum)
                             throws ScoringFilterException

      Description copied from interface: `ScoringFilter`
      Set an initial score for newly injected pages. Note: newly injected pages may have no inlinks, so filter implementations may wish to set this score to a non-zero value, to give newly injected pages some initial credit.
      Specified by:
        `injectedScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException`

    - #### initialScore

          public void initialScore(Text url,
                                   CrawlDatum datum)
                            throws ScoringFilterException

      Set to 0.0f (unknown value) - inlink contributions will bring it to a correct level. Newly discovered pages have at least one inlink.
      Specified by:
        `initialScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - new datum. Filters will modify it in-place.
      Throws:
        `ScoringFilterException`

    - #### generatorSortValue

          public float generatorSortValue(Text url,
                                          CrawlDatum datum,
                                          float initSort)
                                   throws ScoringFilterException

      Use `CrawlDatum.getScore()`.
      Specified by:
        `generatorSortValue` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - page's datum, should not be modified
        `initSort` - initial sort value, or a value from previous filters in chain
      Throws:
        `ScoringFilterException`

    - #### updateDbScore

          public void updateDbScore(Text url,
                                    CrawlDatum old,
                                    CrawlDatum datum,
                                    List<CrawlDatum> inlinked)
                             throws ScoringFilterException

      Increase the score by a sum of inlinked scores.
      Specified by:
        `updateDbScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `old` - original datum, with original score. May be null if this is a newly discovered page. If not null, filters should use score values from this parameter as the starting values - the `datum` parameter may contain values that are no longer valid, if other updates occurred between generation and this update.
        `datum` - the new datum, with the original score saved at the time when fetchlist was generated. Filters should update this in-place, and it will be saved in the crawldb.
        `inlinked` - (partial) list of CrawlDatum-s (with their scores) from links pointing to this page, found in the current update batch.
      Throws:
        `ScoringFilterException`

    - #### passScoreBeforeParsing

          public void passScoreBeforeParsing(Text url,
                                             CrawlDatum datum,
                                             Content content)

      Store a float value of CrawlDatum.getScore() under Fetcher.SCORE_KEY.
      Specified by:
        `passScoreBeforeParsing` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `datum` - source datum. NOTE: modifications to this value are not persisted.
        `content` - instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.

    - #### passScoreAfterParsing

          public void passScoreAfterParsing(Text url,
                                            Content content,
                                            Parse parse)

      Copy the value from Content metadata under Fetcher.SCORE_KEY to parseData.
      Specified by:
        `passScoreAfterParsing` in interface `ScoringFilter`
      Parameters:
        `url` - page url
        `content` - original content. NOTE: modifications to this value are not persisted.
        `parse` - target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.

    - #### distributeScoreToOutlinks

          public CrawlDatum distributeScoreToOutlinks(Text fromUrl,
                                                      ParseData parseData,
                                                      Collection<Map.Entry<Text,CrawlDatum>> targets,
                                                      CrawlDatum adjust,
                                                      int allCount)
                                               throws ScoringFilterException

      Get a float value from Fetcher.SCORE_KEY, divide it by the number of outlinks and apply.
      Specified by:
        `distributeScoreToOutlinks` in interface `ScoringFilter`
      Parameters:
        `fromUrl` - url of the source page
        `parseData` - ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.
        `targets` - <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.
        `adjust` - a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status to `CrawlDatum.STATUS_LINKED`.
        `allCount` - number of all collected outlinks from the source page
      Returns:
        if needed, implementations may return an instance of CrawlDatum, with status `CrawlDatum.STATUS_LINKED`, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed.
      Throws:
        `ScoringFilterException`

    - #### indexerScore

          public float indexerScore(Text url,
                                    NutchDocument doc,
                                    CrawlDatum dbDatum,
                                    CrawlDatum fetchDatum,
                                    Parse parse,
                                    Inlinks inlinks,
                                    float initScore)
                             throws ScoringFilterException

      Dampen the boost value by scorePower.
      Specified by:
        `indexerScore` in interface `ScoringFilter`
      Parameters:
        `url` - url of the page
        `doc` - indexed document. NOTE: this already contains all information collected by indexing filters. Implementations may modify this instance, in order to store/remove some information.
        `dbDatum` - current page from CrawlDb. NOTE:
        - changes made to this instance are not persisted
        - may be null if indexing is done without CrawlDb or if the segment is generated not from the CrawlDb (via FreeGenerator).
        `fetchDatum` - datum from FetcherOutput (containing among others the fetching status)
        `parse` - parsing result. NOTE: changes made to this instance are not persisted.
        `inlinks` - current inlinks from LinkDb. NOTE: changes made to this instance are not persisted.
        `initScore` - initial boost value for the indexed document.
      Returns:
        boost value for the indexed document. This value is passed as an argument to the next scoring filter in chain. NOTE: implementations may also express other scoring strategies by modifying the indexed document directly.
      Throws:
        `ScoringFilterException`

Copyright © 2021 The Apache Software Foundation