org.apache.nutch.protocol.http.api

## Class HttpBase

- java.lang.Object
- - org.apache.nutch.protocol.http.api.HttpBase

- All Implemented Interfaces:
    Configurable, Pluggable, Protocol
  Direct Known Subclasses:
    Http, Http, Http, Http, Http, OkHttp

      public abstract class HttpBase
      extends Object
      implements Protocol

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String` | `accept` The "Accept" request header value.
    `protected String` | `acceptCharset` The "Accept-Charset" request header value.
    `protected String` | `acceptLanguage` The "Accept-Language" request header value.
    `static int` | `BUFFER_SIZE`
    `static Text` | `COOKIE`
    `protected boolean` | `enableCookieHeader` Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata
    `protected boolean` | `enableIfModifiedsinceHeader` Configuration directive for If-Modified-Since HTTP header
    `protected int` | `maxContent` The length limit for downloaded content, in bytes.
    `protected long` | `maxCrawlDelay` Skip page if Crawl-Delay longer than this value.
    `protected int` | `maxDuration` The time limit to download the entire content, in seconds.
    `protected boolean` | `partialAsTruncated` Whether to save partial fetches as truncated content.
    `protected HashMap<String,String>` | `proxyException` The proxy exception list.
    `protected String` | `proxyHost` The proxy hostname.
    `protected int` | `proxyPort` The proxy port.
    `protected Proxy.Type` | `proxyType` The proxy port.
    `static Text` | `RESPONSE_TIME`
    `protected boolean` | `responseTime` Record response time in CrawlDatum's meta data, see property http.store.responsetime.
    `protected boolean` | `storeHttpHeaders` Record the HTTP response header in the metadata, see property `store.http.headers`.
    `protected boolean` | `storeHttpRequest` Record the HTTP request in the metadata, see property `store.http.request`.
    `protected boolean` | `storeIPAddress` Record the IP address of the responding server, see property `store.ip.address`.
    `protected int` | `timeout` The network timeout in millisecond
    `protected boolean` | `tlsCheckCertificate` Whether to check TLS/SSL certificates
    `protected Set<String>` | `tlsPreferredCipherSuites` Which TLS/SSL cipher suites to support
    `protected Set<String>` | `tlsPreferredProtocols` Which TLS/SSL protocols to support
    `protected boolean` | `useHttp11` Do we use HTTP/1.1?
    `protected boolean` | `useHttp2` Whether to use HTTP/2
    `protected boolean` | `useProxy` Indicates if a proxy is used
    `protected String` | `userAgent` The Nutch 'User-Agent' request header

    - ### Fields inherited from interface org.apache.nutch.protocol.Protocol

      `X_POINT_ID`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `HttpBase()` Creates a new instance of HttpBase
    `HttpBase(org.slf4j.Logger logger)` Creates a new instance of HttpBase

  - ### Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `String` | `getAccept()`
    `String` | `getAcceptCharset()`
    `String` | `getAcceptLanguage()` Value of "Accept-Language" request header sent by Nutch.
    `Configuration` | `getConf()`
    `String` | `getCookie(URL url)` If per-host cookies are configured, this method will look it up for the given url.
    `int` | `getMaxContent()`
    `int` | `getMaxDuration()` The time limit to download the entire content, in seconds.
    `ProtocolOutput` | `getProtocolOutput(Text url, CrawlDatum datum)` Returns the `Content` for a fetchlist entry.
    `String` | `getProxyHost()`
    `int` | `getProxyPort()`
    `protected abstract Response` | `getResponse(URL url, CrawlDatum datum, boolean followRedirects)`
    `crawlercommons.robots.BaseRobotRules` | `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)` Retrieve robot rules applicable for this URL.
    `int` | `getTimeout()`
    `Set<String>` | `getTlsPreferredCipherSuites()`
    `Set<String>` | `getTlsPreferredProtocols()`
    `boolean` | `getUseHttp11()`
    `String` | `getUserAgent()`
    `boolean` | `isCookieEnabled()`
    `boolean` | `isIfModifiedSinceEnabled()`
    `boolean` | `isStoreHttpHeaders()`
    `boolean` | `isStoreHttpRequest()`
    `boolean` | `isStoreIPAddress()`
    `boolean` | `isStorePartialAsTruncated()` Whether to save partial fetches as truncated content, cf.
    `boolean` | `isTlsCheckCertificates()`
    `protected void` | `logConf()`
    `protected static void` | `main(HttpBase http, String[] args)`
    `byte[]` | `processDeflateEncoded(byte[] compressed, URL url)`
    `byte[]` | `processGzipEncoded(byte[] compressed, URL url)`
    `void` | `setConf(Configuration conf)`
    `boolean` | `useProxy(String host)`
    `boolean` | `useProxy(URI uri)`
    `boolean` | `useProxy(URL url)`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### RESPONSE_TIME

          public static final Text RESPONSE_TIME

    - #### COOKIE

          public static final Text COOKIE

    - #### BUFFER_SIZE

          public static final int BUFFER_SIZE

      See Also:
        Constant Field Values

    - #### proxyHost

          protected String proxyHost

      The proxy hostname.

    - #### proxyPort

          protected int proxyPort

      The proxy port.

    - #### proxyType

          protected Proxy.Type proxyType

      The proxy port.

    - #### proxyException

          protected HashMap<String,String> proxyException

      The proxy exception list.

    - #### useProxy

          protected boolean useProxy

      Indicates if a proxy is used

    - #### timeout

          protected int timeout

      The network timeout in millisecond

    - #### maxContent

          protected int maxContent

      The length limit for downloaded content, in bytes.

    - #### maxDuration

          protected int maxDuration

      The time limit to download the entire content, in seconds.

    - #### partialAsTruncated

          protected boolean partialAsTruncated

      Whether to save partial fetches as truncated content.

    - #### userAgent

          protected String userAgent

      The Nutch 'User-Agent' request header

    - #### acceptLanguage

          protected String acceptLanguage

      The "Accept-Language" request header value.

    - #### acceptCharset

          protected String acceptCharset

      The "Accept-Charset" request header value.

    - #### accept

          protected String accept

      The "Accept" request header value.

    - #### useHttp11

          protected boolean useHttp11

      Do we use HTTP/1.1?

    - #### useHttp2

          protected boolean useHttp2

      Whether to use HTTP/2

    - #### responseTime

          protected boolean responseTime

      Record response time in CrawlDatum's meta data, see property http.store.responsetime.

    - #### storeIPAddress

          protected boolean storeIPAddress

      Record the IP address of the responding server, see property `store.ip.address`.

    - #### storeHttpRequest

          protected boolean storeHttpRequest

      Record the HTTP request in the metadata, see property `store.http.request`.

    - #### storeHttpHeaders

          protected boolean storeHttpHeaders

      Record the HTTP response header in the metadata, see property `store.http.headers`.

    - #### maxCrawlDelay

          protected long maxCrawlDelay

      Skip page if Crawl-Delay longer than this value.

    - #### tlsCheckCertificate

          protected boolean tlsCheckCertificate

      Whether to check TLS/SSL certificates

    - #### tlsPreferredProtocols

          protected Set<String> tlsPreferredProtocols

      Which TLS/SSL protocols to support

    - #### tlsPreferredCipherSuites

          protected Set<String> tlsPreferredCipherSuites

      Which TLS/SSL cipher suites to support

    - #### enableIfModifiedsinceHeader

          protected boolean enableIfModifiedsinceHeader

      Configuration directive for If-Modified-Since HTTP header

    - #### enableCookieHeader

          protected boolean enableCookieHeader

      Controls whether or not to set Cookie HTTP header based on CrawlDatum metadata

  - ### Constructor Detail

    - #### HttpBase

          public HttpBase()

      Creates a new instance of HttpBase

    - #### HttpBase

          public HttpBase(org.slf4j.Logger logger)

      Creates a new instance of HttpBase

  - ### Method Detail

    - #### setConf

          public void setConf(Configuration conf)

      Specified by:
        `setConf` in interface `Configurable`

    - #### getConf

          public Configuration getConf()

      Specified by:
        `getConf` in interface `Configurable`

    - #### getProtocolOutput

          public ProtocolOutput getProtocolOutput(Text url,
                                                  CrawlDatum datum)

      Description copied from interface: `Protocol`
      Returns the `Content` for a fetchlist entry.
      Specified by:
        `getProtocolOutput` in interface `Protocol`

    - #### getProxyHost

          public String getProxyHost()

    - #### getProxyPort

          public int getProxyPort()

    - #### useProxy

          public boolean useProxy(URL url)

    - #### useProxy

          public boolean useProxy(URI uri)

    - #### useProxy

          public boolean useProxy(String host)

    - #### getTimeout

          public int getTimeout()

    - #### isIfModifiedSinceEnabled

          public boolean isIfModifiedSinceEnabled()

    - #### isCookieEnabled

          public boolean isCookieEnabled()

    - #### isStoreIPAddress

          public boolean isStoreIPAddress()

    - #### isStoreHttpRequest

          public boolean isStoreHttpRequest()

    - #### isStoreHttpHeaders

          public boolean isStoreHttpHeaders()

    - #### getMaxContent

          public int getMaxContent()

    - #### getMaxDuration

          public int getMaxDuration()

      The time limit to download the entire content, in seconds. See the property `http.time.limit`.

    - #### isStorePartialAsTruncated

          public boolean isStorePartialAsTruncated()

      Whether to save partial fetches as truncated content, cf. the property `http.partial.truncated`.

    - #### getUserAgent

          public String getUserAgent()

    - #### getCookie

          public String getCookie(URL url)

      If per-host cookies are configured, this method will look it up for the given url.
      Parameters:
        `url` - the url to look-up a cookie for
      Returns:
        the cookie or null

    - #### getAcceptLanguage

          public String getAcceptLanguage()

      Value of "Accept-Language" request header sent by Nutch.
      Returns:
        The value of the header "Accept-Language" header.

    - #### getAcceptCharset

          public String getAcceptCharset()

    - #### getAccept

          public String getAccept()

    - #### getUseHttp11

          public boolean getUseHttp11()

    - #### isTlsCheckCertificates

          public boolean isTlsCheckCertificates()

    - #### getTlsPreferredCipherSuites

          public Set<String> getTlsPreferredCipherSuites()

    - #### getTlsPreferredProtocols

          public Set<String> getTlsPreferredProtocols()

    - #### logConf

          protected void logConf()

    - #### processGzipEncoded

          public byte[] processGzipEncoded(byte[] compressed,
                                           URL url)
                                    throws IOException

      Throws:
        `IOException`

    - #### processDeflateEncoded

          public byte[] processDeflateEncoded(byte[] compressed,
                                              URL url)
                                       throws IOException

      Throws:
        `IOException`

    - #### main

          protected static void main(HttpBase http,
                                     String[] args)
                              throws Exception

      Throws:
        `Exception`

    - #### getResponse

          protected abstract Response getResponse(URL url,
                                                  CrawlDatum datum,
                                                  boolean followRedirects)
                                           throws ProtocolException,
                                                  IOException

      Throws:
        `ProtocolException`
        `IOException`

    - #### getRobotRules

          public crawlercommons.robots.BaseRobotRules getRobotRules(Text url,
                                                                    CrawlDatum datum,
                                                                    List<Content> robotsTxtContent)

      Description copied from interface: `Protocol`
      Retrieve robot rules applicable for this URL.
      Specified by:
        `getRobotRules` in interface `Protocol`
      Parameters:
        `url` - URL to check
        `datum` - page datum
        `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
      Returns:
        robot rules (specific for this URL or default), never null

Copyright © 2021 The Apache Software Foundation