CrawlConfig (edu.uci.ics:crawler4j 4.4.0 API)

java.lang.Object
- edu.uci.ics.crawler4j.crawler.CrawlConfig

```
public class CrawlConfig
extends Object
```

Constructor Summary

Constructors
Constructor and Description

CrawlConfig()

Constructors
Constructor and Description
`CrawlConfig()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addAuthInfo(AuthInfo authInfo)`
`List<AuthInfo>`	`getAuthInfos()`
`int`	`getCleanupDelaySeconds()`
`int`	`getConnectionTimeout()`
`String`	`getCookiePolicy()`
`org.apache.http.client.CookieStore`	`getCookieStore()` Gets the configured `CookieStore` or null if none is set
`String`	`getCrawlStorageFolder()`
`long`	`getDbLockTimeout()`
`Collection<org.apache.http.message.BasicHeader>`	`getDefaultHeaders()` Return a copy of the default header collection.
`org.apache.http.conn.DnsResolver`	`getDnsResolver()`
`int`	`getMaxConnectionsPerHost()`
`int`	`getMaxDepthOfCrawling()`
`int`	`getMaxDownloadSize()`
`int`	`getMaxOutgoingLinksToFollow()`
`int`	`getMaxPagesToFetch()`
`int`	`getMaxTotalConnections()`
`int`	`getPolitenessDelay()`
`String`	`getProxyHost()`
`String`	`getProxyPassword()`
`int`	`getProxyPort()`
`String`	`getProxyUsername()`
`int`	`getSocketTimeout()`
`int`	`getThreadMonitoringDelaySeconds()`
`int`	`getThreadShutdownDelaySeconds()`
`String`	`getUserAgentString()`
`boolean`	`isFollowRedirects()`
`boolean`	`isIncludeBinaryContentInCrawling()`
`boolean`	`isIncludeHttpsPages()`
`boolean`	`isOnlineTldListUpdate()`
`boolean`	`isProcessBinaryContentInCrawling()`
`boolean`	`isRespectNoFollow()` Gets the current `CookieStore` used
`boolean`	`isRespectNoIndex()`
`boolean`	`isResumableCrawling()`
`boolean`	`isShutdownOnEmptyQueue()`
`void`	`setAuthInfos(List<AuthInfo> authInfos)`
`void`	`setCleanupDelaySeconds(int delay)`
`void`	`setConnectionTimeout(int connectionTimeout)`
`void`	`setCookiePolicy(String cookiePolicy)`
`void`	`setCookieStore(org.apache.http.client.CookieStore cookieStore)` Sets the `to be used`
`void`	`setCrawlStorageFolder(String crawlStorageFolder)` The folder which will be used by crawler for storing the intermediate crawl data.
`void`	`setDbLockTimeout(long dbLockTimeout)` Set the lock timeout for the underlying sleepycat DB, in milliseconds.
`void`	`setDefaultHeaders(Collection<? extends org.apache.http.Header> defaultHeaders)` Set the default header collection (creating copies of the provided headers).
`void`	`setDnsResolver(org.apache.http.conn.DnsResolver dnsResolver)` DNS resolver to use, #`SystemDefaultDnsResolver()` is default.
`void`	`setFollowRedirects(boolean followRedirects)`
`void`	`setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)`
`void`	`setIncludeHttpsPages(boolean includeHttpsPages)`
`void`	`setMaxConnectionsPerHost(int maxConnectionsPerHost)`
`void`	`setMaxDepthOfCrawling(int maxDepthOfCrawling)` Maximum depth of crawling For unlimited depth this parameter should be set to -1
`void`	`setMaxDownloadSize(int maxDownloadSize)`
`void`	`setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)`
`void`	`setMaxPagesToFetch(int maxPagesToFetch)` Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
`void`	`setMaxTotalConnections(int maxTotalConnections)`
`void`	`setOnlineTldListUpdate(boolean online)` Should the TLD list be updated automatically on each run? Alternatively, it can be loaded from the embedded tld-names.txt resource file that was obtained from https://publicsuffix.org/list/effective_tld_names.dat
`void`	`setPolitenessDelay(int politenessDelay)` Politeness delay in milliseconds (delay between sending two requests to the same host).
`void`	`setProcessBinaryContentInCrawling(boolean processBinaryContentInCrawling)` Should we process binary content such as images, audio, ...
`void`	`setProxyHost(String proxyHost)`
`void`	`setProxyPassword(String proxyPassword)` If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
`void`	`setProxyPort(int proxyPort)`
`void`	`setProxyUsername(String proxyUsername)`
`void`	`setRespectNoFollow(boolean respectNoFollow)`
`void`	`setRespectNoIndex(boolean respectNoIndex)`
`void`	`setResumableCrawling(boolean resumableCrawling)` If this feature is enabled, you would be able to resume a previously stopped/crashed crawl.
`void`	`setShutdownOnEmptyQueue(boolean shutdown)` Should the crawler stop running when the queue is empty?
`void`	`setSocketTimeout(int socketTimeout)`
`void`	`setThreadMonitoringDelaySeconds(int delay)`
`void`	`setThreadShutdownDelaySeconds(int delay)`
`void`	`setUserAgentString(String userAgentString)` user-agent string that is used for representing your crawler to web servers.
`String`	`toString()`
`void`	`validate()` Validates the configs specified by this instance.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - CrawlConfig
```
public CrawlConfig()
```
- Method Detail
  - setDnsResolver
```
public void setDnsResolver(org.apache.http.conn.DnsResolver dnsResolver)
```
    DNS resolver to use, #SystemDefaultDnsResolver() is default.
  - getDnsResolver
```
public org.apache.http.conn.DnsResolver getDnsResolver()
```
  - validate
```
public void validate()
              throws Exception
```
    Validates the configs specified by this instance.
    
    Throws:
    
    Exception - on Validation fail
  - getCrawlStorageFolder
```
public String getCrawlStorageFolder()
```
  - setCrawlStorageFolder
```
public void setCrawlStorageFolder(String crawlStorageFolder)
```
    The folder which will be used by crawler for storing the intermediate crawl data. The content of this folder should not be modified manually.
    
    Parameters:
    
    crawlStorageFolder - The folder for the crawler's storage
  - isResumableCrawling
```
public boolean isResumableCrawling()
```
  - setResumableCrawling
```
public void setResumableCrawling(boolean resumableCrawling)
```
    If this feature is enabled, you would be able to resume a previously stopped/crashed crawl. However, it makes crawling slightly slower
    
    Parameters:
    
    resumableCrawling - Should crawling be resumable between runs ?
  - setDbLockTimeout
```
public void setDbLockTimeout(long dbLockTimeout)
```
    Set the lock timeout for the underlying sleepycat DB, in milliseconds. Default is 500.
    
    Parameters:
    
    dbLockTimeout -
    
    See Also:
    
    EnvironmentConfig.setLockTimeout(long, java.util.concurrent.TimeUnit)
  - getDbLockTimeout
```
public long getDbLockTimeout()
```
  - getMaxDepthOfCrawling
```
public int getMaxDepthOfCrawling()
```
  - setMaxDepthOfCrawling
```
public void setMaxDepthOfCrawling(int maxDepthOfCrawling)
```
    Maximum depth of crawling For unlimited depth this parameter should be set to -1
    
    Parameters:
    
    maxDepthOfCrawling - Depth of crawling (all links on current page = depth of 1)
  - getMaxPagesToFetch
```
public int getMaxPagesToFetch()
```
  - setMaxPagesToFetch
```
public void setMaxPagesToFetch(int maxPagesToFetch)
```
    Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
    
    Parameters:
    
    maxPagesToFetch - How many pages to fetch from all threads together ?
  - getUserAgentString
```
public String getUserAgentString()
```
    Returns:
    
    userAgentString
  - setUserAgentString
```
public void setUserAgentString(String userAgentString)
```
    user-agent string that is used for representing your crawler to web servers. See http://en.wikipedia.org/wiki/User_agent for more details
    
    Parameters:
    
    userAgentString - Custom userAgent string to use as your crawler's identifier
  - getDefaultHeaders
```
public Collection<org.apache.http.message.BasicHeader> getDefaultHeaders()
```
    Return a copy of the default header collection.
  - setDefaultHeaders
```
public void setDefaultHeaders(Collection<? extends org.apache.http.Header> defaultHeaders)
```
    Set the default header collection (creating copies of the provided headers).
  - getPolitenessDelay
```
public int getPolitenessDelay()
```
  - setPolitenessDelay
```
public void setPolitenessDelay(int politenessDelay)
```
    Politeness delay in milliseconds (delay between sending two requests to the same host).
    
    Parameters:
    
    politenessDelay - the delay in milliseconds.
  - isIncludeHttpsPages
```
public boolean isIncludeHttpsPages()
```
  - setIncludeHttpsPages
```
public void setIncludeHttpsPages(boolean includeHttpsPages)
```
    Parameters:
    
    includeHttpsPages - Should we crawl https pages?
  - isIncludeBinaryContentInCrawling
```
public boolean isIncludeBinaryContentInCrawling()
```
  - setIncludeBinaryContentInCrawling
```
public void setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)
```
    Parameters:
    
    includeBinaryContentInCrawling - Should we fetch binary content such as images, audio, ...?
  - isProcessBinaryContentInCrawling
```
public boolean isProcessBinaryContentInCrawling()
```
  - setProcessBinaryContentInCrawling
```
public void setProcessBinaryContentInCrawling(boolean processBinaryContentInCrawling)
```
    Should we process binary content such as images, audio, ... using TIKA?
  - getMaxConnectionsPerHost
```
public int getMaxConnectionsPerHost()
```
  - setMaxConnectionsPerHost
```
public void setMaxConnectionsPerHost(int maxConnectionsPerHost)
```
    Parameters:
    
    maxConnectionsPerHost - Maximum Connections per host
  - getMaxTotalConnections
```
public int getMaxTotalConnections()
```
  - setMaxTotalConnections
```
public void setMaxTotalConnections(int maxTotalConnections)
```
    Parameters:
    
    maxTotalConnections - Maximum total connections
  - getSocketTimeout
```
public int getSocketTimeout()
```
  - setSocketTimeout
```
public void setSocketTimeout(int socketTimeout)
```
    Parameters:
    
    socketTimeout - Socket timeout in milliseconds
  - getConnectionTimeout
```
public int getConnectionTimeout()
```
  - setConnectionTimeout
```
public void setConnectionTimeout(int connectionTimeout)
```
    Parameters:
    
    connectionTimeout - Connection timeout in milliseconds
  - getMaxOutgoingLinksToFollow
```
public int getMaxOutgoingLinksToFollow()
```
  - setMaxOutgoingLinksToFollow
```
public void setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)
```
    Parameters:
    
    maxOutgoingLinksToFollow - Max number of outgoing links which are processed from a page
  - getMaxDownloadSize
```
public int getMaxDownloadSize()
```
  - setMaxDownloadSize
```
public void setMaxDownloadSize(int maxDownloadSize)
```
    Parameters:
    
    maxDownloadSize - Max allowed size of a page. Pages larger than this size will not be fetched.
  - isFollowRedirects
```
public boolean isFollowRedirects()
```
  - setFollowRedirects
```
public void setFollowRedirects(boolean followRedirects)
```
    Parameters:
    
    followRedirects - Should we follow redirects?
  - isShutdownOnEmptyQueue
```
public boolean isShutdownOnEmptyQueue()
```
  - setShutdownOnEmptyQueue
```
public void setShutdownOnEmptyQueue(boolean shutdown)
```
    Should the crawler stop running when the queue is empty?
  - isOnlineTldListUpdate
```
public boolean isOnlineTldListUpdate()
```
  - setOnlineTldListUpdate
```
public void setOnlineTldListUpdate(boolean online)
```
    Should the TLD list be updated automatically on each run? Alternatively, it can be loaded from the embedded tld-names.txt resource file that was obtained from https://publicsuffix.org/list/effective_tld_names.dat
  - getProxyHost
```
public String getProxyHost()
```
  - setProxyHost
```
public void setProxyHost(String proxyHost)
```
    Parameters:
    
    proxyHost - If crawler should run behind a proxy, this parameter can be used for specifying the proxy host.
  - getProxyPort
```
public int getProxyPort()
```
  - setProxyPort
```
public void setProxyPort(int proxyPort)
```
    Parameters:
    
    proxyPort - If crawler should run behind a proxy, this parameter can be used for specifying the proxy port.
  - getProxyUsername
```
public String getProxyUsername()
```
  - setProxyUsername
```
public void setProxyUsername(String proxyUsername)
```
    Parameters:
    
    proxyUsername - If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the username.
  - getProxyPassword
```
public String getProxyPassword()
```
  - setProxyPassword
```
public void setProxyPassword(String proxyPassword)
```
    If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
    
    Parameters:
    
    proxyPassword - String Password
  - getAuthInfos
```
public List<AuthInfo> getAuthInfos()
```
    Returns:
    
    the authentications Information
  - addAuthInfo
```
public void addAuthInfo(AuthInfo authInfo)
```
  - setAuthInfos
```
public void setAuthInfos(List<AuthInfo> authInfos)
```
    Parameters:
    
    authInfos - authenticationInformations to set
  - getThreadMonitoringDelaySeconds
```
public int getThreadMonitoringDelaySeconds()
```
  - setThreadMonitoringDelaySeconds
```
public void setThreadMonitoringDelaySeconds(int delay)
```
  - getThreadShutdownDelaySeconds
```
public int getThreadShutdownDelaySeconds()
```
  - setThreadShutdownDelaySeconds
```
public void setThreadShutdownDelaySeconds(int delay)
```
  - getCleanupDelaySeconds
```
public int getCleanupDelaySeconds()
```
  - setCleanupDelaySeconds
```
public void setCleanupDelaySeconds(int delay)
```
  - getCookiePolicy
```
public String getCookiePolicy()
```
  - setCookiePolicy
```
public void setCookiePolicy(String cookiePolicy)
```
  - getCookieStore
```
public org.apache.http.client.CookieStore getCookieStore()
```
    Gets the configured CookieStore or null if none is set
    
    Returns:
    
    the CookieStore
  - setCookieStore
```
public void setCookieStore(org.apache.http.client.CookieStore cookieStore)
```
    Sets the to be used
    
    Parameters:
    
    cookieStore - the CookieStore
  - isRespectNoFollow
```
public boolean isRespectNoFollow()
```
    Gets the current CookieStore used
    
    Returns:
    
    the CookieStore
  - setRespectNoFollow
```
public void setRespectNoFollow(boolean respectNoFollow)
```
  - isRespectNoIndex
```
public boolean isRespectNoIndex()
```
  - setRespectNoIndex
```
public void setRespectNoIndex(boolean respectNoIndex)
```
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class CrawlConfig

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

CrawlConfig

Method Detail

setDnsResolver

getDnsResolver

validate

getCrawlStorageFolder

setCrawlStorageFolder

isResumableCrawling

setResumableCrawling

setDbLockTimeout

getDbLockTimeout

getMaxDepthOfCrawling

setMaxDepthOfCrawling

getMaxPagesToFetch

setMaxPagesToFetch

getUserAgentString

setUserAgentString

getDefaultHeaders

setDefaultHeaders

getPolitenessDelay

setPolitenessDelay

isIncludeHttpsPages

setIncludeHttpsPages

isIncludeBinaryContentInCrawling

setIncludeBinaryContentInCrawling

isProcessBinaryContentInCrawling

setProcessBinaryContentInCrawling

getMaxConnectionsPerHost

setMaxConnectionsPerHost

getMaxTotalConnections

setMaxTotalConnections

getSocketTimeout

setSocketTimeout

getConnectionTimeout

setConnectionTimeout

getMaxOutgoingLinksToFollow

setMaxOutgoingLinksToFollow

getMaxDownloadSize

setMaxDownloadSize

isFollowRedirects

setFollowRedirects

isShutdownOnEmptyQueue

setShutdownOnEmptyQueue

isOnlineTldListUpdate

setOnlineTldListUpdate

getProxyHost

setProxyHost

getProxyPort

setProxyPort

getProxyUsername

setProxyUsername

getProxyPassword

setProxyPassword

getAuthInfos

addAuthInfo

setAuthInfos

getThreadMonitoringDelaySeconds

setThreadMonitoringDelaySeconds

getThreadShutdownDelaySeconds

setThreadShutdownDelaySeconds

getCleanupDelaySeconds

setCleanupDelaySeconds

getCookiePolicy

setCookiePolicy

getCookieStore

setCookieStore

isRespectNoFollow

setRespectNoFollow

isRespectNoIndex

setRespectNoIndex

toString