static WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.builder() |
|
default WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.crawlerLimits(Consumer<WebCrawlerLimits.Builder> crawlerLimits) |
The configuration of crawl limits for the web URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.crawlerLimits(WebCrawlerLimits crawlerLimits) |
The configuration of crawl limits for the web URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.exclusionFilters(String... exclusionFilters) |
A list of one or more exclusion regular expression patterns to exclude certain URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.exclusionFilters(Collection<String> exclusionFilters) |
A list of one or more exclusion regular expression patterns to exclude certain URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.inclusionFilters(String... inclusionFilters) |
A list of one or more inclusion regular expression patterns to include certain URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.inclusionFilters(Collection<String> inclusionFilters) |
A list of one or more inclusion regular expression patterns to include certain URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.scope(String scope) |
The scope of what is crawled for your URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.scope(WebScopeType scope) |
The scope of what is crawled for your URLs.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.toBuilder() |
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.userAgent(String userAgent) |
Returns the user agent suffix for your web crawler.
|
WebCrawlerConfiguration.Builder |
WebCrawlerConfiguration.Builder.userAgentHeader(String userAgentHeader) |
A string used for identifying the crawler or bot when it accesses a web server.
|