com.github.vector4wang

类 VWCrawler

java.lang.Object

com.github.vector4wang.VWCrawler

public class VWCrawler
extends Object

vw爬虫的简单配置中心

作者:

vector

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	VWCrawler.Builder

构造器概要

构造器

构造器和说明
VWCrawler()

方法概要

限定符和类型	方法和说明
void	addWaitCrawlerUrl(String href) 将url保存在目标集合中
String	generateUrl()
Set<String>	getCrawledUrls()
CrawlerService	getCrawlerService()
AbstractDownloader	getDownloader()
Map<String,String>	getHeaderMap()
AbstractProxyExtractor	getProxyExtractor()
int	getRetryCount()
Set<String>	getSeedsPageUrlRex()
Set<String>	getTargetUrlRex()
int	getThreadCount()
int	getTimeout()
String	getUrl()
LinkedBlockingQueue<String>	getWaitCrawlerUrls()
boolean	isTargetUrl(String url)
void	setProxyExtractor(AbstractProxyExtractor proxyExtractor)
void	start()
void	stop() 停止爬虫
void	tryStop() 尝试停止(正常情况的停止)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

VWCrawler

public VWCrawler()

方法详细资料

getUrl

public String getUrl()

getTimeout

public int getTimeout()

getRetryCount

public int getRetryCount()

getThreadCount

public int getThreadCount()

getWaitCrawlerUrls

public LinkedBlockingQueue<String> getWaitCrawlerUrls()

getCrawledUrls

public Set<String> getCrawledUrls()

getTargetUrlRex

public Set<String> getTargetUrlRex()

getSeedsPageUrlRex

public Set<String> getSeedsPageUrlRex()

getCrawlerService

public CrawlerService getCrawlerService()

getProxyExtractor

public AbstractProxyExtractor getProxyExtractor()

setProxyExtractor

public void setProxyExtractor(AbstractProxyExtractor proxyExtractor)

getHeaderMap

public Map<String,String> getHeaderMap()

getDownloader

public AbstractDownloader getDownloader()

start

public void start()

tryStop

public void tryStop()

尝试停止(正常情况的停止)

isTargetUrl

public boolean isTargetUrl(String url)

generateUrl

public String generateUrl()
                   throws InterruptedException

返回:

取出一个待抓取的url

抛出:

InterruptedException - 线程中断异常

addWaitCrawlerUrl

public void addWaitCrawlerUrl(String href)

将url保存在目标集合中

参数:

href - 即将添加到待抓取列表的url

stop

public void stop()

停止爬虫