org.dice_research.squirrel.robots

Class RobotsManagerImpl

java.lang.Object

org.dice_research.squirrel.robots.RobotsManagerImpl

All Implemented Interfaces:

RobotsManager

public class RobotsManagerImpl
extends Object
implements RobotsManager

Field Summary

Fields

Modifier and Type	Field and Description
private static long	DEFAULT_MIN_WAITING_TIME
private long	defaultMinWaitingTime
private crawlercommons.fetcher.http.BaseHttpFetcher	fetcher
private InetAddress	lastIpAddress
private static org.slf4j.Logger	LOGGER
private crawlercommons.robots.BaseRobotsParser	parser
private crawlercommons.robots.BaseRobotRules	robotRules
private static String	ROBOTS_FILE_NAME

Constructor Summary

Constructors

Constructor and Description
RobotsManagerImpl(crawlercommons.fetcher.http.BaseHttpFetcher fetcher)
RobotsManagerImpl(crawlercommons.fetcher.http.BaseHttpFetcher fetcher, crawlercommons.robots.BaseRobotsParser parser)

Method Summary

Modifier and Type	Method and Description
long	getMinWaitingTime(org.dice_research.squirrel.data.uri.CrawleableUri curi) Returns the minimum time a crawler should wait before sending a new request to the given domain.
protected crawlercommons.robots.BaseRobotRules	getRules(org.dice_research.squirrel.data.uri.CrawleableUri curi)
boolean	isUriCrawlable(org.dice_research.squirrel.data.uri.CrawleableUri curi) Returns true, if the robots.txt file does not forbid the crawling of that URI.
void	setDefaultMinWaitingTime(long defaultMinWaitingTime)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

ROBOTS_FILE_NAME

private static final String ROBOTS_FILE_NAME

See Also:

Constant Field Values

DEFAULT_MIN_WAITING_TIME

private static final long DEFAULT_MIN_WAITING_TIME

See Also:

Constant Field Values

defaultMinWaitingTime

private long defaultMinWaitingTime

fetcher

private crawlercommons.fetcher.http.BaseHttpFetcher fetcher

parser

private crawlercommons.robots.BaseRobotsParser parser

lastIpAddress

private InetAddress lastIpAddress

robotRules

private crawlercommons.robots.BaseRobotRules robotRules

Constructor Detail

RobotsManagerImpl

public RobotsManagerImpl(crawlercommons.fetcher.http.BaseHttpFetcher fetcher)

RobotsManagerImpl

public RobotsManagerImpl(crawlercommons.fetcher.http.BaseHttpFetcher fetcher,
                         crawlercommons.robots.BaseRobotsParser parser)

Method Detail

getRules

protected crawlercommons.robots.BaseRobotRules getRules(org.dice_research.squirrel.data.uri.CrawleableUri curi)

isUriCrawlable

public boolean isUriCrawlable(org.dice_research.squirrel.data.uri.CrawleableUri curi)

Description copied from interface: RobotsManager

Returns true, if the robots.txt file does not forbid the crawling of that URI.

Specified by:

isUriCrawlable in interface RobotsManager

Parameters:

curi - the URI that should be crawled

Returns:

false, if the robots.txt forbids the crawling of this URI, else true.

getMinWaitingTime

public long getMinWaitingTime(org.dice_research.squirrel.data.uri.CrawleableUri curi)

Description copied from interface: RobotsManager

Returns the minimum time a crawler should wait before sending a new request to the given domain. If the robots.txt file can not be retrieved or does not define a minimum waiting time, 0 is returned.

Specified by:

getMinWaitingTime in interface RobotsManager

Parameters:

curi - a URI containing the domain to which two or more requests should be send.

Returns:

the minimum time to wait in milliseconds.

setDefaultMinWaitingTime

public void setDefaultMinWaitingTime(long defaultMinWaitingTime)