org.dice_research.squirrel.robots

Interface RobotsManager

All Known Implementing Classes:

RobotsManagerImpl

public interface RobotsManager

Interface of a class that can be used to access the robots.txt files. This interface hides the whole retrieving and processing of the robots.txt files an provides only those methods, that are needed by a worker of a crawler.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Method Summary

Modifier and Type	Method and Description
long	getMinWaitingTime(org.dice_research.squirrel.data.uri.CrawleableUri curi) Returns the minimum time a crawler should wait before sending a new request to the given domain.
boolean	isUriCrawlable(org.dice_research.squirrel.data.uri.CrawleableUri curi) Returns true, if the robots.txt file does not forbid the crawling of that URI.

Method Detail

isUriCrawlable

boolean isUriCrawlable(org.dice_research.squirrel.data.uri.CrawleableUri curi)

Returns true, if the robots.txt file does not forbid the crawling of that URI.

Parameters:

curi - the URI that should be crawled

Returns:

false, if the robots.txt forbids the crawling of this URI, else true.

getMinWaitingTime

long getMinWaitingTime(org.dice_research.squirrel.data.uri.CrawleableUri curi)

Returns the minimum time a crawler should wait before sending a new request to the given domain. If the robots.txt file can not be retrieved or does not define a minimum waiting time, 0 is returned.

Parameters:

curi - a URI containing the domain to which two or more requests should be send.

Returns:

the minimum time to wait in milliseconds.