org.dice_research.squirrel.components

Class WorkerComponent

java.lang.Object

org.hobbit.core.components.AbstractComponent

org.dice_research.squirrel.components.WorkerComponent

All Implemented Interfaces:

Closeable, AutoCloseable, org.dice_research.squirrel.frontier.Frontier, org.hobbit.core.components.Component

@Component
 @Qualifier(value="workerComponent")
public class WorkerComponent
extends org.hobbit.core.components.AbstractComponent
implements org.dice_research.squirrel.frontier.Frontier

Field Summary

Fields

Modifier and Type	Field and Description
private org.hobbit.core.rabbit.RabbitRpcClient	client
private org.hobbit.core.rabbit.RabbitRpcClient	clientFrontier
private boolean	deduplicationActive Indicates whether deduplication is active.
private static org.slf4j.Logger	LOGGER
static String	OUTPUT_FOLDER_KEY
private org.hobbit.core.rabbit.DataSender	sender
private org.hobbit.core.rabbit.DataSender	senderDeduplicator
private org.hobbit.core.rabbit.DataSender	senderFrontier
private org.dice_research.squirrel.data.uri.serialize.Serializer	serializer
private Timer	timerAliveMessages
private byte[]	uriSetRequest
private org.dice_research.squirrel.worker.Worker	worker

Fields inherited from class org.hobbit.core.components.AbstractComponent

connectionFactory, incomingDataQueueFactory, NUMBER_OF_RETRIES_TO_CONNECT_TO_RABBIT_MQ, outgoingDataQueuefactory, rabbitMQHostName, START_WAITING_TIME_BEFORE_RETRY

Constructor Summary

Constructors

Constructor and Description
WorkerComponent()

Method Summary

Modifier and Type	Method and Description
void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)
void	addNewUris(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)
void	close()
void	crawlingDone(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)
boolean	doesRecrawling()
List<org.dice_research.squirrel.data.uri.CrawleableUri>	getNextUris()
int	getNumberOfPendingUris()
void	init()
void	run()
void	setWorker(org.dice_research.squirrel.worker.Worker worker)

Methods inherited from class org.hobbit.core.components.AbstractComponent

createConnection, generateSessionQueueName, getHobbitSessionId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

OUTPUT_FOLDER_KEY

public static final String OUTPUT_FOLDER_KEY

See Also:

Constant Field Values

deduplicationActive

private boolean deduplicationActive

Indicates whether deduplication is active. If it is not active, this component will send data to the deduplicator.

worker

@Qualifier(value="workerBean")
 @Autowired
private org.dice_research.squirrel.worker.Worker worker

senderFrontier

private org.hobbit.core.rabbit.DataSender senderFrontier

senderDeduplicator

private org.hobbit.core.rabbit.DataSender senderDeduplicator

clientFrontier

private org.hobbit.core.rabbit.RabbitRpcClient clientFrontier

sender

@Qualifier(value="sender")
 @Autowired
private org.hobbit.core.rabbit.DataSender sender

client

@Qualifier(value="client")
 @Autowired
private org.hobbit.core.rabbit.RabbitRpcClient client

uriSetRequest

private byte[] uriSetRequest

serializer

@Qualifier(value="serializerBean")
 @Autowired
private org.dice_research.squirrel.data.uri.serialize.Serializer serializer

timerAliveMessages

private Timer timerAliveMessages

Constructor Detail

WorkerComponent

public WorkerComponent()

Method Detail

init

public void init()
          throws Exception

Specified by:

init in interface org.hobbit.core.components.Component

Overrides:

init in class org.hobbit.core.components.AbstractComponent

Throws:

Exception

run

public void run()

Specified by:

run in interface org.hobbit.core.components.Component

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.hobbit.core.components.AbstractComponent

Throws:

IOException

getNextUris

public List<org.dice_research.squirrel.data.uri.CrawleableUri> getNextUris()

Specified by:

getNextUris in interface org.dice_research.squirrel.frontier.Frontier

setWorker

public void setWorker(org.dice_research.squirrel.worker.Worker worker)

addNewUri

public void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

addNewUri in interface org.dice_research.squirrel.frontier.Frontier

addNewUris

public void addNewUris(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)

Specified by:

addNewUris in interface org.dice_research.squirrel.frontier.Frontier

crawlingDone

public void crawlingDone(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)

Specified by:

crawlingDone in interface org.dice_research.squirrel.frontier.Frontier

getNumberOfPendingUris

public int getNumberOfPendingUris()

Specified by:

getNumberOfPendingUris in interface org.dice_research.squirrel.frontier.Frontier

doesRecrawling

public boolean doesRecrawling()

Specified by:

doesRecrawling in interface org.dice_research.squirrel.frontier.Frontier