org.dice_research.squirrel.collect

Class SimpleUriCollector

java.lang.Object

org.dice_research.squirrel.collect.SimpleUriCollector

All Implemented Interfaces:

UriCollector, org.dice_research.squirrel.sink.SinkBase

public class SimpleUriCollector
extends Object
implements UriCollector

Simple in-memory implementation of the UriCollector interface based on the given Serializer.

Author:

Michael Röder (michael.roeder@uni-paderborn.de)

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOGGER
protected org.dice_research.squirrel.data.uri.serialize.Serializer	serializer Serializer used to serialize the given URIs.
private long	total_uris Number of all URIs collected.
protected Map<String,Map<String,byte[]>>	urisOfUris Mapping from URIs to the new URIs that have been found.

Constructor Summary

Constructors

Constructor and Description
SimpleUriCollector(org.dice_research.squirrel.data.uri.serialize.Serializer serializer) Constructor.

Method Summary

Modifier and Type	Method and Description
void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.dice_research.squirrel.data.uri.CrawleableUri newUri) Adds the given new URI to the list of URIs collected for the given URI.
void	closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)
long	getSize() Returns the total number of new URIs that have been added to this collector.
long	getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns the total of uris that have been collected
Iterator<byte[]>	getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns a list of serialized CrawleableUri instances that have been collected for the given URI.
void	openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.collect.UriCollector

addNewUri, addNewUri, addTriple

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

total_uris

private long total_uris

Number of all URIs collected.

urisOfUris

protected Map<String,Map<String,byte[]>> urisOfUris

Mapping from URIs to the new URIs that have been found.

serializer

protected org.dice_research.squirrel.data.uri.serialize.Serializer serializer

Serializer used to serialize the given URIs.

Constructor Detail

SimpleUriCollector

public SimpleUriCollector(org.dice_research.squirrel.data.uri.serialize.Serializer serializer)

Constructor.

Parameters:

serializer - the serializer that is used to serialize the new URIs.

Method Detail

openSinkForUri

public void openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

openSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

getUris

public Iterator<byte[]> getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Description copied from interface: UriCollector

Returns a list of serialized CrawleableUri instances that have been collected for the given URI.

Specified by:

getUris in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

An Iterator that iterates over the already serialized URIs that have been collected for the given URI.

addNewUri

public void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                      org.dice_research.squirrel.data.uri.CrawleableUri newUri)

Description copied from interface: UriCollector

Adds the given new URI to the list of URIs collected for the given URI.

Specified by:

addNewUri in interface UriCollector

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

getSize

public long getSize()

Returns the total number of new URIs that have been added to this collector.

Returns:

the total number of new URIs that have been added to this collector.

closeSinkForUri

public void closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

closeSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

getSize

public long getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Description copied from interface: UriCollector

Returns the total of uris that have been collected

Specified by:

getSize in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

A long number of total uris collected