org.dice_research.squirrel.collect

Interface UriCollector

All Superinterfaces:

org.dice_research.squirrel.sink.SinkBase

All Known Implementing Classes:

SimpleUriCollector, SqlBasedUriCollector

public interface UriCollector
extends org.dice_research.squirrel.sink.SinkBase

A URI collector stores the URIs that have been found by a worker while crawling/processing a certain URI. After the crawling, the URI collector can be asked for these URIs using the getUris(CrawleableUri) method.

Author:

Michael Röder (michael.roeder@uni-paderborn.de)

Method Summary

Modifier and Type	Method and Description
void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.dice_research.squirrel.data.uri.CrawleableUri newUri) Adds the given new URI to the list of URIs collected for the given URI.
default void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.graph.Node newUri) Adds the given new URI to the list of URIs collected for the given URI.
default void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, String newUri) Adds the given new URI to the list of URIs collected for the given URI.
default void	addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.graph.Triple triple) Adds the given triple to the list of URIs collected from the given URI.
long	getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns the total of uris that have been collected
Iterator<byte[]>	getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns a list of serialized CrawleableUri instances that have been collected for the given URI.

Methods inherited from interface org.dice_research.squirrel.sink.SinkBase

closeSinkForUri, openSinkForUri

Method Detail

addTriple

default void addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                       org.apache.jena.graph.Triple triple)

Adds the given triple to the list of URIs collected from the given URI. Note that it is suggested to use the addNewUri(CrawleableUri, CrawleableUri) method instead since this enables the addition of meta data to the collected URI.

Parameters:

uri - The URI from which the given triple has been collected.

triple - The triple that has been collected.

addNewUri

void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
               org.dice_research.squirrel.data.uri.CrawleableUri newUri)

Adds the given new URI to the list of URIs collected for the given URI.

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

addNewUri

default void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                       org.apache.jena.graph.Node newUri)

Adds the given new URI to the list of URIs collected for the given URI.

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

addNewUri

default void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                       String newUri)

Adds the given new URI to the list of URIs collected for the given URI.

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

getUris

Iterator<byte[]> getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Returns a list of serialized CrawleableUri instances that have been collected for the given URI.

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

An Iterator that iterates over the already serialized URIs that have been collected for the given URI.

getSize

long getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Returns the total of uris that have been collected

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

A long number of total uris collected