org.dice_research.squirrel.collect

Class SqlBasedUriCollector

java.lang.Object

org.dice_research.squirrel.collect.SqlBasedUriCollector

All Implemented Interfaces:

Closeable, AutoCloseable, UriCollector, org.dice_research.squirrel.sink.SinkBase

@NotThreadSafe
public class SqlBasedUriCollector
extends Object
implements UriCollector, Closeable

An implementation of the UriCollector interface that is backed by a SQL database.

Author:

Geralod Souza Junior (gsjunior@mail.uni-paderborn.de), Michael Röder (michael.roeder@uni-paderborn.de) NotThreadSafe because the prepared statement objects used internally are not stateless.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SqlBasedUriCollector.UriTableStatus

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bufferSize
protected static String	COUNT_URIS_QUERY
protected static String	CREATE_TABLE_QUERY
protected Connection	dbConnection
private static int	DEFAULT_BUFFER_SIZE
protected static String	DROP_TABLE_QUERY
protected static String	INSERT_URI_QUERY_PART_1
protected static String	INSERT_URI_QUERY_PART_2
protected Map<String,SqlBasedUriCollector.UriTableStatus>	knownUris
private static org.slf4j.Logger	LOGGER
private static int	MAX_ALPHANUM_PART_OF_TABLE_NAME
private static String	SELECT_TABLE_QUERY
protected org.dice_research.squirrel.data.uri.serialize.Serializer	serializer
private static Pattern	TABLE_NAME_GENERATE_REGEX
private static String	TABLE_NAME_KEY
private long	total_uris

Constructor Summary

Constructors

Constructor and Description
SqlBasedUriCollector(org.dice_research.squirrel.data.uri.serialize.Serializer serializer, String dbPath)

Method Summary

Modifier and Type	Method and Description
void	addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.dice_research.squirrel.data.uri.CrawleableUri newUri) Adds the given new URI to the list of URIs collected for the given URI.
void	addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.graph.Triple triple) Adds the given triple to the list of URIs collected from the given URI.
protected void	addUri(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.graph.Node node)
void	close()
void	closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)
void	create(String dbPath)
protected static String	generateTableName(String uri) Generates a table name based on the given URI.
long	getSize()
long	getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns the total of uris that have been collected
protected static String	getTableName(org.dice_research.squirrel.data.uri.CrawleableUri uri) Retrieves the URIs table name from its properties or generates a new table name and adds it to the URI (using the "URI_COLLECTOR_TABLE_NAME" property).
Iterator<byte[]>	getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri) Returns a list of serialized CrawleableUri instances that have been collected for the given URI.
void	openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.collect.UriCollector

addNewUri, addNewUri

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

COUNT_URIS_QUERY

protected static final String COUNT_URIS_QUERY

See Also:

Constant Field Values

CREATE_TABLE_QUERY

protected static final String CREATE_TABLE_QUERY

See Also:

Constant Field Values

DROP_TABLE_QUERY

protected static final String DROP_TABLE_QUERY

See Also:

Constant Field Values

INSERT_URI_QUERY_PART_1

protected static final String INSERT_URI_QUERY_PART_1

See Also:

Constant Field Values

INSERT_URI_QUERY_PART_2

protected static final String INSERT_URI_QUERY_PART_2

See Also:

Constant Field Values

SELECT_TABLE_QUERY

private static final String SELECT_TABLE_QUERY

See Also:

Constant Field Values

TABLE_NAME_KEY

private static final String TABLE_NAME_KEY

See Also:

Constant Field Values

MAX_ALPHANUM_PART_OF_TABLE_NAME

private static final int MAX_ALPHANUM_PART_OF_TABLE_NAME

See Also:

Constant Field Values

DEFAULT_BUFFER_SIZE

private static final int DEFAULT_BUFFER_SIZE

See Also:

Constant Field Values

TABLE_NAME_GENERATE_REGEX

private static final Pattern TABLE_NAME_GENERATE_REGEX

total_uris

private long total_uris

dbConnection

protected Connection dbConnection

serializer

protected org.dice_research.squirrel.data.uri.serialize.Serializer serializer

bufferSize

protected int bufferSize

knownUris

protected Map<String,SqlBasedUriCollector.UriTableStatus> knownUris

Constructor Detail

SqlBasedUriCollector

public SqlBasedUriCollector(org.dice_research.squirrel.data.uri.serialize.Serializer serializer,
                            String dbPath)
                     throws SQLException

Throws:

SQLException

Method Detail

create

public void create(String dbPath)

openSinkForUri

public void openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

openSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

getUris

public Iterator<byte[]> getUris(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Description copied from interface: UriCollector

Returns a list of serialized CrawleableUri instances that have been collected for the given URI.

Specified by:

getUris in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

An Iterator that iterates over the already serialized URIs that have been collected for the given URI.

addTriple

public void addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                      org.apache.jena.graph.Triple triple)

Description copied from interface: UriCollector

Adds the given triple to the list of URIs collected from the given URI. Note that it is suggested to use the UriCollector.addNewUri(CrawleableUri, CrawleableUri) method instead since this enables the addition of meta data to the collected URI.

Specified by:

addTriple in interface UriCollector

Parameters:

uri - The URI from which the given triple has been collected.

triple - The triple that has been collected.

addUri

protected void addUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                      org.apache.jena.graph.Node node)

addNewUri

public void addNewUri(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                      org.dice_research.squirrel.data.uri.CrawleableUri newUri)

Description copied from interface: UriCollector

Adds the given new URI to the list of URIs collected for the given URI.

Specified by:

addNewUri in interface UriCollector

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

closeSinkForUri

public void closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

closeSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

getSize

public long getSize()

getSize

public long getSize(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Description copied from interface: UriCollector

Returns the total of uris that have been collected

Specified by:

getSize in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

A long number of total uris collected

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getTableName

protected static String getTableName(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Retrieves the URIs table name from its properties or generates a new table name and adds it to the URI (using the "URI_COLLECTOR_TABLE_NAME" property).

Parameters:

uri - the URI for which a table name is needed.

Returns:

the table name of the URI

generateTableName

protected static String generateTableName(String uri)

Generates a table name based on the given URI. Only alphanumeric characters of the URI are kept. If the URI exceeds the length of MAX_ALPHANUM_PART_OF_TABLE_NAME=30 the exceeding part is cut off. After that the hash value of the original URI is appended.

Parameters:

uri - the URI for which a table name has to be generated

Returns:

the table name of the URI