org.dice_research.squirrel.data.uri.filter

Class MongoDBKnowUriFilter

java.lang.Object

org.dice_research.squirrel.data.uri.filter.MongoDBKnowUriFilter

All Implemented Interfaces:

Closeable, AutoCloseable, Cloneable, org.dice_research.squirrel.data.uri.filter.KnownUriFilter, org.dice_research.squirrel.data.uri.filter.UriFilter, org.dice_research.squirrel.deduplication.hashing.UriHashCustodian

public class MongoDBKnowUriFilter
extends Object
implements org.dice_research.squirrel.data.uri.filter.KnownUriFilter, Cloneable, Closeable, org.dice_research.squirrel.deduplication.hashing.UriHashCustodian

Filter implementation for use with MongoDB * @author Geraldo Souza Junior (gsjunior@mail.uni-paderborn.de)

Field Summary

Fields

Modifier and Type	Field and Description
private com.mongodb.MongoClient	client
static String	COLLECTION_NAME
static String	COLUMN_CRAWLING_IN_PROCESS
static String	COLUMN_HASH_VALUE
static String	COLUMN_IP
static String	COLUMN_TIMESTAMP_LAST_CRAWL
static String	COLUMN_TIMESTAMP_NEXT_CRAWL
static String	COLUMN_TYPE
static String	COLUMN_URI
static String	DB_NAME
private static String	DUMMY_HASH_VALUE Used as a default hash value for URIS, will be replaced by real hash value as soon as it has been computed.
private static org.slf4j.Logger	LOGGER
private com.mongodb.client.MongoDatabase	mongoDB
private static boolean	PERSIST
private Integer	recrawlEveryWeek

Constructor Summary

Constructors

Constructor and Description
MongoDBKnowUriFilter(String hostName, Integer port)

Method Summary

Modifier and Type	Method and Description
void	add(org.dice_research.squirrel.data.uri.CrawleableUri uri, long nextCrawlTimestamp)
void	add(org.dice_research.squirrel.data.uri.CrawleableUri uri, long lastCrawlTimestamp, long nextCrawlTimestamp)
void	addHashValuesForUris(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)
void	close()
long	count()
org.bson.Document	crawleableUriToMongoDocument(org.dice_research.squirrel.data.uri.CrawleableUri uri)
List<org.dice_research.squirrel.data.uri.CrawleableUri>	getOutdatedUris()
Set<org.dice_research.squirrel.data.uri.CrawleableUri>	getUrisWithSameHashValues(Set<org.dice_research.squirrel.deduplication.hashing.HashValue> hashValuesForComparison)
boolean	isUriGood(org.dice_research.squirrel.data.uri.CrawleableUri uri)
boolean	knowUriTableExists()
void	open()
void	purge()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

add

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

client

private com.mongodb.MongoClient client

mongoDB

private com.mongodb.client.MongoDatabase mongoDB

DB_NAME

public static final String DB_NAME

See Also:

Constant Field Values

recrawlEveryWeek

private Integer recrawlEveryWeek

COLLECTION_NAME

public static final String COLLECTION_NAME

See Also:

Constant Field Values

COLUMN_TIMESTAMP_LAST_CRAWL

public static final String COLUMN_TIMESTAMP_LAST_CRAWL

See Also:

Constant Field Values

COLUMN_URI

public static final String COLUMN_URI

See Also:

Constant Field Values

COLUMN_CRAWLING_IN_PROCESS

public static final String COLUMN_CRAWLING_IN_PROCESS

See Also:

Constant Field Values

COLUMN_TIMESTAMP_NEXT_CRAWL

public static final String COLUMN_TIMESTAMP_NEXT_CRAWL

See Also:

Constant Field Values

COLUMN_IP

public static final String COLUMN_IP

See Also:

Constant Field Values

COLUMN_TYPE

public static final String COLUMN_TYPE

See Also:

Constant Field Values

COLUMN_HASH_VALUE

public static final String COLUMN_HASH_VALUE

See Also:

Constant Field Values

PERSIST

private static final boolean PERSIST

DUMMY_HASH_VALUE

private static final String DUMMY_HASH_VALUE

Used as a default hash value for URIS, will be replaced by real hash value as soon as it has been computed.

See Also:

Constant Field Values

Constructor Detail

MongoDBKnowUriFilter

public MongoDBKnowUriFilter(String hostName,
                            Integer port)

Method Detail

isUriGood

public boolean isUriGood(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

isUriGood in interface org.dice_research.squirrel.data.uri.filter.UriFilter

add

public void add(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                long nextCrawlTimestamp)

Specified by:

add in interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

crawleableUriToMongoDocument

public org.bson.Document crawleableUriToMongoDocument(org.dice_research.squirrel.data.uri.CrawleableUri uri)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

open

public void open()

Specified by:

open in interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

knowUriTableExists

public boolean knowUriTableExists()

add

public void add(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                long lastCrawlTimestamp,
                long nextCrawlTimestamp)

Specified by:

add in interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

addHashValuesForUris

public void addHashValuesForUris(List<org.dice_research.squirrel.data.uri.CrawleableUri> uris)

Specified by:

addHashValuesForUris in interface org.dice_research.squirrel.deduplication.hashing.UriHashCustodian

purge

public void purge()

getOutdatedUris

public List<org.dice_research.squirrel.data.uri.CrawleableUri> getOutdatedUris()

Specified by:

getOutdatedUris in interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

count

public long count()

Specified by:

count in interface org.dice_research.squirrel.data.uri.filter.KnownUriFilter

getUrisWithSameHashValues

public Set<org.dice_research.squirrel.data.uri.CrawleableUri> getUrisWithSameHashValues(Set<org.dice_research.squirrel.deduplication.hashing.HashValue> hashValuesForComparison)

Specified by:

getUrisWithSameHashValues in interface org.dice_research.squirrel.deduplication.hashing.UriHashCustodian