org.dice_research.squirrel.sink.impl.file

Class FileBasedSink

java.lang.Object

org.dice_research.squirrel.sink.impl.file.FileBasedSink

All Implemented Interfaces:

Closeable, AutoCloseable, org.dice_research.squirrel.sink.quadbased.QuadBasedSink, org.dice_research.squirrel.sink.Sink, org.dice_research.squirrel.sink.SinkBase, org.dice_research.squirrel.sink.triplebased.TripleBasedSink, org.dice_research.squirrel.sink.UnstructuredDataSink

Direct Known Subclasses:

HdtBasedSink

public class FileBasedSink
extends Object
implements org.dice_research.squirrel.sink.Sink

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	FileBasedSink.StreamStatus

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.jena.riot.Lang	DEFAULT_OUTPUT_LANG
private static org.slf4j.Logger	LOGGER
protected File	outputDirectory Directory to which the files of this sink are written.
protected org.apache.jena.riot.Lang	outputLang Language used for the output files.
protected Map<String,FileBasedSink.StreamStatus>	streamMapping Synchronized mapping of crawled URIs to their output stream.
protected boolean	useCompression Flag whether a compression algorithm should be used.

Constructor Summary

Constructors

Constructor and Description
FileBasedSink(File outputDirectory, boolean useCompression)
FileBasedSink(File outputDirectory, org.apache.jena.riot.Lang outputLang, boolean useCompression)

Method Summary

Modifier and Type	Method and Description
void	addData(org.dice_research.squirrel.data.uri.CrawleableUri uri, InputStream is)
void	addQuad(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.sparql.core.Quad quad)
void	addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.graph.Triple triple)
void	closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)
void	flushMetadata()
static String	generateFileName(org.dice_research.squirrel.data.uri.CrawleableUri uri, org.apache.jena.riot.Lang outputLang, boolean useCompression)
private FileBasedSink.StreamStatus	getStream(org.dice_research.squirrel.data.uri.CrawleableUri uri)
void	openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.sink.Sink

addMetaData, close

Methods inherited from interface org.dice_research.squirrel.sink.UnstructuredDataSink

addData, addData

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

DEFAULT_OUTPUT_LANG

public static final org.apache.jena.riot.Lang DEFAULT_OUTPUT_LANG

outputDirectory

protected File outputDirectory

Directory to which the files of this sink are written.

useCompression

protected boolean useCompression

Flag whether a compression algorithm should be used.

streamMapping

protected Map<String,FileBasedSink.StreamStatus> streamMapping

Synchronized mapping of crawled URIs to their output stream.

outputLang

protected org.apache.jena.riot.Lang outputLang

Language used for the output files.

Constructor Detail

FileBasedSink

public FileBasedSink(File outputDirectory,
                     boolean useCompression)

FileBasedSink

public FileBasedSink(File outputDirectory,
                     org.apache.jena.riot.Lang outputLang,
                     boolean useCompression)

Method Detail

addTriple

public void addTriple(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                      org.apache.jena.graph.Triple triple)

Specified by:

addTriple in interface org.dice_research.squirrel.sink.triplebased.TripleBasedSink

flushMetadata

public void flushMetadata()

Specified by:

flushMetadata in interface org.dice_research.squirrel.sink.Sink

addData

public void addData(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                    InputStream is)

Specified by:

addData in interface org.dice_research.squirrel.sink.UnstructuredDataSink

openSinkForUri

public void openSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

openSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

getStream

private FileBasedSink.StreamStatus getStream(org.dice_research.squirrel.data.uri.CrawleableUri uri)

closeSinkForUri

public void closeSinkForUri(org.dice_research.squirrel.data.uri.CrawleableUri uri)

Specified by:

closeSinkForUri in interface org.dice_research.squirrel.sink.SinkBase

generateFileName

public static String generateFileName(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                                      org.apache.jena.riot.Lang outputLang,
                                      boolean useCompression)

addQuad

public void addQuad(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                    org.apache.jena.sparql.core.Quad quad)

Specified by:

addQuad in interface org.dice_research.squirrel.sink.quadbased.QuadBasedSink