org.dice_research.squirrel.fetcher.http

Class HTTPFetcher

java.lang.Object

org.dice_research.squirrel.fetcher.http.HTTPFetcher

All Implemented Interfaces:

Closeable, AutoCloseable, Fetcher

public class HTTPFetcher
extends Object
implements Fetcher

Fetcher which uses an HTTP client to fetch data and store it in a temporary directory.

Author:

Michael Röder (michael.roeder@uni-paderborn.de)

Field Summary

Fields

Modifier and Type	Field and Description
protected String	acceptCharset The value which will be used for the HTTP Accept Charset header if the give CrawleableUri object does not define a header value.
protected static Set<String>	ACCEPTED_SCHEMES URI schemes which are accepted by this fetcher (i.e., "http" and "https").
protected String	acceptHeader The value which will be used for the HTTP Accept header if the give CrawleableUri object does not define a header value.
protected org.apache.http.impl.client.CloseableHttpClient	client The HTTP client instance used by this feature.
protected File	dataDirectory The temporary directory which will be used to store downloaded data.
static String	DEFAULT_ACCEPT_HEADER_STRING The default HTTP Accept header value which simply accepts everything.
static String	HTTP_RESPONSE_HEADER_PREFIX The prefix which is added to HTTP response headers before they are stored the CrawleableUri's data map.
private static org.slf4j.Logger	LOGGER

Constructor Summary

Constructors

Constructor and Description
HTTPFetcher()
HTTPFetcher(org.apache.http.impl.client.CloseableHttpClient client)
HTTPFetcher(String userAgent)

Method Summary

Modifier and Type	Method and Description
void	close()
File	fetch(org.dice_research.squirrel.data.uri.CrawleableUri uri, Delayer delayer) Fetches a stream of data from the given URI by following the delay implemented by the given Delayer, stores it and returns a File object pointing to the stored data.
protected File	requestData(org.dice_research.squirrel.data.uri.CrawleableUri uri, File outputFile)
void	setAcceptCharset(String acceptCharset)
void	setAcceptHeader(String acceptHeader) The value of the HTTP Accept header field that is used if the given CrawleableUri instance does not define this.
void	setDataDirectory(File dataDirectory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.fetcher.Fetcher

fetch

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

DEFAULT_ACCEPT_HEADER_STRING

public static final String DEFAULT_ACCEPT_HEADER_STRING

The default HTTP Accept header value which simply accepts everything.

See Also:

Constant Field Values

HTTP_RESPONSE_HEADER_PREFIX

public static final String HTTP_RESPONSE_HEADER_PREFIX

The prefix which is added to HTTP response headers before they are stored the CrawleableUri's data map.

See Also:

Constant Field Values

ACCEPTED_SCHEMES

protected static final Set<String> ACCEPTED_SCHEMES

URI schemes which are accepted by this fetcher (i.e., "http" and "https").

acceptHeader

protected String acceptHeader

The value which will be used for the HTTP Accept header if the give CrawleableUri object does not define a header value.

acceptCharset

protected String acceptCharset

The value which will be used for the HTTP Accept Charset header if the give CrawleableUri object does not define a header value.

client

protected org.apache.http.impl.client.CloseableHttpClient client

The HTTP client instance used by this feature.

dataDirectory

protected File dataDirectory

The temporary directory which will be used to store downloaded data.

Constructor Detail

HTTPFetcher

public HTTPFetcher()

HTTPFetcher

public HTTPFetcher(String userAgent)

HTTPFetcher

public HTTPFetcher(org.apache.http.impl.client.CloseableHttpClient client)

Method Detail

fetch

public File fetch(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                  Delayer delayer)

Description copied from interface: Fetcher

Fetches a stream of data from the given URI by following the delay implemented by the given Delayer, stores it and returns a File object pointing to the stored data. If an error occurs, null is returned.

Specified by:

fetch in interface Fetcher

Parameters:

uri - The URI from which data should be fetched.

Returns:

A File object pointing to the downloaded data or null if an error occurred.

requestData

protected File requestData(org.dice_research.squirrel.data.uri.CrawleableUri uri,
                           File outputFile)
                    throws org.apache.http.client.ClientProtocolException,
                           FileNotFoundException,
                           IOException

Throws:

org.apache.http.client.ClientProtocolException

FileNotFoundException

IOException

setAcceptHeader

public void setAcceptHeader(String acceptHeader)

The value of the HTTP Accept header field that is used if the given CrawleableUri instance does not define this. Note that the given string has to follow section 5.3.2 of RFC-7231.

Parameters:

acceptHeader - the new value of the accept header as defined in RFC-7231.

setAcceptCharset

public void setAcceptCharset(String acceptCharset)

setDataDirectory

public void setDataDirectory(File dataDirectory)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException