org.apache.manifoldcf.crawler.connectors.webcrawler

Class ThrottledFetcher.ThrottledConnection

java.lang.Object

org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher.ThrottledConnection

All Implemented Interfaces:

IThrottledConnection

Enclosing class:

ThrottledFetcher

protected static class ThrottledFetcher.ThrottledConnection
extends java.lang.Object
implements IThrottledConnection

Throttled connections. Each instance of a connection describes the bins to which it belongs, along with the actual open connection itself, and the last time the connection was used.

Field Summary

Fields

Modifier and Type	Field and Description
protected AbortChecker	abortCheck Abort checker
protected PageCredentials	authentication Authentication
protected int	connectionTimeoutMilliseconds Connection timeout milliseconds
protected org.apache.http.conn.HttpClientConnectionManager	connManager The http connection manager.
protected long	expireTime This is when the connection will expire.
protected long	fetchCounter The current bytes in the current fetch
protected org.apache.http.client.methods.HttpRequestBase	fetchMethod The method object
protected org.apache.manifoldcf.connectorcommon.interfaces.IFetchThrottler	fetchThrottler Fetch throttler
protected java.lang.String	fetchType The kind of fetch we are doing
protected org.apache.http.client.HttpClient	httpClient The http client object.
protected javax.net.ssl.SSLSocketFactory	httpsSocketFactory Https protocol
protected LoginCookies	lastFetchCookies The cookies from the last fetch
protected ThrottledFetcher.ExecuteMethodThread	methodThread The thread that is actually doing the work
protected ThrottledFetcher.ConnectionPool	myPool Connection pool
protected java.lang.String	myUrl The current URL being fetched
protected int	port Port
protected java.lang.String	protocol Protocol
protected java.lang.String	proxyAuthDomain Proxy auth domain
protected java.lang.String	proxyAuthPassword Proxy auth password
protected java.lang.String	proxyAuthUsername Proxy auth user name
protected java.lang.String	proxyHost Proxy host
protected int	proxyPort Proxy port
protected java.lang.String	server Server
protected int	socketTimeoutMilliseconds Socket timeout milliseconds
protected long	startFetchTime The start of the current fetch
protected int	statusCode The status code fetched, if any
protected boolean	threadStarted Set if thread has been started
protected java.lang.Throwable	throwable The error trace, if any

Fields inherited from interface org.apache.manifoldcf.crawler.connectors.webcrawler.IThrottledConnection

_rcsid, FETCH_BAD_URI, FETCH_CIRCULAR_REDIRECT, FETCH_INTERRUPTED, FETCH_IO_ERROR, FETCH_NOT_TRIED, FETCH_SEQUENCE_ERROR, FETCH_UNKNOWN_ERROR

Constructor Summary

Constructors

Constructor and Description

ThrottledConnection(ThrottledFetcher.ConnectionPool myPool, org.apache.manifoldcf.connectorcommon.interfaces.IFetchThrottler fetchThrottler, java.lang.String protocol, java.lang.String server, int port, PageCredentials authentication, javax.net.ssl.SSLSocketFactory httpsSocketFactory, java.lang.String proxyHost, int proxyPort, java.lang.String proxyAuthDomain, java.lang.String proxyAuthUsername, java.lang.String proxyAuthPassword, int socketTimeoutMilliseconds, int connectionTimeoutMilliseconds) Constructor.

Method Summary

Modifier and Type	Method and Description
void	beginFetch(java.lang.String fetchType) Begin the fetch process.
void	close() Close the connection.
void	destroy() Destroy the connection forever
void	doneFetch(org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities) Done with the fetch.
void	executeFetch(java.lang.String urlPath, java.lang.String userAgent, java.lang.String from, boolean redirectOK, java.lang.String host, FormData formData, LoginCookies loginCookies) Execute the fetch and get the return code.
LoginCookies	getLastFetchCookies() Get the last fetch cookies.
java.lang.String	getLimitedResponseBody(int maxSize, java.lang.String encoding) Get limited response as a string.
java.io.InputStream	getResponseBodyStream() Get the response input stream.
int	getResponseCode() Get the http response code.
java.lang.String	getResponseHeader(java.lang.String headerName) Get a specified response header, if it exists.
java.util.Map<java.lang.String,java.util.List<java.lang.String>>	getResponseHeaders() Get response headers
protected void	handleHTTPException(org.apache.http.HttpException e, java.lang.String activity)
protected void	handleIOException(java.io.IOException e, java.lang.String activity)
boolean	hasExpired(long currentTime) Check whether the connection has expired.
void	logFetchCount(int count) Log the fetch of a number of bytes, from within a stream.
void	noteInterrupted(java.lang.Throwable e) Note that the connection fetch was interrupted by something.
void	setAbortChecker(AbortChecker abortCheck) Set the abort checker.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

myPool

protected final ThrottledFetcher.ConnectionPool myPool

Connection pool

fetchThrottler

protected final org.apache.manifoldcf.connectorcommon.interfaces.IFetchThrottler fetchThrottler

Fetch throttler

protocol

protected final java.lang.String protocol

Protocol

server

protected final java.lang.String server

Server

port

protected final int port

Port

authentication

protected final PageCredentials authentication

Authentication

expireTime

protected long expireTime

This is when the connection will expire. Only valid if connection is in the pool.

connManager

protected org.apache.http.conn.HttpClientConnectionManager connManager

The http connection manager. The pool is of size 1.

httpClient

protected org.apache.http.client.HttpClient httpClient

The http client object.

fetchMethod

protected org.apache.http.client.methods.HttpRequestBase fetchMethod

The method object

throwable

protected java.lang.Throwable throwable

The error trace, if any

myUrl

protected java.lang.String myUrl

The current URL being fetched

statusCode

protected int statusCode

The status code fetched, if any

fetchType

protected java.lang.String fetchType

The kind of fetch we are doing

fetchCounter

protected long fetchCounter

The current bytes in the current fetch

startFetchTime

protected long startFetchTime

The start of the current fetch

lastFetchCookies

protected LoginCookies lastFetchCookies

The cookies from the last fetch

proxyHost

protected final java.lang.String proxyHost

Proxy host

proxyPort

protected final int proxyPort

Proxy port

proxyAuthDomain

protected final java.lang.String proxyAuthDomain

Proxy auth domain

proxyAuthUsername

protected final java.lang.String proxyAuthUsername

Proxy auth user name

proxyAuthPassword

protected final java.lang.String proxyAuthPassword

Proxy auth password

httpsSocketFactory

protected final javax.net.ssl.SSLSocketFactory httpsSocketFactory

Https protocol

socketTimeoutMilliseconds

protected final int socketTimeoutMilliseconds

Socket timeout milliseconds

connectionTimeoutMilliseconds

protected final int connectionTimeoutMilliseconds

Connection timeout milliseconds

methodThread

protected ThrottledFetcher.ExecuteMethodThread methodThread

The thread that is actually doing the work

threadStarted

protected boolean threadStarted

Set if thread has been started

abortCheck

protected AbortChecker abortCheck

Abort checker

Constructor Detail

ThrottledConnection

public ThrottledConnection(ThrottledFetcher.ConnectionPool myPool,
                           org.apache.manifoldcf.connectorcommon.interfaces.IFetchThrottler fetchThrottler,
                           java.lang.String protocol,
                           java.lang.String server,
                           int port,
                           PageCredentials authentication,
                           javax.net.ssl.SSLSocketFactory httpsSocketFactory,
                           java.lang.String proxyHost,
                           int proxyPort,
                           java.lang.String proxyAuthDomain,
                           java.lang.String proxyAuthUsername,
                           java.lang.String proxyAuthPassword,
                           int socketTimeoutMilliseconds,
                           int connectionTimeoutMilliseconds)

Constructor. Create a connection with a specific server and port, and register it as active against all bins.

Method Detail

setAbortChecker

public void setAbortChecker(AbortChecker abortCheck)

Set the abort checker. This must be done before the connection is actually used.

Specified by:

setAbortChecker in interface IThrottledConnection

hasExpired

public boolean hasExpired(long currentTime)

Check whether the connection has expired.

Specified by:

hasExpired in interface IThrottledConnection

Parameters:

currentTime - is the current time to use to judge if a connection has expired.

Returns:

true if the connection has expired, and should be closed.

logFetchCount

public void logFetchCount(int count)

Log the fetch of a number of bytes, from within a stream.

destroy

public void destroy()

Destroy the connection forever

Specified by:

destroy in interface IThrottledConnection

beginFetch

public void beginFetch(java.lang.String fetchType)
                throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                       org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Begin the fetch process.

Specified by:

beginFetch in interface IThrottledConnection

Parameters:

fetchType - is a short descriptive string describing the kind of fetch being requested. This is used solely for logging purposes.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

executeFetch

public void executeFetch(java.lang.String urlPath,
                         java.lang.String userAgent,
                         java.lang.String from,
                         boolean redirectOK,
                         java.lang.String host,
                         FormData formData,
                         LoginCookies loginCookies)
                  throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                         org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Execute the fetch and get the return code. This method uses the standard logging mechanism to keep track of the fetch attempt. It also signals the following conditions: ServiceInterruption (if a dynamic error occurs), or ManifoldCFException if a fatal error occurs, or nothing if a standard protocol error occurs. Note that, for proxies etc, the idea is for this fetch request to handle whatever redirections are needed to support proxies.

Specified by:

executeFetch in interface IThrottledConnection

Parameters:

urlPath - is the path part of the url, e.g. "/robots.txt"

userAgent - is the value of the userAgent header to use.

from - is the value of the from header to use.

redirectOK - should be set to true if you want redirects to be automatically followed.

host - is the value to use as the "Host" header, or null to use the default.

formData - describes additional form arguments and how to fetch the page.

loginCookies - describes the cookies that should be in effect for this page fetch.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseCode

public int getResponseCode()
                    throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                           org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the http response code.

Specified by:

getResponseCode in interface IThrottledConnection

Returns:

the response code. This is either an HTTP response code, or one of the codes above.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getLastFetchCookies

public LoginCookies getLastFetchCookies()
                                 throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the last fetch cookies.

Specified by:

getLastFetchCookies in interface IThrottledConnection

Returns:

the cookies now in effect from the last fetch.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseHeaders

public java.util.Map<java.lang.String,java.util.List<java.lang.String>> getResponseHeaders()
                                                                                    throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                                                                           org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get response headers

Specified by:

getResponseHeaders in interface IThrottledConnection

Returns:

a map keyed by header name containing a list of values.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseHeader

public java.lang.String getResponseHeader(java.lang.String headerName)
                                   throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                          org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get a specified response header, if it exists.

Specified by:

getResponseHeader in interface IThrottledConnection

Parameters:

headerName - is the name of the header.

Returns:

the header value, or null if it doesn't exist.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseBodyStream

public java.io.InputStream getResponseBodyStream()
                                          throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                                 org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the response input stream. It is the responsibility of the caller to close this stream when done.

Specified by:

getResponseBodyStream in interface IThrottledConnection

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getLimitedResponseBody

public java.lang.String getLimitedResponseBody(int maxSize,
                                               java.lang.String encoding)
                                        throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                               org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get limited response as a string.

Specified by:

getLimitedResponseBody in interface IThrottledConnection

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

noteInterrupted

public void noteInterrupted(java.lang.Throwable e)

Note that the connection fetch was interrupted by something.

Specified by:

noteInterrupted in interface IThrottledConnection

doneFetch

public void doneFetch(org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
               throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Done with the fetch. Call this when the fetch has been completed. A log entry will be generated describing what was done.

Specified by:

doneFetch in interface IThrottledConnection

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

close

public void close()

Close the connection. Call this to return the connection to its pool.

Specified by:

close in interface IThrottledConnection

handleHTTPException

protected void handleHTTPException(org.apache.http.HttpException e,
                                   java.lang.String activity)
                            throws org.apache.manifoldcf.agents.interfaces.ServiceInterruption,
                                   org.apache.manifoldcf.core.interfaces.ManifoldCFException

Throws:

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

org.apache.manifoldcf.core.interfaces.ManifoldCFException

handleIOException

protected void handleIOException(java.io.IOException e,
                                 java.lang.String activity)
                          throws org.apache.manifoldcf.agents.interfaces.ServiceInterruption,
                                 org.apache.manifoldcf.core.interfaces.ManifoldCFException

Throws:

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

org.apache.manifoldcf.core.interfaces.ManifoldCFException