public class ThrottledFetcher
extends java.lang.Object
| Modifier and Type | Class and Description |
|---|---|
protected static class |
ThrottledFetcher.ConnectionPool
Each connection pool has identical connections we can draw on.
|
protected static class |
ThrottledFetcher.ConnectionPoolKey
Connection pool key
|
protected static class |
ThrottledFetcher.ExecuteMethodThread
This thread does the actual socket communication with the server.
|
protected static class |
ThrottledFetcher.LaxBrowserCompatSpecProvider
Class to create a cookie spec.
|
protected static class |
ThrottledFetcher.OurBasicCookieStore |
protected static class |
ThrottledFetcher.PoolException
Pool exception class
|
protected static class |
ThrottledFetcher.ThrottledConnection
Throttled connections.
|
protected static class |
ThrottledFetcher.ThrottledInputstream
This class throttles an input stream based on the specified byte rate parameters.
|
protected static class |
ThrottledFetcher.WaitException
Wait exception class
|
| Modifier and Type | Field and Description |
|---|---|
static java.lang.String |
_rcsid |
protected static java.util.Map<ThrottledFetcher.ConnectionPoolKey,ThrottledFetcher.ConnectionPool> |
connectionPools
Connection pools.
|
protected static long |
idleTimeout
Idle timeout
|
protected static int |
READ_CHUNK_LENGTH
The read chunk length
|
protected static boolean |
recordEverything
This flag determines whether we record everything to the disk, as a means of doing a web snapshot
|
protected static long |
TIME_15MIN |
protected static long |
TIME_1DAY |
protected static long |
TIME_2HRS |
protected static long |
TIME_5MIN |
protected static long |
TIME_6HRS |
protected static java.lang.String |
webThrottleGroupType
Web throttle group type
|
| Modifier and Type | Method and Description |
|---|---|
static void |
flushIdleConnections(org.apache.manifoldcf.core.interfaces.IThreadContext threadContext)
Flush connections that have timed out from inactivity.
|
static IThrottledConnection |
getConnection(org.apache.manifoldcf.core.interfaces.IThreadContext threadContext,
java.lang.String throttleGroupName,
java.lang.String protocol,
java.lang.String server,
int port,
PageCredentials authentication,
org.apache.manifoldcf.connectorcommon.interfaces.IKeystoreManager trustStore,
org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec throttleDescription,
java.lang.String[] binNames,
int connectionLimit,
java.lang.String proxyHost,
int proxyPort,
java.lang.String proxyAuthDomain,
java.lang.String proxyAuthUsername,
java.lang.String proxyAuthPassword,
int socketTimeoutMilliseconds,
int connectionTimeoutMilliseconds,
org.apache.manifoldcf.crawler.interfaces.IAbortActivity activities)
Obtain a connection to specified protocol, server, and port.
|
public static final java.lang.String _rcsid
protected static final java.lang.String webThrottleGroupType
protected static final long idleTimeout
protected static final boolean recordEverything
protected static final long TIME_2HRS
protected static final long TIME_5MIN
protected static final long TIME_15MIN
protected static final long TIME_6HRS
protected static final long TIME_1DAY
protected static final int READ_CHUNK_LENGTH
protected static final java.util.Map<ThrottledFetcher.ConnectionPoolKey,ThrottledFetcher.ConnectionPool> connectionPools
public static IThrottledConnection getConnection(org.apache.manifoldcf.core.interfaces.IThreadContext threadContext, java.lang.String throttleGroupName, java.lang.String protocol, java.lang.String server, int port, PageCredentials authentication, org.apache.manifoldcf.connectorcommon.interfaces.IKeystoreManager trustStore, org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec throttleDescription, java.lang.String[] binNames, int connectionLimit, java.lang.String proxyHost, int proxyPort, java.lang.String proxyAuthDomain, java.lang.String proxyAuthUsername, java.lang.String proxyAuthPassword, int socketTimeoutMilliseconds, int connectionTimeoutMilliseconds, org.apache.manifoldcf.crawler.interfaces.IAbortActivity activities) throws org.apache.manifoldcf.core.interfaces.ManifoldCFException, org.apache.manifoldcf.agents.interfaces.ServiceInterruption
protocol - is the protocol, e.g. "http"server - is the server IP address, e.g. "10.32.65.1"port - is the port to connect to, e.g. 80. Pass -1 if the default port for the protocol is desired.authentication - is the page credentials object to use for the fetch. If null, no credentials are available.trustStore - is the current trust store in effect for the fetch.binNames - is the set of bins, in order, that should be used for throttling this connection.
Note that the bin names for a given IP address and port MUST be the same for every connection!
This must be enforced by whatever it is that builds the bins - it must do so given an IP and port.throttleDescription - is the description of all the throttling that should take place.connectionLimit - isthe maximum number of connections permitted.org.apache.manifoldcf.core.interfaces.ManifoldCFExceptionorg.apache.manifoldcf.agents.interfaces.ServiceInterruptionpublic static void flushIdleConnections(org.apache.manifoldcf.core.interfaces.IThreadContext threadContext)
throws org.apache.manifoldcf.core.interfaces.ManifoldCFException
org.apache.manifoldcf.core.interfaces.ManifoldCFException