org.apache.manifoldcf.crawler.connectors.webcrawler

Class ThrottleDescription

java.lang.Object

org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottleDescription

All Implemented Interfaces:

org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec

public class ThrottleDescription
extends java.lang.Object
implements org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec

This class describes complex throttling criteria pulled from a configuration. The data contained is organized by regular expression performed on a bin. What we store for each regular expression is a Pattern, for efficiency. This structure deals with bandwidth limits, maximum connection limits, and maximum fetch rate limits. Average fetch rate limits are handled in the infrastructure. Generally it is a good thing to limit the number of regexps that need to be evaluated against any given bin value as much as possible. For that reason I've organized this structure accordingly.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	ThrottleDescription.ThrottleItem Class representing an individual throttle item.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	_rcsid
protected java.util.Map<java.lang.String,ThrottleDescription.ThrottleItem>	patternHash This is the hash that contains everything.

Constructor Summary

Constructors

Constructor and Description
ThrottleDescription(org.apache.manifoldcf.core.interfaces.ConfigParams configData) Constructor.

Method Summary

Modifier and Type	Method and Description
int	getMaxOpenConnections(java.lang.String binName) Given a bin name, find the max open connections to use for that bin.
double	getMinimumMillisecondsPerByte(java.lang.String binName) Look up minimum milliseconds per byte for a bin.
long	getMinimumMillisecondsPerFetch(java.lang.String binName) Look up minimum milliseconds for a fetch for a bin.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

patternHash

protected java.util.Map<java.lang.String,ThrottleDescription.ThrottleItem> patternHash

This is the hash that contains everything. It's keyed by the regexp string itself. Values are ThrottleItem's.

Constructor Detail

ThrottleDescription

public ThrottleDescription(org.apache.manifoldcf.core.interfaces.ConfigParams configData)
                    throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Constructor. Build the description from the ConfigParams.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

Method Detail

getMaxOpenConnections

public int getMaxOpenConnections(java.lang.String binName)

Given a bin name, find the max open connections to use for that bin.

Specified by:

getMaxOpenConnections in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec

Returns:

Integer.MAX_VALUE if no limit found.

getMinimumMillisecondsPerByte

public double getMinimumMillisecondsPerByte(java.lang.String binName)

Look up minimum milliseconds per byte for a bin.

Specified by:

getMinimumMillisecondsPerByte in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec

Returns:

0.0 if no limit found.

getMinimumMillisecondsPerFetch

public long getMinimumMillisecondsPerFetch(java.lang.String binName)

Look up minimum milliseconds for a fetch for a bin.

Specified by:

getMinimumMillisecondsPerFetch in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec

Returns:

0 if no limit found.