RobotRulesParser.RobotRuleSet

org.supermind.crawl.http
Class RobotRulesParser.RobotRuleSet

java.lang.Object
  org.supermind.crawl.http.RobotRulesParser.RobotRuleSet

This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

Field Summary
`(package private) org.supermind.crawl.http.RobotRulesParser.RobotRuleSet.RobotsEntry[]`	`entries`
`(package private) long`	`expireTime`
`(package private) java.util.ArrayList`	`tmpEntries`

Method Summary
`long`	`getExpireTime()` Get expire time
`boolean`	`isAllowed(java.lang.String path)` Returns `false` if the `robots.txt` file prohibits us from accessing the given `path`, or `true` otherwise.
`void`	`setExpireTime(long expireTime)` Change when the ruleset goes stale.
`java.lang.String`	`toString()`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

Field Detail

org.supermind.crawl.http.RobotRulesParser.RobotRuleSet.RobotsEntry[] entries

long expireTime

java.util.ArrayList tmpEntries

Method Detail

public long getExpireTime()

Get expire time

public boolean isAllowed(java.lang.String path)

Returns false if the robots.txt file prohibits us from accessing the given path, or true otherwise.

public void setExpireTime(long expireTime)

Change when the ruleset goes stale.

public java.lang.String toString()