Index: conf/nutch-default.xml
===================================================================
--- conf/nutch-default.xml (revision 1605762)
+++ conf/nutch-default.xml (working copy)
@@ -684,7 +684,10 @@
fetcher.server.delay
5.0
The number of seconds the fetcher will delay between
- successive requests to the same server.
+ successive requests to the same server. Note that this might get
+ overriden by a Crawl-Delay from a robots.txt and is used ONLY if
+ fetcher.threads.per.queue is set to 1.
+
@@ -692,7 +695,7 @@
0.0
The minimum number of seconds the fetcher will delay between
successive requests to the same server. This value is applicable ONLY
- if fetcher.threads.per.host is greater than 1 (i.e. the host blocking
+ if fetcher.threads.per.queue is greater than 1 (i.e. the host blocking
is turned off).
@@ -723,7 +726,10 @@
fetcher.threads.per.queue
1
This number is the maximum number of threads that
- should be allowed to access a queue at one time.
+ should be allowed to access a queue at one time. Setting it to
+ a value > 1 will cause the Crawl-Delay value from robots.txt to
+ be ignored and the value of fetcher.server.min.delay to be used
+ as a delay between successive requests to the same server.