UNNECESSARY DELAYS IN FETCHER2
------------------------------
KEY: NUTCH-495
URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-495
PROJECT: NUTCH
ISSUE TYPE: BUG
COMPONENTS: FETCHER
AFFECTS VERSIONS: 1.0.0
REPORTER: DO?ACAN GüNEY
PRIORITY: MINOR
FIX FOR: 1.0.0
EVEN IF A URL IS BLOCKED BY ROBOTS.TXT (OR HAS A CRAWL DELAY
LARGER THAT MAX.CRAWL.DELAY), FETCHER2 STILL WAITS
FETCHER.SERVER.DELAY BEFORE FETCHING ANOTHER URL FROM SAME
HOST, WHICH IS NOT NECESSARY, CONSIDERING THAT FETCHER2
DIDN'T MAKE A REQUEST TO SERVER ANYWAY.
--
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.
|