[ http://issues.apache.org/jira/browse/NUTCH-105?page=all ]
Sami Siren resolved NUTCH-105.
------------------------------
Resolution: Fixed
This is now committed, thanks!
> Network error during robots.txt fetch causes file to be
ignored
>
------------------------------------------------------------
---
>
> Key: NUTCH-105
> URL: http:/
/issues.apache.org/jira/browse/NUTCH-105
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 0.8, 0.8.1, 0.9.0
> Reporter: Rod Taylor
> Priority: Critical
> Fix For: 0.8.1, 0.9.0
>
> Attachments: RobotRulesParser.patch
>
>
> Earlier we had a small network glitch which prevented
us from retrieving
> the robots.txt file for a site we were crawling at the
time:
>
nutch-root-tasktracker-sbider1.sitebuildit.com.log:051005
193021
> task_m_h02y5t Couldn't get robots.txt for
> http://w
ww.japanesetranslator.co.uk/translation/:
>
org.apache.commons.httpclient.ConnectTimeoutException: The
host
> did not accept the connection within timeout of
10000 ms
> Nutch then assumed that because we were unable to
retrieve the file due
> to network issues, that it didn't exist and we could
crawl the entire
> website. Nutch then successfully grabbed a few pages
which were listed
> in the robots.txt as being disallowed.
> I think Nutch should continue attempting to retrieve
the robots.txt file
> until, at very least, we are able to establish a
connection to the host,
> otherwise the host should be ignored until the next
round of fetches.
> The webmaster of japanesetranslator.co.uk filed a
complaint informing us
> of the issue.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the
administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atl
assian.com/software/jira
|
