Protocol plugins should report progress to the fetcher
------------------------------------------------------
Key: NUTCH-569
URL: https
://issues.apache.org/jira/browse/NUTCH-569
Project: Nutch
Issue Type: Improvement
Components: fetcher
Affects Versions: 1.0.0
Reporter: Andrzej Bialecki
Priority: Minor
When downloading very large files over slow connections,
protocol plugins spend long time in
Protocol.getProtocolOutput(...). This sometimes leads to a
timeout in Fetcher / Fetcher2, with the message
"aborting with hung threads". Protocol plugins
should periodically notify their caller about progress. In a
situation when the call to getProtocolOutput takes very long
time to return, this will help the caller to determine
whether the wait is justified.
Preferably, the callback interface should allow the
monitoring of not only the binary progress / no-progress,
but also the download speed, so that the caller could
terminate slow connections. E.g.
interface ProtocolReporter {
void progress(long bytesDownloaded);
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.
|
