Improvement suggestion for emerge: Not using a new connection for every file

Hi,

I recommend not to use wget and not to reconnect to the
server for every 
single packet, but to hold the connection
therefore spare traffic and download more fast.

Beginner
-- 
gentoo-portage-devgentoo.org mailing list

Beginner wrote:
> Hi,
> 
> I recommend not to use wget and not to reconnect to the
server for every 
> single packet, but to hold the connection
> therefore spare traffic and download more fast.

Uhh, what exactly are you talking about? There is very
little overhead to 
reconnecting to a server to download a second file. Also,
how often are multiple 
files downloaded from the same server? Probably not very
often. I've heard of 
people trying to rice their binaries, but ricing your
downloads? :P

-- 
Andrew Gaffney                            http://dev.gentoo.or
g/~agaffney/
Gentoo Linux Developer                                  
Installer Project
-- 
gentoo-portage-devgentoo.org mailing list

Andrew Gaffney schrieb:
> Beginner wrote:
>> Hi,
>>
>> I recommend not to use wget and not to reconnect to
the server for 
>> every single packet, but to hold the connection
>> therefore spare traffic and download more fast.
>
> Uhh, what exactly are you talking about? There is very
little overhead 
> to reconnecting to a server to download a second file.
Also, how often 
> are multiple files downloaded from the same server?
Probably not very 
> often. I've heard of people trying to rice their
binaries, but ricing 
> your downloads? :P
>
When I make a -uDa world emerge loads an lot of files from a
ftp-mirror 
and at every file wget ist logging in as anonymous/chdir/set
passive 
mode, i thinked that it is better to keep the ftpconnection
and only 
send a retr for every file, i think emerge is coded in
python so this 
can be handeld by the ftplib.
-- 
gentoo-portage-devgentoo.org mailing list

On Sat, Feb 24, 2007 at 10:00:29PM +0100, Beginner wrote:
> I recommend not to use wget and not to reconnect to the
server for every 
> single packet, but to hold the connection
> therefore spare traffic and download more fast.
If you are doing lots of downloads, use 'emerge -pvf FOO'
and feed each
line of that output to whatever you want to do your
fetching.

On that, I haven't kept up with the code in recent years, is
there a way
that portage itself can hand off those entire lines to a
fetching
application, instead of putting them in one by one? (Telling
the app
about the expected size and checksums would be handy too).

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488
4E85

On Sat, Feb 24, 2007 at 05:55:47PM -0800, Robin H. Johnson
wrote:
> On Sat, Feb 24, 2007 at 10:00:29PM +0100, Beginner
wrote:
> > I recommend not to use wget and not to reconnect
to the server for every 
> > single packet, but to hold the connection
> > therefore spare traffic and download more fast.
> If you are doing lots of downloads, use 'emerge -pvf
FOO' and feed each
> line of that output to whatever you want to do your
fetching.
> 
> On that, I haven't kept up with the code in recent
years, is there a way
> that portage itself can hand off those entire lines to
a fetching
> application, instead of putting them in one by one?
(Telling the app
> about the expected size and checksums would be handy
too).

Current fetch implementation... not worth trying.  No
abstraction 
built into it- would suggest if you're looking to try this,
either rip 
off the old EBD/saviour fetch refactoring (ick), or rip what
we've got 
in pkgcore now.

EBD version had ftplib/httplib direct usage; for pkgcore,
dropped the 
builtin mainly... since I was too lazy to update it.

Either way, trying it with current fetch implementation in
portage, 
would suggest either gutting from codebases mentioned above,
or 
refactoring fetch such that FETCHCOMMAND/RESUMECOMMAND are 
encapsulated and the fetcher functor/obj is pulled from the
passed in 
settings instance.

~harring