Simple speedup of 'yum install'

In poking around the yum code, I noticed that performance of
'yum
install foo' can be greatly sped up (assuming a warm cache,
decent
network connection etc.)

Right now, the way in which the user's input of a pakcage
spec is
translated into a package object in packages.parsePackages
forces yum to
load all the data of the packages tables into memory, i.e.
forces a
'select * from packages' for each enabled repo, and the
ensuing creation
of a dict with seven entries for every package. That is slow
for a
number of reasons.

I attach a patch that turns that around so that the database
is queried
directly when the user runs 'yum install', avoiding sucking
lots of data
into memory. If this looks like something that can be
committed, I'll
look into doing similar things for the other uses of
packages.parsePackages.

I did some profiling without and with the patch on a fairly
beefy FC6
machine with core, extras and updates enabled (a total of
8700 packages)
Profiling times are based on a warm cache, local repo
mirrors and are
the average of a handful of runs. I used two simple tests:
      * Install of a nonexisting package; this is one use
that should be
        fast because it's frustrating to have yum take a
long time to
        tell the user about a type. Initially, running 'yum
-d 0 -e 0
        install foobar' took 2.9s (user + system), with 1.3s
spent in
        loading the package metadata and 0.5 s in
packages.parsePackages
        (the patch eliminates the metadata loading step).
With the patch
        applied, this now takes 0.8s, with 0.2s for parsing
packages
      * Install of a single package, another operation that
users expect
        to be fast; I ran 'yum -d 0 -e 0 -y install xpdf'
repeatedly for
        this. Initially, this took 4.3s with 1.3s for
loading the
        package metadata, and 0.5s for
packages.parsePackages. With the
        patch, this now takes 2.7s with 0.2s spent in
parsePackages

David

On Mon, 2007-01-15 at 19:55 +0000, David Lutterkort wrote:
> I attach a patch that turns that around so that the
database is queried
> directly when the user runs 'yum install'

My previous patch was braindead when combining the unmatched
entries
from several repos. This updated patch fixes that.

David

On Wed, 2007-01-17 at 09:49 -0800, David Lutterkort wrote:
> On Tue, 2007-01-16 at 23:46 -0500, seth vidal wrote:
> > As cracktastic as it may seem - our current
globbing mechanism covers
> > things like
> > 
> > [abcdefghijklmno]pkgname.i386
> 
> Out of curiosity, how exactly is that used ? I have a
hard time coming
> up with an example where I would want to glob by
character set.

in the mock x86_64 config we use it to exclude ALL the
packages
named .i386 or .i686 EXCEPT for glibc and a few others. It's
hurky but
required to get stuff to build.

> > I don't think the globtolike function you wrote
will work in that. We
> > use the [] mechanism in mock and for funky
excludes.
> 
> globtolike goes to show that hours of hacking can
easily save you 5
> minutes of reading docs. Turns out that sqlite has a
(non-ANSI) operator
> 'glob' which does exactly what is needed here. Updated
patch attached; I
> can now successfully run 'yum install [xyz]p?f*'.

rock.


> BTW, packages.parsePackages checks if the pattern
contains '[*,[,],
> {,},?]' - I think the '{}' shouldn't be in there,
they are not
> special in globs, and fnmatch.translate just puts them
literally into
> the regexp.

ah, a bug, how nice. Thank you.



I don't have your original email anymore - how much of a
speed
improvement were you seeing with your patch?

-sv


_______________________________________________
Yum-devel mailing list
Yum-devellinux.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/yum-devel