Wishlist: an automated package upgrade system with fine-tunable sysadmin control

Pasted from bugzilla.  Please pardon the ugly newline
formatting.


I'm a longtime (>10 yrs) Linux admin and I've been
using Gentoo for
perhaps 2
years and I'm super impressed with Gentoo, having gotten
very annoyed
with the
rpm-based nightmare upgrade situation presented by most of
the other
distros,
but one thing I'd really like to see in Gentoo is a way of
safely keeping my
Gentoo boxes up to date in an automated way.  I know that
may sound
paradoxical
and mutually contradictory.  I realize that production
systems should not be
upgraded before trying out the upgrade on a testbed system,
but I've
found that
routine cron jobs of emerge world are unsafe because some
packages need a
human's attention for upgrading (like apache or postfix
when config files
should be left untouched or updated or merged with new
config files or some
other issue that needs a human's attention) whereas doing
nothing for a long
time (while the portage tree evolves) makes for a box that
has been
veritably
left behind, sometimes making it difficult or impossible to
upgrade.

I'd like to have the capability of being able to list some
packages that
should
never be upgraded automatically (I realize I can do this to
some degree
already
with portage), some others that are very unlikely to break
from an automated
upgrade and thus should always be upgraded automatically,
and some packages
(which may fit in either or both of these categories) that
must be
upgraded in
a certain order in order to avoid breaking something and
thus, should
probably
be upgraded automatically or (if flagged for preventing
automatic
upgrades by
the admin) should be brought to the attention of the admin
(say with an
email
to root or something) as needing attention to avoid
breakage.

I am asking for this feature after having spent an entire
weekend upgrading
various packages by hand, one or a few at a time, after
carefully
considering
whether or not it would be safe to upgrade this or that
package, and after
having (lazily) not upgraded anything on this production box
in a long
time.
The experience has left me rather exhausted (with a sore ass
from
sitting down
for so long) and wishing for something better.  One
noteworthy experience in
particular is that I found that many packages suffered
sandbox violations on
attempted upgrades, and I troubleshot this problem for a
long time before it
occurred to me that I might want to upgrade the sandbox
package and then try
upgrading these packages.  That solved the sandbox violation
problem.
It seems
to me that this was a case where an automated system could
have insisted on
upgrading the sandbox package first, before the others. 
Perhaps there
should
have been a dependency, so that when I tried to upgrade the
ncurses
library, it
automatically pulled in the newer sandbox package as a
prerequisite (for
that
is what it turned out to be).

After writing this much, it occurs to me that perhaps the
capabilities
that I
describe here may already be in Gentoo/portage in some way
that I've yet to
fully discover and/or utilize, but despite having installed
many Gentoo
systems
and read the Handbook (and many other Gentoo docs) many
times, I've yet
to see
a good write-up on how to do what I describe here.  And
perhaps the fact
that
the sandbox package was not a dependency for the ncurses
package (and
several
others that also broke during emerge with sandbox
violations) was the real
"bug" so to speak, rather than the idea that
Gentoo is missing this major
feature that I'm asking for.  I'm really not sure which
might be true, but I
just thought I'd ask.

One thing that I'm pretty sure is currently not possible
with portage,
however, and that I'd definitely like to see as a part of
this idea is a
way of setting thresholds on version numbers of packages in
portage such
that the automated upgrade system will only upgrade a
package
automatically if the difference in version numbers between
the installed
package and the newest available package in portage is
greater than some
admin-tunable amount.  For example, I might not want to
upgrade emacs or
xemacs just because a new -r number becomes available. 
Maybe I don't
want to have such a big package upgraded automatically
unless there is a
new major or minor version number.

Thanks again to all the developers who have made Gentoo. 
It's a really
terrific distro.


------- Comment #1 From Radek Podgorny 2006-04-24 08:25 PST
[reply] -------

Maybe, the packages themselves could be assigned something
like a
safe-upgrade-flag...


------- Comment #2 From Jakub Moc 2006-04-26 08:46 PST
[reply] -------

Please, take such ideas to portage/devel mailing lists...
Bugzilla is
not the
best place to discuss abstract ideas. Thanks.

-- 
gentoo-devgentoo.org mailing list

On Wed, 2006-04-26 at 12:29 -0400, Kevin wrote:
> One thing that I'm pretty sure is currently not
possible with portage,
> however, and that I'd definitely like to see as a part
of this idea is a
> way of setting thresholds on version numbers of
packages in portage such
> that the automated upgrade system will only upgrade a
package
> automatically if the difference in version numbers
between the installed
> package and the newest available package in portage is
greater than some
> admin-tunable amount.  For example, I might not want to
upgrade emacs or
> xemacs just because a new -r number becomes available. 
Maybe I don't
> want to have such a big package upgraded automatically
unless there is a
> new major or minor version number.
> 
> Thanks again to all the developers who have made
Gentoo.  It's a really
> terrific distro.

Jakub meant the portage-devel mailing list, not this one.

Anyway, most of this can be done already using /etc/portage
files and
some well-written cron scripts.  You can lock down versions
of specific
packages quite easily using your own package.mask and
package.unmask
files, along with package.keywords.  However, one thing you
can *never*
do is assume that *any* package that has *any* kind of
configuration
files or is a library will *never* change in an incompatible
way.

Basically, what you want is the assurances of a binary
distribution that
things will "just work" when upgraded, yet you
still want the power of
Gentoo.  Honestly, I don't see portage ever being able to
really support
anything like this so long as the tree continues to change. 
It simply
doesn't seem to be compatible with how Gentoo development
is done.  Then
again, I could be completely off my rocker and the portage
team might
already have all of this implemented in some super-secret
internal-only
version.  As I said, this would probably be best conversed
with them on
the portage-devel list.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

Chris Gianelloni wrote:
> On Wed, 2006-04-26 at 12:29 -0400, Kevin wrote:
>> One thing that I'm pretty sure is currently not
possible with portage,
>> however, and that I'd definitely like to see as a
part of this idea is a
>> way of setting thresholds on version numbers of
packages in portage such
>> that the automated upgrade system will only upgrade
a package
>> automatically if the difference in version numbers
between the installed
>> package and the newest available package in portage
is greater than some
>> admin-tunable amount.  For example, I might not
want to upgrade emacs or
>> xemacs just because a new -r number becomes
available.  Maybe I don't
>> want to have such a big package upgraded
automatically unless there is a
>> new major or minor version number.
>>
>> Thanks again to all the developers who have made
Gentoo.  It's a really
>> terrific distro.
> 
> Jakub meant the portage-devel mailing list, not this
one.

woops.  when i saw portage/devel i figured one or the
other... guess not...

> 
> Anyway, most of this can be done already using
/etc/portage files and
> some well-written cron scripts.  You can lock down
versions of specific
> packages quite easily using your own package.mask and
package.unmask
> files, along with package.keywords.

Yes, and as I wrote, I'm aware of your points here, and I
do use these
capabilities fairly extensively now, but it is not the sort
of
fine-tunable system that I have in mind where I could inform
the
automated-update-system (for lack of a better name) of my
wishes in
terms of which packages are safe (in my judgment, as the
sysadmin of
that particular box) to upgrade in an unattended manner and
which are
not (at least, not unless I'm much less well-informed on
Gentoo than I
believe myself to be, which I'm not denying might be true).

And unless I'm way off-base, the
version-difference-threshold notion
described above is not implemented in portage now.  Someone
please
correct me if I'm wrong.

>  However, one thing you can *never*
> do is assume that *any* package that has *any* kind of
configuration
> files or is a library will *never* change in an
incompatible way.

Well your comment is certainly true in the most extreme
interpretation,
but the same thing can be accurately said about whether or
not one
should assume that the sun is going to rise tomorrow or that
the
universe won't disappear in a quantum fluctuation while
you're sleeping,
but IMO, such extreme statements have little value in
day-to-day
application.  Everyone must make some assumptions about
nearly
everything or it becomes nearly impossible to function.  I
make all
kinds of assumptions in administering computers and they
almost always
make my life much, MUCH easier than it would be without the
assumptions.
 Sometimes they bite me, but only rarely.  The key to
success here is
having the judgment to know what is relatively safe to make
assumptions
about and what is not.  Judgment is something that only a
human can
provide... not a computer.  This is why I want greater and
more granular
control over upgrading packages in Gentoo.  Aside from the
points you
make above (and I may be missing some other features
currently present
in Gentoo), my choices now are, in the grossest terms:
upgrade every
package by hand, one at a time, while sitting in front of
the computer
(which is very close to what I spent last weekend doing) or
do an emerge
world and hope for the best.  IMO, that's not much control
and does not
allow for the application of judgment except in the former
option (which
is very, very time consuming).

> 
> Basically, what you want is the assurances of a binary
distribution that
> things will "just work" when upgraded, 

No.  Not true at all.  And for that matter, binary
distributions (in my
10+ years of experience with them) simply do NOT just work:
not for
Slackware, not for Yellowdog, not for SuSE, not for Redhat,
nor
Mandrake, nor any of a dozen others I've tried.  I found
that quite the
opposite was true which was one of the reasons I switched to
Gentoo
(which I've found, usually DOES just work after upgrades;
not always,
but usually---and this is a credit to the Gentoo
developers).

What I really want is to make the process of maintaining
Gentoo boxes
over the long term easier (IOW: less time-consuming) than is
now true,
by adding some functionality that AFAICT does not now exist
which would
allow me to automate some things, turn off automation of
other things,
and as the sysadmin, have control over what those things
should be.  In
my mind at least, the central theme in Gentoo of choice
dovetails nicely
with what I'm trying to describe here: control and choice
that is highly
fine-tunable by the owner of the box in regards to package
upgrades.

I'm not a member of the portage-devel mailing list so I'm
going to drop
this now.  If someone here is a member of both, then please
feel free to
cross-post this thread to whatever forum is most appropriate
for it.
After spending 30-45 minutes trying to help improve Gentoo
by posting a
new (AFAICT) idea in bugzilla and again here, I feel like
I've done
enough.  IMHO, this is an idea that would add great value to
Gentoo and
I can't help but think that many sysadmins who must
maintain many boxes
would agree, but I have no particular attachment to the idea
that would
make me want to go around every mailing list under the sun
trumpeting my
idea to anyone who will listen.  I'm just posting an idea
that seemed
like a good one to me.  The devs may take it or leave it as
they see fit.

Regards,
Kevin
-- 
gentoo-devgentoo.org mailing list

On 4/26/06, Kevin <listsgnosysllc.com> wrote:
>
> What I really want is to make the process of
maintaining Gentoo boxes
> over the long term easier (IOW: less time-consuming)
than is now true,
> by adding some functionality that AFAICT does not now
exist which would
> allow me to automate some things, turn off automation
of other things,
> and as the sysadmin, have control over what those
things should be.  In
> my mind at least, the central theme in Gentoo of choice
dovetails nicely
> with what I'm trying to describe here: control and
choice that is highly
> fine-tunable by the owner of the box in regards to
package upgrades.
>
>
> Regards,
> Kevin
> --

Have a look at GLCU (http://www.panhorst.com
/glcu/). Might not be the
perfect solution for your needs but it might help. Tools
like this
one, /etc/portage and a private overlay for testing and/or
pinning
would be pretty usefull for you right now. Might want to
check GLEP 19
IIRC for the enterprise tree idea.

Regards,

Jean-Francois

-- 
gentoo-devgentoo.org mailing list

On Wed, 2006-04-26 at 14:24 -0400, Kevin wrote:
> And unless I'm way off-base, the
version-difference-threshold notion
> described above is not implemented in portage now. 
Someone please
> correct me if I'm wrong.

You're off-base.

See, you can, for example, mask all revisions within the
same version of
a package in your /etc/portage/package.mask file quite
easily.  So you
*could* have xemacs, using your own example, only ask to
upgrade once
the stable version in the tree went over a certain
threshold.

> Well your comment is certainly true in the most extreme
interpretation,
> but the same thing can be accurately said about whether
or not one
> should assume that the sun is going to rise tomorrow or
that the
> universe won't disappear in a quantum fluctuation
while you're sleeping,
> but IMO, such extreme statements have little value in
day-to-day
> application.  Everyone must make some assumptions about
nearly
> everything or it becomes nearly impossible to function.
 I make all
> kinds of assumptions in administering computers and
they almost always
> make my life much, MUCH easier than it would be without
the assumptions.

I'm sorry, but do your friends call you Duncan?  I'll
leave it at that.

>  Sometimes they bite me, but only rarely.  The key to
success here is
> having the judgment to know what is relatively safe to
make assumptions
> about and what is not.  Judgment is something that only
a human can
> provide... not a computer.  This is why I want greater
and more granular
> control over upgrading packages in Gentoo.  Aside from
the points you
> make above (and I may be missing some other features
currently present
> in Gentoo), my choices now are, in the grossest terms:
upgrade every
> package by hand, one at a time, while sitting in front
of the computer
> (which is very close to what I spent last weekend
doing) or do an emerge
> world and hope for the best.  IMO, that's not much
control and does not
> allow for the application of judgment except in the
former option (which
> is very, very time consuming).

You missed the ability to lock down to specific package
versions, which
is already a 100% possibility with current portage.  You can
lock down
the versions to *anything that you want* via package.mask
and
package.unmask, then simply have your system run an
"emerge --update
--deep world" to automatically upgrade any and all
packages not listed
in your mask files.

> What I really want is to make the process of
maintaining Gentoo boxes
> over the long term easier (IOW: less time-consuming)
than is now true,
> by adding some functionality that AFAICT does not now
exist which would
> allow me to automate some things, turn off automation
of other things,
> and as the sysadmin, have control over what those
things should be.  In
> my mind at least, the central theme in Gentoo of choice
dovetails nicely
> with what I'm trying to describe here: control and
choice that is highly
> fine-tunable by the owner of the box in regards to
package upgrades.

Yup.  It's called /etc/portage and we've had it for a
while.  You simply
seem to be missing its flexibility.

> I'm not a member of the portage-devel mailing list so
I'm going to drop
> this now.  If someone here is a member of both, then
please feel free to
> cross-post this thread to whatever forum is most
appropriate for it.
> After spending 30-45 minutes trying to help improve
Gentoo by posting a
> new (AFAICT) idea in bugzilla and again here, I feel
like I've done
> enough.  IMHO, this is an idea that would add great
value to Gentoo and
> I can't help but think that many sysadmins who must
maintain many boxes
> would agree, but I have no particular attachment to the
idea that would
> make me want to go around every mailing list under the
sun trumpeting my
> idea to anyone who will listen.  I'm just posting an
idea that seemed
> like a good one to me.  The devs may take it or leave
it as they see fit.

Yep.  It was such a good idea that the portage team
implemented it quite
some time ago.  *grin*

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

Jean-Francois Gagnon Laporte wrote:
> On 4/26/06, Kevin <listsgnosysllc.com> wrote:
>> What I really want is to make the process of
maintaining Gentoo boxes
>> over the long term easier (IOW: less
time-consuming) than is now true,
>> by adding some functionality that AFAICT does not
now exist which would
>> allow me to automate some things, turn off
automation of other things,
>> and as the sysadmin, have control over what those
things should be.  In
>> my mind at least, the central theme in Gentoo of
choice dovetails nicely
>> with what I'm trying to describe here: control and
choice that is highly
>> fine-tunable by the owner of the box in regards to
package upgrades.
> 
> Have a look at GLCU (http://www.panhorst.com
/glcu/). Might not be the
> perfect solution for your needs but it might help.
Tools like this
> one, /etc/portage and a private overlay for testing
and/or pinning
> would be pretty usefull for you right now. Might want
to check GLEP 19
> IIRC for the enterprise tree idea.

Thanks very kindly for your reply and pointer,
Jean-Francois!

-Kevin
-- 
gentoo-devgentoo.org mailing list

Chris Gianelloni wrote:
>        Honestly, I don't see portage ever being able
to really
> support anything like this so long as the tree
continues to change.
> It simply doesn't seem to be compatible with how
Gentoo development is
> done.

and Chris Gianelloni wrote:

> 
> Yup.  It's called /etc/portage and we've had it for a
while.  You simply
> seem to be missing its flexibility.
<snip>
> Yep.  It was such a good idea that the portage team
implemented it quite
> some time ago.  *grin*
> 

Which is it, Chris?

Impossible that portage will ever be able to really support
anything
like this...

or...

already done and I (along with Jean-Francois and Michael
Schilling) am
just too stupid to see it?

Make up your mind...

For all the credit that I give to the Gentoo developers, you
are one
from whom I would withdraw it.  You are merely a polarity
responder and
as such, not worth engaging any further.  You would do
Gentoo a service
by keeping your mouth shut in response to posts like mine
(and many
others I'm sure).  If I were not so self-assured, you would
simply drive
me away from Gentoo with such ugly remarks, and I've no
doubt that you
have driven away many others who are less self-assured.

You're in my kill file now, so don't waste your time and
the bandwidth
by responding.  All mailing lists have a topic and
attempting to
ridicule other people is not on-topic in any of them... 
certainly not
here.  It just facilitates making you look like a jerk which
you need no
assistance with.

-Kevin
-- 
gentoo-devgentoo.org mailing list

zing !

and with this post it's probably best to let this subthread
die before we get 
any more offtrack
-mike
-- 
gentoo-devgentoo.org mailing list

On Wed, 2006-04-26 at 15:55 -0400, Kevin wrote:
> Which is it, Chris?
You've taken that out of context ...

> Make up your mind...
I think he has, but wasn't able to communicate his ideas to
you in an adequte way

> For all the credit that I give to the Gentoo
developers, you are one
> from whom I would withdraw it.  You are merely a
polarity responder and
> as such, not worth engaging any further.  You would do
Gentoo a service
> by keeping your mouth shut in response to posts like
mine (and many
> others I'm sure).  If I were not so self-assured, you
would simply drive
> me away from Gentoo with such ugly remarks, and I've
no doubt that you
> have driven away many others who are less self-assured.
Dude ... calm down. Seriously. Insulting or angering us
won't help anything and will only make us ignore any good
ideas you may have.
Your condescending tone makes me want to ignore anything you
say ...

> You're in my kill file now, so don't waste your time
and the bandwidth
> by responding.  All mailing lists have a topic and
attempting to
> ridicule other people is not on-topic in any of them...
 certainly not
> here.  It just facilitates making you look like a jerk
which you need no
> assistance with.
Deliberately misunderstanding people and then yelling at
them is also
not something that's on-topic for a mailinglist. 

In the last two weeks or so I've seen many users that
ignore advice,
feel insulted at every turn and are not cooperative in any
way. This is
frustrating and makes those of us who interact with users
irritated. I
really wish we could have a level-headed discussion instead
of
insulting, insinuating and trying to annoy people.

If you feel that a developer is uncooperative, that you
don't get the
attention you deserve ... if you have any idea how to make
things
better, send a mail to user-relationsgentoo.org 
Please don't take your frustrations out on developers, let
us try to fix
the problem.
I hope that we can help you resolve any conflicts you may or
may not
have.

wkr,
Patrick,
UserRel Monkey 
-- 
Stand still, and let the rest of the universe move

On 4/26/06, Kevin <listsgnosysllc.com> wrote:
> I'd like to have the capability of being able to list
some packages that should
> never be upgraded automatically (I realize I can do
this to some degree already
> with portage), some others that are very unlikely to
break from an automated
> upgrade and thus should always be upgraded
automatically, and some packages

So maybe this could be satisified by allowing user-defined
categories
of packages beyond system and world?  Something like world,
system,
fragile, non-fragile?

I think you could probably implement something like this
yourself with
a bit of trickery with the /var/lib/portage/world list.  You
could
copy world to non-fragile, remove anything that you consider
fragile
from it, and then do an "automatic" update with:

cp /var/lib/portage/world /var/lib/portage/world.bak
cp /var/lib/portage/non-fragile /var/lib/portage/world
emerge -DNuv world
cp /var/lib/portage/world.bak /var/lib/portage/world

A similar script for the fragile packages would let you
update those
as a group, obviously using the -p and/or --ask options as
you like.

Going back to -user now...

-Richard

-- 
gentoo-devgentoo.org mailing list