catalyst enhancement

Hi List,

We use catalyst to build our server install images for our
serverpark,
we do so on a couple of servers and a couple of development
laptops.
Sadly these are not all set up equally. Which makes editting
and
developing specfiles a pain at times.

We would like to be able to specify the following in
specfiles:

portage_confdir: portage/webserver

This would enable us to keep our specfiles in svn and check
them out on
every host neccesary, as long as it has a valid
catalyst.conf file
building stages should "Just Work (TM)". Currently
it doesn't because
some of the vars in the specfiles need to be in absolute
form to work.

I would like to suggest the following solution:

Introduction of a new variable in catalyst.conf:
relative_basepath="somepath"

Introduction of new code in catalyst to do the following:
detect usage of a relative path in the specfile vars (aka
does it start
with a /) and add relative_base in front of the relative
path.
If relative_base is not defined, default to / ( or maybe to
catalyst.conf/share_dir ?)

This would affect the following vars in the specfiles:
portage_confdir, portage_overlay, pkgcache_path,
kerncache_path,
fsscript, linuxrc (?), motd, root_overlay, xinitrc,
kernel/<label>/config
We're mainly using stage4 specfiles, I had a quick look
through the
other examples in catalyst but couldn't find any other
likely
candidates, but if I'm missing any please let me know.

To be clear, I'm willing to do all the work to get this
implemented, I
already hacked up our current catalyst-2.0.1 install to do
just this.
It is however rather hackish and i would like to get this in
the main
tree. The patch would be against current svn.

I'm curious about a number of things:

* Would this functionality be useful for more people on the
list ?
* Would this patch ever stand a chance of getting integrated
?

Apart from that I'd appreciate any comments, caveats,
critisism etc.

Grtz Ramon

-- 
gentoo-catalystgentoo.org mailing list

On Mon, 2007-04-23 at 14:02 +0200, Ramon van Alteren wrote:
> To be clear, I'm willing to do all the work to get this
implemented, I
> already hacked up our current catalyst-2.0.1 install to
do just this.
> It is however rather hackish and i would like to get
this in the main
> tree. The patch would be against current svn.
> 
> I'm curious about a number of things:
> 
> * Would this functionality be useful for more people on
the list ?

What is stopping you from using an absolute path on the
machines?  If
you control them, standardize on a checkout location.

> * Would this patch ever stand a chance of getting
integrated ?

Not unless we can come up with some reason why we would need
to add the
code complexity to catalyst.  Essentially, there would have
to be a few
use cases that would absolutely prohibit using absolute
paths, otherwise
I don't see a reason for changing it.  This has been brought
up before
and always shot down simply because nobody could ever give
me a reason
why an absolute path wouldn't work for them and only a
relative would.
If you can show that what you want to do cannot be
accomplished with the
current code and absolute paths, then it would be accepted. 
Remember
that simply making something easier for you isn't a valid
reason for
hacking up catalyst internals that much.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

> Not unless we can come up with some reason why we would
need to add the
> code complexity to catalyst.  Essentially, there would
have to be a few
> use cases that would absolutely prohibit using absolute
paths,
> otherwise I don't see a reason for changing it.  This
has been brought
> up before and always shot down simply because nobody
could ever give me
> a reason why an absolute path wouldn't work for them
and only a
> relative would. If you can show that what you want to
do cannot be
> accomplished with the current code and absolute paths,
then it would be
> accepted.  Remember that simply making something easier
for you isn't a
> valid reason for hacking up catalyst internals that
much.

i'd ask the other way around: why only allow absolute
paths?
this limitation strikes me as counter intuitive.

when i started using catalyst, i was expecting it to work
this way and 
wondered why it didnt. so far, i find your arguments against
it a bit 
weak.... i would love to have that extra flexibility.
(the guy even said he wanted to do the coding!)

heck, show the (hackish) patch, and lets examine "the
bloat"...
perhaps this needs to be part of the 2.1 branch!


have fun
kind regards
Thilo

Chris Gianelloni wrote:
> On Mon, 2007-04-23 at 14:02 +0200, Ramon van Alteren
wrote:
>   
>> To be clear, I'm willing to do all the work to get
this implemented, I
>> already hacked up our current catalyst-2.0.1
install to do just this.
>> It is however rather hackish and i would like to
get this in the main
>> tree. The patch would be against current svn.
>>
>> I'm curious about a number of things:
>>
>> * Would this functionality be useful for more
people on the list ?
>>     
>
> What is stopping you from using an absolute path on the
machines?  If
> you control them, standardize on a checkout location.
>   
To be honest, nothing, it's a nuisance or an itch that's
driving this 
request not a necessity.
Most of us develop on laptops and commit code once it works,
with 
deadline pressure as it is this tends to fuck up paths in
the specfiles, 
which need to be reverted, yadadada.
Apart from that having the ability to checkout both devel-
and 
production-branches  on the same buildmachine without the
need to change 
the paths in the specfiles would be cool.
>> * Would this patch ever stand a chance of getting
integrated ?
>>     
>
> Not unless we can come up with some reason why we would
need to add the
> code complexity to catalyst.  Essentially, there would
have to be a few
> use cases that would absolutely prohibit using absolute
paths, otherwise
> I don't see a reason for changing it.  This has been
brought up before
> and always shot down simply because nobody could ever
give me a reason
> why an absolute path wouldn't work for them and only a
relative would.
> If you can show that what you want to do cannot be
accomplished with the
> current code and absolute paths, then it would be
accepted.

OK, clear enough.
I can't think of a use-case where absolute paths will not
work and 
relative paths will, I have trouble finding such a use-case
in general.
If that's the criteria then I guess it's a doomed
enhancement request.

> Remember that simply making something easier for you
isn't a valid reason for
> hacking up catalyst internals that much.
>   
This puzzles me, if making a tool easier to use for it's
users isn't a 
valid reason for hacking at the tool, then what is ?
I understand issues with code-quality, reluctance to touch
internals and 
show me code before talk..........

It was a dual request, if nobody on list would be using the

functionality I'll maintain a patch outside the tree for our
benefit.
No sense in intergrating then.

Best regards,

Ramon
-- 
gentoo-catalystgentoo.org mailing list

Thilo Bangert wrote:
> heck, show the (hackish) patch, and lets examine
"the bloat"...
> perhaps this needs to be part of the 2.1 branch!
> 
>   
Naahh, it isn't in any shape to show anyone at all,
currently it doesn't 
even work correctly if catalyst is not called from within
share_dir.
I did the patch in 15 minutes with no prior knowledge of
catalyst 
internals after loosing about an hour of work on figuring
out what went 
wrong to find yet another mixed-up absolute path.

If I have some spare time next week I'll see if I can come
up with 
something I am not too embarrassed to show 

Regards,

Ramon
-- 
gentoo-catalystgentoo.org mailing list

<snip>
> > Remember that simply making something easier for
you isn't a valid reason for
> > hacking up catalyst internals that much.
> >   
> This puzzles me, if making a tool easier to use for
it's users isn't a 
> valid reason for hacking at the tool, then what is ?
> I understand issues with code-quality, reluctance to
touch internals and 
> show me code before talk..........
</snip>

The reasoning is quite simple. The target "users"
of catalyst are first,
foremost, and finally members of Gentoo's Release
Engineering team. The
fact that other people use it is a byproduct of making the
code
available for such use, not the actual intention. When
considering
enhancement requests they have to filter through 3 major
questions.

1). Is this functionality going to make catalyst easier to
use or more
robust as *Gentoo's* release management tool.

If it doesn't make the job of releasing *official* Gentoo
media easier
then we fall to question 2.

2). Is this feature in enough public demand and can enough
use cases for
it be provided that even though it won't necessarily enhance
the process
of building *official* release materials.

If so then question 3 comes into play.

3). Can it be done without complicating the code enough that
Gentoo's
capability to make reproducible releases is impaired *and*
without
making harder to maintain and/or extend the code for the
purpose of a
Gentoo's official releases.

Again I want to stress that the fact that other people use
catalyst is
nice, and we encourage it to a certain extent, but the
purpose of
catalyst is to build official Gentoo media, no more, no
less.

--Dan

On Mon, 2007-04-23 at 23:39 +0200, Ramon van Alteren wrote:
> This puzzles me, if making a tool easier to use for
it's users isn't a 
> valid reason for hacking at the tool, then what is ?

Well, I know I have said this before, but I'll say it again
here.  We
develop catalyst for our own usage first, and everyone else
second.  If
it doesn't directly impact Release Engineering, it
immediately gets a
back seat to changes that we need/want.  When I said
"you" here, I meant
you specifically, not any other form of you.

> It was a dual request, if nobody on list would be using
the 
> functionality I'll maintain a patch outside the tree
for our benefit.

This was really my question.  Would other people use it?

One of the biggest problems that we have had with catalyst
is people
that want to change catalyst to meet their own specific
needs and our
need to balance things out so that we don't end up with
unused code
paths.  Having a single, consistent interface for the spec
files allows
for much simpler support on a product that we honestly
wished we didn't
have to support, at all.  If the change is something that
lots of people
would likely use, such as the stage4 target, then we will
add it even if
we don't use it ourselves.  Our general rule is don't change
anything
unless there is a really good reason.  As I said, simply
making things
slightly more convenient isn't really a good enough reason,
IMO, unless
a lot of people would use the functionality, and even then,
it would
depend on code availability and maintainability.  Of course,
writing up
a patch resolves the first issue, but the second would still
remain.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

Chris Gianelloni wrote:
> On Mon, 2007-04-23 at 23:39 +0200, Ramon van Alteren
wrote:
>   
>> This puzzles me, if making a tool easier to use for
it's users isn't a 
>> valid reason for hacking at the tool, then what is
?
>>     
> Well, I know I have said this before, but I'll say it
again here.  We
> develop catalyst for our own usage first, and everyone
else second.  If
> it doesn't directly impact Release Engineering, it
immediately gets a
> back seat to changes that we need/want.  When I said
"you" here, I meant
> you specifically, not any other form of you.
>   
Fair enough and very clear.
I have to say that this is a much clearer rejection reason
then "show me 
a use-case which can only be solved with relative paths in
the specfile".
I'd be very hard-pressed to come up with such a use-case
regardless of 
the tool.
>> It was a dual request, if nobody on list would be
using the 
>> functionality I'll maintain a patch outside the
tree for our benefit.
>>     
>
> This was really my question.  Would other people use
it?
>
> One of the biggest problems that we have had with
catalyst is people
> that want to change catalyst to meet their own specific
needs and our
> need to balance things out so that we don't end up with
unused code
> paths.  Having a single, consistent interface for the
spec files allows
> for much simpler support on a product that we honestly
wished we didn't
> have to support, at all.  If the change is something
that lots of people
> would likely use, such as the stage4 target, then we
will add it even if
> we don't use it ourselves.  Our general rule is don't
change anything
> unless there is a really good reason.  As I said,
simply making things
> slightly more convenient isn't really a good enough
reason, IMO, unless
> a lot of people would use the functionality, and even
then, it would
> depend on code availability and maintainability.  Of
course, writing up
> a patch resolves the first issue, but the second would
still remain.
>   
Well let's see if there are others chiming in who want this

functionality 

I'll give a short description what we do with catalyst just
for the hell 
of it,
you might enjoy hearing what others do with the tool even
though it's 
primarily developed for gentoo-releases.

Catalyst is one of the two gentoo projects which have a
central role in 
our hands-off deployment system in our serverpark.
We prepare stage4 server images for our servers with
catalyst for both 
amd64 and x86 archs.
We also generate development vmware images for our
developers with it, 
which we try to keep as similar to production as we can get
them.

We boot new servers with pxe and load them with a read-only
nfs-root and 
a tmpfs overlay for host-specifics.
After that the install-tool by agaffney kicks in and
installs the 
server-image created with catalyst on the new server.
We've modified install-tools to be able to look up install
parameters in 
a mysql-database and use that to setup server-task specific
parameters 
and networking.

After installing the server is rebooted and comes up with
correct 
hostname and network settings.
Catalyst stage4 allows us to setup a few services to be
started at boot, 
one of these is puppet a configuration client.
It connects to (one of) our puppet servers (aptly named
puppetmasters 
 and
loads the configuration for the services the server is
supposed 
to perform.

The entire system helps us deploy new hardware:
Sticking a server in a rack in one of our datacenters to
having it ready 
for production takes 30 minutes and we estimate (but haven't
tested) 
that we can easily do 100 in an hour.

Catalyst is a key component in the system because it has
made our build 
process for system images repeatable.
Before catalyst we were using chroots to build system images

semi-manually, rebuilding a chroot to solve a bug after
deployment was 
not a nice process to go through.
It's also removing a lot of bugs which were due to
development vmware 
images being totally different from server-images.

We use the package-cache from catalyst to drive our binary
package 
server and are investigating the use of the tinderbox target
to generate 
repeatable builds for binary packages on top of the
system-image 
versions we have deployed in our serverpark.

Right now we're busy converting our entire serverpark of 600
servers to 
catalyst-based server-images.
For a lot of server classes this is quick and painless
re-install, for 
our data-intensive servers this is harder but we'll get
there in the end.

If it wasn't clear from the above, we're extremely happy
with catalyst 
If anyone is curious or wants to know more, feel free to
poke me on irc, 
my nick is innocenti

Regards,

Ramon

-- 
gentoo-catalystgentoo.org mailing list

On Tue, 2007-04-24 at 10:41 -0400, John Eckhart wrote:
>    I often build different types of images using
variations on the
> spec files and in order to not pollute my tested
working environments,
> I copy the whole thing into a new dir. I currently use
some sed magic
> to set the absolute paths but if catalyst used relative
paths it would
> make life just a little better.

I guess I just don't get why people are putting paths they
know are
going to be variable in a spec file.

Example stage1.spec:
subarch: x86
target: stage1
rel_type: default
profile: default-linux/x86/no-nptl
chost: i486-pc-linux-gnu

Example script:
#!/bin/bash
version=$1
source=$2

catalyst -f stage1.spec -c /path/to/some/catalyst.conf -C
version_stamp=
$version snapshot=$version source_subpath=$source

Remember that you don't have to have everything in the spec
file.
There's no need for any sed-fu, at all.  Simply have your
script provide
whatever it is that you want and give it to catalyst via -C
on the
command line, which works in conjunction with the -f
specfile options.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation