saving actual information in RPMs to remedy chaining limitations

Hi!

We at OpenPKG fight from a subtle problem in RPM which seems
to be quite
complex to describe.  OpenPKG focuses on building from SRPMs
and it also
predominantly uses static linking - two properties that seem
to be
unusual in the typical RPM application area. We have
identified that the
combination of both reveals a shortcoming of RPM which hits
us
frequently - but the problem is not exclusively tied to
static linking.

The short and general problem description is that during the
course of
building complex applications the chain of dependency
packages loosens
information important for later maintenance. We need to
carry some state
information from the build process of one package forward to
later
builds in order to make qualified decisions regarding
rebuild
requirements later. This is especially required when
resuming canceled
or retrying broken builds.

The OpenPKG practical detail description in example form is
to assume we
want to build an application (A) which requires a library
(L) which in
turn requires a sublibrary (S) and all three components have
their own
package.

    A.spec lists BuildPreReq: L
    L.spec lists BuildPreReq: S
    S.spec is standalone

Based on the entwined build (BuildPreReq) and install
(PreReq)
dependencies it is currently possible to create the
necessary process
sequence of

    build S
    install S
    build L
    install L
    build A
    install A

So far, so good. No problem for building and installing.
Note in the
OpenPKG static linking world this means that L embeds S and
A embeds L
including S. The final deployment (on another machine) would
require
installation of A only. The same situation applies every
time one
package sucks in information from another. This might be
arbitrary data
but is also true for API information, thus affecting RPM
environments
with shared libraries, too.

Now to the maintenance nightmare. Assume there is an
important change
within S, e.g. a major bug fix which requires an API change,
data being
updated for legal reasons or a security issue being fixed.
There is no
sign in A directly telling us that it is affected because
there is not
even a direct dependency in A listing S. Currently, we
circumvent this
problem by means of using external metadata, an index that
carries all
dependency information, thus allowing a maintenance
application to
traverse the dependency chain down from A through L to spot
S.  The
external index could be avoided if the requirements were
carried forward
during the build process to finally have S listed in A as
being the
provider of information or code embedded in A.

Worse, the nightmare continues and we have no solution to a
more
sophisticated case. It simply is not enough to list S inside
A as being
a provider of embedded information. Additionally, the
version-release and
probably more information of S must be listed inside A to
make qualified
decision whether A needs a rebuild or not. Again, assume a
problem has
been fixed in S.  This requires a rebuild of L to get the
updated S
information into L and this information must be saved in L.
Then the
updated L must trigger an update of A to get the updated L,
including
the updated S, information into A. Without saving the state
information
every broken build compromises the integrity of the whole
installation.
Assume S was successfully rebuilt and installed.  Without
saved state
information the decision that this updated S requires an
update of L
which in turn requires an update of A is highly volatile. If
the machine
crashes, runs out of disk space or simply is reset during
the rebuild of
L, the indirect update requirements are lost and the
dependency chain is
broken. If an update tool tries to resume without saved
state
information it will find an updated S and the world looks
fine. However,
both L and A still carry outdated information from the old S
inside ...

I believe the approach to remedy this chained requirements
problem is to
simply save the actual information at least for all
requirements. Well,
the implementation might not be so simple. My idea is to at
least save
the actual values of requirements, probably much more up the
whole
available header information, from required packages into a
genealogy
header of the currently build package. This must be done in
a recursive
way.

    build S
    install S
    build L,
        saving a L->S header with information about S
    install L
    build A
        saving a A->L header with information about L
including
        saving a L->S header with information about S
    install A

Having the actual values saved means that after updating S
at any time
and in a stateless way it is possible to decide which
packages require
rebuild although their own version number does not change
but just
because it is embedding outdated information. Remember, this
problem
hurts OpenPKG hard with statically linked libraries but
there are other
situations where similar problems occur and the same
solution applies.

I'd prefer to see a generic RPM solution. 

-- 
Thomas
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

Ah, perfect timing, I just rebuilt /bin/rpm with static
linking in
order to confirm the existence of ELF goo that could be
extracted from statically linked executables in order to
construct a dependency chain that will track static
linkages through rpm package dependencies.

You saved me much typing describing a rather subtle and
complex  
deficiency
in rpm dependencies.

So, without further ado (please study what Thomas wrote , here's
a  
possible
implementation. Perhaps someone can think of a better
implementation.

There are two problems that need solving to track static
linking through
rpm package dependencies AFAICS.

The first problem is how to extract Provides: and Requires:
from  
statically linked
executables and libraries. What I needed to confirm by
running
      readelf -a /bin/rpm
is what information is available from which a statically
linked  
dependency
chain might be constructed.

It turns out that the /bin/rpm executable contains ELF
entries like
	...
        231: 00000000     0 FILE    LOCAL  DEFAULT  ABS
depends.c
	...
while /usr/lib/librpm.a static libraries contain ELF entries
like
	...
	     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS
depends.c
	...
which can be extracted reliably from $RPM_BUILD_ROOT ELF
files.
(Note: I'll wave my hands for the moment about sorting out
duplicate
file names, statistically counting hits/misses should be
sufficient to
weed out accidental foo.c collisions, and there are almost
certainly
better than statistical bets to unambiguously associate a
static  
executable
with a library that contains an object file that was used in
linking.)

Assume (for now) that Requires(hint): rather than Requires:
is used
for static library linkages to be minimally disruptive to
existing  
package
collections.

The other problem has to do with build versus install
dependencies.

There's no reason a priori why build dependencies have
anything  
whatsoever
to do with install dependencies, building and installing are
quite  
different
operations, and one might expect very different dependency
relations.

I've always been surprised at how well build dependencies
have "worked"
in rpm (ignoring the fundamental design flaw with
BuildRequires:, is  
that it
is manually added to spec files, rather than automatically
detected).

What saves rpm build dependencies is that invariably the
build machine
is fundamentally an image of the install machine.

To focus on on Thomas's example:

>     A.spec lists BuildPreReq: L
>     L.spec lists BuildPreReq: S
>     S.spec is standalone

Note that A -> L through a build, not an install,
dependency, and
so the "BuildProvides:" (which already exist for
spec files iirc,
although there has never been a well defined need for
*.src.rpm's
to carry BuildProvides: elements, nothing whatsoever uses
atm).

Note: if you didn't follow my comment regarding
BuildProvides:, please
recall that build dependencies are represented identically
to install
dependencies, the only difference is that build dependencies
are
resident in *.src.rpm's, while Good Old rpm install
dependencies
are resident only in binary rpm's, the representation of
dependencies
is identcal in both source and binary rpm's.

So, AFAICT, the goal of easily knowing what *.src.rpms need
to
be rebuilt when a static library changes might reasonably be
threaded through BuildProvides: in *.src/rpm's. What has
been gained is that only src.rpm's would be needed to
do the lookup of what other packages need to be rebuilt
when static linkages are involved.

There are further issues, like the fact that relations are
directed from Requires -> Provides, the most similar
existing problem is the difficulty of discovering all
the binary rpm's built from the src.rpm given one of
the binary rpm's, and the additional issue that the
number of binary rpm's produced from a src.rpm
can be arch dependent, but there are means to
handle the directionality of dependency relations
outside of packaging.

Realistically, I suspect that populating BuildProvides:
in *.src.rpm's is likely unnecessary, as I expect that
even though A->L->S is threaded through
BuildRequires:,
I expect that the BuildRequires: to be mirrored into the
produced binary packaging Provides: in almost every case.

What's missing is that there is no attempt yet by rpm to
represent
static library linkage explicitly in package dependencies.

Does any of the above make sense? Or have I lost everyone
... 

hth

73 de Jeff



On Nov 19, 2006, at 5:40 PM, Thomas Lotterer wrote:

> Hi!
>
> We at OpenPKG fight from a subtle problem in RPM which
seems to be  
> quite
> complex to describe.  OpenPKG focuses on building from
SRPMs and it  
> also
> predominantly uses static linking - two properties that
seem to be
> unusual in the typical RPM application area. We have
identified  
> that the
> combination of both reveals a shortcoming of RPM which
hits us
> frequently - but the problem is not exclusively tied to
static  
> linking.
>
> The short and general problem description is that
during the course of
> building complex applications the chain of dependency
packages loosens
> information important for later maintenance. We need to
carry some  
> state
> information from the build process of one package
forward to later
> builds in order to make qualified decisions regarding
rebuild
> requirements later. This is especially required when
resuming canceled
> or retrying broken builds.
>
> The OpenPKG practical detail description in example
form is to  
> assume we
> want to build an application (A) which requires a
library (L) which in
> turn requires a sublibrary (S) and all three components
have their own
> package.
>
>     A.spec lists BuildPreReq: L
>     L.spec lists BuildPreReq: S
>     S.spec is standalone
>
> Based on the entwined build (BuildPreReq) and install
(PreReq)
> dependencies it is currently possible to create the
necessary process
> sequence of
>
>     build S
>     install S
>     build L
>     install L
>     build A
>     install A
>
> So far, so good. No problem for building and
installing. Note in the
> OpenPKG static linking world this means that L embeds S
and A embeds L
> including S. The final deployment (on another machine)
would require
> installation of A only. The same situation applies
every time one
> package sucks in information from another. This might
be arbitrary  
> data
> but is also true for API information, thus affecting
RPM environments
> with shared libraries, too.
>
> Now to the maintenance nightmare. Assume there is an
important change
> within S, e.g. a major bug fix which requires an API
change, data  
> being
> updated for legal reasons or a security issue being
fixed. There is no
> sign in A directly telling us that it is affected
because there is not
> even a direct dependency in A listing S. Currently, we
circumvent this
> problem by means of using external metadata, an index
that carries all
> dependency information, thus allowing a maintenance
application to
> traverse the dependency chain down from A through L to
spot S.  The
> external index could be avoided if the requirements
were carried  
> forward
> during the build process to finally have S listed in A
as being the
> provider of information or code embedded in A.
>
> Worse, the nightmare continues and we have no solution
to a more
> sophisticated case. It simply is not enough to list S
inside A as  
> being
> a provider of embedded information. Additionally, the
version- 
> release and
> probably more information of S must be listed inside A
to make  
> qualified
> decision whether A needs a rebuild or not. Again,
assume a problem has
> been fixed in S.  This requires a rebuild of L to get
the updated S
> information into L and this information must be saved
in L. Then the
> updated L must trigger an update of A to get the
updated L, including
> the updated S, information into A. Without saving the
state  
> information
> every broken build compromises the integrity of the
whole  
> installation.
> Assume S was successfully rebuilt and installed. 
Without saved state
> information the decision that this updated S requires
an update of L
> which in turn requires an update of A is highly
volatile. If the  
> machine
> crashes, runs out of disk space or simply is reset
during the  
> rebuild of
> L, the indirect update requirements are lost and the
dependency  
> chain is
> broken. If an update tool tries to resume without saved
state
> information it will find an updated S and the world
looks fine.  
> However,
> both L and A still carry outdated information from the
old S  
> inside ...
>
> I believe the approach to remedy this chained
requirements problem  
> is to
> simply save the actual information at least for all
requirements.  
> Well,
> the implementation might not be so simple. My idea is
to at least save
> the actual values of requirements, probably much more
up the whole
> available header information, from required packages
into a genealogy
> header of the currently build package. This must be
done in a  
> recursive
> way.
>
>     build S
>     install S
>     build L,
>         saving a L->S header with information about
S
>     install L
>     build A
>         saving a A->L header with information about
L including
>         saving a L->S header with information about
S
>     install A
>
> Having the actual values saved means that after
updating S at any time
> and in a stateless way it is possible to decide which
packages require
> rebuild although their own version number does not
change but just
> because it is embedding outdated information. Remember,
this problem
> hurts OpenPKG hard with statically linked libraries but
there are  
> other
> situations where similar problems occur and the same
solution applies.
>
> I'd prefer to see a generic RPM solution.
>
> -- 
> Thomas
> _______________________________________________
> Rpm-devel mailing list
> Rpm-devellists.dulug.duke.edu
> https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

On November 19, 2006 7:05:27 PM -0500 Jeff Johnson
<n3npq.jbjgmail.com> 
wrote:
> I've always been surprised at how well build
dependencies have "worked"
> in rpm (ignoring the fundamental design flaw with
BuildRequires:, is that
> it
> is manually added to spec files, rather than
automatically detected).

I don't see how you could possibly auto-detect them, other
than the
crude systrace method which isn't all that useful.

-frank
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

On 11/19/06, Frank Cusack <fcusackfcusack.com> wrote:
> On November 19, 2006 7:05:27 PM -0500 Jeff Johnson
<n3npq.jbjgmail.com>
> wrote:
> > I've always been surprised at how well build
dependencies have "worked"
> > in rpm (ignoring the fundamental design flaw with
BuildRequires:, is that
> > it
> > is manually added to spec files, rather than
automatically detected).
>
> I don't see how you could possibly auto-detect them,
other than the
> crude systrace method which isn't all that useful.
>

We agree that build dependencies work surprisingly well.

73 de Jeff
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

Jeff Johnson wrote:
> ...
> I've always been surprised at how well build
dependencies have "worked"
> in rpm (ignoring the fundamental design flaw with
BuildRequires:, is
> that it
> is manually added to spec files, rather than
automatically detected).
> 
> What saves rpm build dependencies is that invariably
the build machine
> is fundamentally an image of the install machine.

I'm often surprised as well at how well build dependencies
work.. but
one note..  in the case of Cross-Compiled embedded OSes
(i.e. Wind River
Linux and MontaVista Linux) the build machine often looks
nothing like
the resulting system.  Both in CPU type, and in some cases
it's not even
the same OS.  (i.e. Solaris or Cygwin building Linux
packages.)

All of the build systems I have worked on in the past
generally
alphabetically order the build, and then BuildRequires (or
the
equivalent) are used to filter things "to the top"
of the list.  Effect
is generally you only need a minimal set of build
dependencies, however
you end up with an implicit alphabetic ordering which hides
many
dependencies.

> To focus on on Thomas's example:
> 
>>     A.spec lists BuildPreReq: L
>>     L.spec lists BuildPreReq: S
>>     S.spec is standalone
> 
> ...
> What's missing is that there is no attempt yet by rpm
to represent
> static library linkage explicitly in package
dependencies.
> 
> Does any of the above make sense? Or have I lost
everyone ... 

Makes sense to me.  I've written scripts before to attempt
to tackle
this, but as you and Thomas noted.. it's not an easy task to
identify
all of the static libraries that go into building a package.

BuildRequires alone aren't enough either, since many times a
build
requirement will be something like "awk" which is
needed to run a script
that generates code.

This is certainly an interesting problem.  One of the not so
nice tricks
I've done in the past is make a small change to something
which I
suspect affects other things.. do a full system rebuild..
and then do a
diff of the text sections of all of the executables.  That
then returns
back a list of everything that changed "because
of" the other thing
changing..  (so far only one bug update has required that..
but it's a
nasty nasty thing to have to do.)

--Mark

_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

On 11/20/06, Mark Hatle <mark.hatlewindriver.com> wrote:
> Jeff Johnson wrote:
> > ...
> > I've always been surprised at how well build
dependencies have "worked"
> > in rpm (ignoring the fundamental design flaw with
BuildRequires:, is
> > that it
> > is manually added to spec files, rather than
automatically detected).
> >
> > What saves rpm build dependencies is that
invariably the build machine
> > is fundamentally an image of the install machine.
>
> I'm often surprised as well at how well build
dependencies work.. but
> one note..  in the case of Cross-Compiled embedded OSes
(i.e. Wind River
> Linux and MontaVista Linux) the build machine often
looks nothing like
> the resulting system.  Both in CPU type, and in some
cases it's not even
> the same OS.  (i.e. Solaris or Cygwin building Linux
packages.)
>

We agree that -- even a cross build environment -- ELF is
fundamentally ELF
for both the build and install machines.

> All of the build systems I have worked on in the past
generally
> alphabetically order the build, and then BuildRequires
(or the
> equivalent) are used to filter things "to the
top" of the list.  Effect
> is generally you only need a minimal set of build
dependencies, however
> you end up with an implicit alphabetic ordering which
hides many
> dependencies.
>

Yep. Missing dependencies are inevitable when build
dependencies
are solely manually generated.

FWIW, I can tell you how to script a QA process to make
build
dependencies reliable using --aid if necessary. I suspect
you know
how to do that already however.

> > To focus on on Thomas's example:
> >
> >>     A.spec lists BuildPreReq: L
> >>     L.spec lists BuildPreReq: S
> >>     S.spec is standalone
> >
> > ...
> > What's missing is that there is no attempt yet by
rpm to represent
> > static library linkage explicitly in package
dependencies.
> >
> > Does any of the above make sense? Or have I lost
everyone ... 
>
> Makes sense to me.  I've written scripts before to
attempt to tackle
> this, but as you and Thomas noted.. it's not an easy
task to identify
> all of the static libraries that go into building a
package.
>
> BuildRequires alone aren't enough either, since many
times a build
> requirement will be something like "awk"
which is needed to run a script
> that generates code.
>

Yep, a missing dependency on, say, awk, was not manually
generated so a build
breaks.

> This is certainly an interesting problem.  One of the
not so nice tricks
> I've done in the past is make a small change to
something which I
> suspect affects other things.. do a full system
rebuild.. and then do a
> diff of the text sections of all of the executables. 
That then returns
> back a list of everything that changed "because
of" the other thing
> changing..  (so far only one bug update has required
that.. but it's a
> nasty nasty thing to have to do.)
>

I suspect that you might be interested in an rpmdb-like
database
where instead of *.rpm headers, ELF headers are stored. See
the
rather terse and crack soaked proposal in the
symclash.{py,sh} thread.

The implememntation I'm thinking of would work with sqlite3
too 

73 de Jeff
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

n3npq.jbjgmail.com wrote on 2006-11-20 01:05:
> Ah, perfect timing, I just rebuilt /bin/rpm with static
linking in
> order to confirm the existence of ELF goo that could be
extracted from
> statically linked executables in order to construct a
dependency chain
> [...]
>
Sounds to me we are talking about two different issues here.

One is the general challenge to teach RPM carrying arbitrary
information
from the build process into the immediate resulting binary
and then
further to other derived binaries and into the database.

The other is the the special case of static library linkange
information
being exactly the kind information that needs to be carried
forward and
the approach to find this special information in an
automated way.

> There are two problems that need solving to track
static linking
> through rpm package dependencies AFAICS.
> 
> The first problem is how to extract Provides: and
Requires: from
> statically linked executables and libraries. [...]
readelf -a /bin/rpm
> 
I was in fact once trying to inject packaging information
into static
libraries in the past to ease forensic analysis of code
genealogy and to
ease debugging and troubleshooting. The problems I was
facing were that
many optimizations exists today in the compiler tool chain
which cause
technically unnecessary information to be omitted in the
executable -
imagine the effect of strip(1). The ultimate killer for my
approach was
the fact that not all platforms we at OpenPKG support are
using ELF
which further multiplies the scope of the problem.

> The other problem has to do with build versus install
dependencies.
> 
> There's no reason a priori why build dependencies have
anything
> whatsoever to do with install dependencies, building
and installing
> are quite  different operations, and one might expect
very different
> dependency relations.
> 
What I was referring to with "entwinded build and
install dependencies"
is that fact that any automated build+install mechanism must
obey both
build and install dependencies in a recursive way. I'm
talking about
tools working on top of rpm. It is obvious that an
application that
requires an extra packaged static library as a BuildPreReq
requires that
library package to be build *and* installed before the
application
build.  However, the final production machine must only obey
the
(Install)PreReq of the application which may well omit the
static
library package.

My A->L->S example in this thread kick-off was
actually very simple and
straightforward. In practice things quickly become much more
complex if
you image that a typical application has multiple
BuildPreReqs, and each
of them has another multiple BuildPreReqs and so on. This
sounds like a
tree of dependencies but in fact it becomes an ordered graph
when
dependencies have interrelationships to each other. Handling
that in an
efficient way like identifying cases which allow parallel
builds or
orders which are "better" in some way over others
is worth another
thread. I'm glad to find or create an file a more
sophisticated example.

>>     A.spec lists BuildPreReq: L
>>     L.spec lists BuildPreReq: S
>>     S.spec is standalone
> 
> Note that A -> L through a build, not an install,
dependency, and
> so the "BuildProvides:" (which already exist
for spec files iirc,
> although there has never been a well defined need for
*.src.rpm's
> to carry BuildProvides: elements, nothing whatsoever
uses atm).
> 
Well, neither BuildProvides nor BuildPreReqs are carried
forward to
derived packages, they only determine immediate relations
like A->L or
L->S, not indirect relations like A->S (through L).

> Note: if you didn't follow my comment regarding
BuildProvides:, please
> recall that build dependencies are represented
identically to install
> dependencies, the only difference is that build
dependencies are
> resident in *.src.rpm's, while Good Old rpm install
dependencies are
> resident only in binary rpm's, the representation of
dependencies is
> identcal in both source and binary rpm's.
> 
That's exactly one of the limitations I see. BuildPreReq
information is
lost in a derived binary.

> So, AFAICT, the goal of easily knowing what *.src.rpms
need to be
> rebuilt when a static library changes might reasonably
be threaded
> through BuildProvides: in *.src/rpm's. What has been
gained is that
> only src.rpm's would be needed to do the lookup of what
other packages
> need to be rebuilt when static linkages are involved.
> 
Sounds good but my final goal is to have that information
available in
the RPM database in order to neither require the binary RPMs
(which is
already the case) nor the source RPMs.

> There are further issues, like the fact that relations
are directed
> from Requires -> Provides, the most similar existing
problem is the
> difficulty of discovering all the binary rpm's built
from the src.rpm
> given one of the binary rpm's, [...]
> 
This is impossible because if source RPM S results in two
binaries B1
and B2 one of the two binaries appears sooner on earth and
cannot list
its future sibling. However, an external database that
tracks S, B1 and
B2 can easily tell with information available today because
both
binaries list S as their "parent", both in
name-version-release and a
unique ID.

> Realistically, I suspect that populating BuildProvides:
in *.src.rpm's
> is likely unnecessary, as I expect that even though
A->L->S is
> threaded through BuildRequires:, I expect that the
BuildRequires: to
> be mirrored into the produced binary packaging
Provides: in almost
> every case.
> 
Oh. Uh. I'm not sure what you are talking about here.
If you assume that the output of

 $ rpm -qp --qf '[%n]' foo.src.rpm

should somehow be "mirrored" into the output of

 $ rpm -qp --qf '[%n]' foo.rpm

then this is not true for my environment. Is this opposed to
work?
 

-- 
Thomas
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

Now to the general challenge teaching RPM to carry arbitrary
information
from the build process into the immediate resulting binary
and then
further to other derived binaries and into the database.

I'm not only talking about absolute technical requirements.
Some
information from the build process is also or only useful
for
troubleshooting, debugging and code genealogy forensics.

My idea along with implementation perception is to add a
"Genealogy"
header to the binary RPM and the database which basically
contains a
whole data vault carrying

1.) arbitrary user data (picked off a well known place)
2.) the contents of the whole rpm database (yes, really)
3.) the spec file in its pristine format
4.) the spec file in its expanded format
5.) build log (stdout and stderr)
... 2-5 are already special cases of 1

This might sound extreme and it probably is. But it should
be easy to
implement. It is fully backwards compatible and allows
maximum
flexibility.  It is even a candidate for a run-time option
in order not
to hurt existing installations. I know the RPMs and the
database might
grow noticeable but I don't believe they grow unacceptable
for today's
computing resources. This header is read only and can be
compressed
efficiently. Please note that the whole thing is recursive!
Any binary
package carries information from it's build environment and
from the
environment of it's dependencies (and even unrelated
packages installed
incidentally) with it.  This new functionality is easy to
use, because
the new header may contain a whole database, existing code
within or
outside rpm can be used to access that information with
little
modifications.

Think of my A->L->S example and guess what I mean with
this:

 $ rpm -qi A
 $ rpm -qi A/L
 $ rpm -qi A/L/S

or this

 $ rpm -q  --qf '[%n]' A/L
 $ rpm -qp --qf '[%n]' A.rpm/L/S

Call me a day dreamer  

-- 
Thomas
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

Your previous message reminded me of something I had
experimented with
in the past.

Specifically modifing gcc (or the linker) to add an
additional .note (or
similar) section to the binaries that put in a
"genealogy" tag.  Then
when static binaries are linked together, you can use simple
ELF tools
to scan for this tag and that information is much easier to
embed into
RPM.  (Note I'm only talking about ELF binaries.. other
binary types may
not be able to do this.)

There is a secondary situation which was brought up before,
include
files have no real tag that gets embedded into the compiled
binaries.
You may be able to play some games and get a string embedded
into the
compiled binary from each header, but that requires a LOT of
potential
changes.

(I looked into this pedigree work as part of a license
interaction/validation experiment...  it worked in
principle, but never
went beyond the prototype phase...)

....just food for thought at this point....

--Mark

Thomas Lotterer wrote:
> Now to the general challenge teaching RPM to carry
arbitrary information
> from the build process into the immediate resulting
binary and then
> further to other derived binaries and into the
database.
> 
> I'm not only talking about absolute technical
requirements. Some
> information from the build process is also or only
useful for
> troubleshooting, debugging and code genealogy
forensics.
> 
> My idea along with implementation perception is to add
a "Genealogy"
> header to the binary RPM and the database which
basically contains a
> whole data vault carrying
> 
> 1.) arbitrary user data (picked off a well known place)
> 2.) the contents of the whole rpm database (yes,
really)
> 3.) the spec file in its pristine format
> 4.) the spec file in its expanded format
> 5.) build log (stdout and stderr)
> ... 2-5 are already special cases of 1
> 
> This might sound extreme and it probably is. But it
should be easy to
> implement. It is fully backwards compatible and allows
maximum
> flexibility.  It is even a candidate for a run-time
option in order not
> to hurt existing installations. I know the RPMs and the
database might
> grow noticeable but I don't believe they grow
unacceptable for today's
> computing resources. This header is read only and can
be compressed
> efficiently. Please note that the whole thing is
recursive! Any binary
> package carries information from it's build environment
and from the
> environment of it's dependencies (and even unrelated
packages installed
> incidentally) with it.  This new functionality is easy
to use, because
> the new header may contain a whole database, existing
code within or
> outside rpm can be used to access that information with
little
> modifications.
> 
> Think of my A->L->S example and guess what I mean
with this:
> 
>  $ rpm -qi A
>  $ rpm -qi A/L
>  $ rpm -qi A/L/S
> 
> or this
> 
>  $ rpm -q  --qf '[%n]' A/L
>  $ rpm -qp --qf '[%n]' A.rpm/L/S
> 
> Call me a day dreamer  
> 

_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel

On Sunday, 19 November 2006, at 19:05:27 (-0500),
Jeff Johnson wrote:

> Ah, perfect timing, I just rebuilt /bin/rpm with static
linking in
> order to confirm the existence of ELF goo that could be
> extracted from statically linked executables in order
to
> construct a dependency chain that will track static
> linkages through rpm package dependencies.
> 
> You saved me much typing describing a rather subtle and
complex
> deficiency in rpm dependencies.
> 
> So, without further ado (please study what Thomas wrote
,
here's
> a possible implementation. Perhaps someone can think of
a better
> implementation.

In my mind, the question here is not one of capability but
of intent.
Generally (i.e., on a typical UNIX/Linux system) if a
particular
package is linked statically against a particular library,
there's a
reason for that, and that reason may directly conflict with
the desire
to track dependencies.

For example, if I statically link against a particular
libneon or
libxml due to API changes, I inherently don't want to
upgrade to
future versions.

So tracking static dependencies seems to be a rather niche
"market,"
so to speak, that should come from patches provided by those
individuals who need that functionality.

Or am I missing something?

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/ 
<mejkainx.org>
n + 1, Inc., http://www.nplus1.net/    
  Author, Eterm (www.eterm.org)
------------------------------------------------------------
-----------
 "You like apples?  Yeah?  Well, I got her number!  How
about them
  apples??!"                        -- Matt Damon,
"Good Will Hunting"
_______________________________________________
Rpm-devel mailing list
Rpm-devellists.dulug.duke.edu
https://lists.dulug.duke.edu/mailman/listinfo/rpm-devel