tree dependency check

So after manifest2 is in, I'll revive the other issue that
IMO is a 
requirement for 2.1: enforcing dependencies needed to use
the tree (see 
old threads or glep44 for reasoning). A patch for that is
available at 
dev.gentoo.org/~genone/patches/treedeps.diff. Unless
somebody objects 
I'll add that somewhere next week.

Marius
-- 
gentoo-portage-devgentoo.org mailing list

Marius Mauch schrieb:
> So after manifest2 is in, I'll revive the other issue
that IMO is a 
> requirement for 2.1: enforcing dependencies needed to
use the tree (see 
> old threads or glep44 for reasoning). A patch for that
is available at 
> dev.gentoo.org/~genone/patches/treedeps.diff. Unless
somebody objects 
> I'll add that somewhere next week.

Ok, from a discussion with Zac and a few others it seems
that we should 
add a layer of indirection, so instead of specifying atoms
the tree has 
a freeform version identifier, and portage can use it to
look up the 
required atoms by using a (remote) mapping file.
This new approach is implemented in 
dev.gentoo.org/~genone/pacthes/format-check.diff (it's
basically a 
combination of strategies D and E in
d.g.o/~genone/docs/treedeps.txt)

This new patch also has a few other improvements:
- checks overlays too
- at sync time first check the tree format of the remote
tree, if we 
can't handle it the sync is aborted
- if no format specification is found warn the user about it
(with the 
intention to fail in a future version)

New intended merge date is this weekend or beginning of next
week, so if 
you have any objections, make them now.

Marius
-- 
gentoo-portage-devgentoo.org mailing list

On Wed, Mar 29, 2006 at 07:21:15PM +0300, Marius Mauch
wrote:
> Marius Mauch schrieb:
> >So after manifest2 is in, I'll revive the other
issue that IMO is a 
> >requirement for 2.1: enforcing dependencies needed
to use the tree (see 
> >old threads or glep44 for reasoning). A patch for
that is available at 
> >dev.gentoo.org/~genone/patches/treedeps.diff.
Unless somebody objects 
> >I'll add that somewhere next week.
> 
> Ok, from a discussion with Zac and a few others it
seems that we should 
> add a layer of indirection, so instead of specifying
atoms the tree has 
> a freeform version identifier, and portage can use it
to look up the 
> required atoms by using a (remote) mapping file.
> This new approach is implemented in 
> dev.gentoo.org/~genone/pacthes/format-check.diff (it's
basically a 
dev.gentoo.org/~genone/patches/format-check.diff moreso...
~harring 

On 3/29/06, Marius Mauch <genonegentoo.org> wrote:
> Marius Mauch schrieb:
> > So after manifest2 is in, I'll revive the other
issue that IMO is a
> > requirement for 2.1: enforcing dependencies needed
to use the tree (see
> > old threads or glep44 for reasoning). A patch for
that is available at
> > dev.gentoo.org/~genone/patches/treedeps.diff.
Unless somebody objects
> > I'll add that somewhere next week.
>
> Ok, from a discussion with Zac and a few others it
seems that we should
> add a layer of indirection, so instead of specifying
atoms the tree has
> a freeform version identifier, and portage can use it
to look up the
> required atoms by using a (remote) mapping file.
> This new approach is implemented in
> dev.gentoo.org/~genone/pacthes/format-check.diff (it's
basically a
> combination of strategies D and E in
d.g.o/~genone/docs/treedeps.txt)
>
> This new patch also has a few other improvements:
> - checks overlays too
> - at sync time first check the tree format of the
remote tree, if we
> can't handle it the sync is aborted
> - if no format specification is found warn the user
about it (with the
> intention to fail in a future version)
>
> New intended merge date is this weekend or beginning of
next week, so if
> you have any objections, make them now.

In doing compatibility code, should wait a bit to make sure
it's sane
and that there are no issues _prior_ to deployment- rushing
it (3 days
notice) kind of conflicts with that requirement 

Mind you this is from looking at the code (not running it),
so verify
it, but here goes...

Implementation issues...

1) format_check should just return the results (supported,
not
supported, need xyz) instead of printing to the term. 
Remember this
code may be triggered by gui apps.  This also means it
shouldn't be
abused to check what's been handed in via commandline to
know what to
exempt, should just return what is required.

2) code isn't root aware.

3) no snapshot integration.

4) code invalidly assumes that all later version of the
pulled atom
from vdb will work.  Format version 1 may not be supported
by portage
2 years down the line, this code assumes all portage
versions from
that point on support that format- that is a recipe for
bloat/borkage;
this is a bit odd of a check/attempt considering the url
crap to try
and get the appropriate atom.

5) Introduced (in the urlopen call) another
KeyboardInterrupt
swallowing except clause.  Don't Do That. 

6) This breaks _all_ syncing for users who have overlays but
lack the
format versions file.  That's a massive no go, you don't
break
compatibility introducing compatibility checks (nor do you
piss off
several thousand overlay using users for questionable gain).

7) even cooler, say you're running max visible portage, and
using an
overlay that lacks a format_version file.  With the vdb
portage
lookup, it'll tell the user that they need a version later
then the
max version.  Nice way to get people to test package masked
portage's,
but it still is wrong.

8) (minor) output of todo is going to be fugly if anyone
uses actual
atom constructs, boolean ORs fex (the print implicitly
assumes it's
just a list of atoms without any boolean constructs).

9) the attempted check to see if a pkg is in the passed in
myfiles
won't work if myfiles holds atoms; eg,
myfiles, pkg = [">=sys-apps/portage-2.0.54"],
"sys-apps/portage"
assert pkg not in myfiles

Perceived design flaws...

1) pollution of the rootdir of PORTDIR.  To head of the
"it's
subjective", aside from this repo id will need to be
pushed into the
tree format also, and any files the tree format version
requires
*should* be in a subdir.  so... basically this single file
is going to
be sitting in rootdir while all other repository metadata is
stuck in
a subdir.  Seems rather daft to have it shoved in it's own
location-
compatibility code _does_ have certain hardcoded assumptions
for
things to work, hardcoding looking in a subdir is no
different then
hardcoding no subdir (only difference is the subdir keeps
things
contained/clean).

2) It's overengineered.  There is _no_ reason to hit up a
webserver
just to get atoms; that data can be bundled in the tree in a
seperate
file.  As is, this breaks users who sync without a
connection without
any gain.  Realistically, I'd be surprised if any alt
package managers
go this route (I know I won't be hitting a webserver up for
pkgcore).

3) What the portage project thinks a repo tree needs does
not map to
what my tree may need.  Clarifying, format 1 specifies
portage xyz and
bash-3 (ebuilds in the tree use bash regex).  My personal
tree needs
portage xyz (manifest/layout changes), but requires just
bash-2.  With
the central db approach, portage will assume my tree is
valid via the
version #, and if the number differs, it'll assume that I
require
bash3 when in reality, my tree is bash-2 and up.  This
points to why
the format -> depends mapping should be bundled with the
tree.

4) portage-2.0 portage-2.1 are bad names for tree formats. 
Back to
the EAPI int discussion, but I really don't see any gain in
string
names, regardless the 2.0/2.1 sucks as a descriptive name
(subjective
I realize) 

Short version- this isn't ready for prime time and
(personally) I
think it needs to be rethought.  At the very least it needs
to _not_
cause user visible uglyness on enabling the compatibility
checks (need
a solution for transitioning from unversioned repos to
versioned
repos).  Requiring folks to jam a file with a random string
in it in
each of their overlays also sucks, ways to make that less
sucky would
be advisable.

Not opposed to the concept, but think this needs a fair
amount of use
case planning- the bash-3 example folks *will* try jamming
in the bash
dep fex (spyderous already wants the go ahead for using
regex), and
while the intention seems to be versioning the tree format,
the bash
requirement for the tree is also there.

So... -1.
~harring

-- 
gentoo-portage-devgentoo.org mailing list

On Thursday 30 March 2006 01:21, Marius Mauch wrote:
> Marius Mauch schrieb:
> > So after manifest2 is in, I'll revive the other
issue that IMO is a 
> > requirement for 2.1: enforcing dependencies needed
to use the tree (see 
> > old threads or glep44 for reasoning).

Can you summarise the reasoning again please?

> - at sync time first check the tree format of the
remote tree, if we 
> can't handle it the sync is aborted

While I haven't checked over the code at all, the point
above above sounds
problematic. What happens if the a user installs the latest
with this patch
and then doesn't sync for three years? The tree may then
require a later
portage but the user can't sync to be able to emerge a
later portage.

--
Jason Stubbs
-- 
gentoo-portage-devgentoo.org mailing list

On Wed, 29 Mar 2006 14:42:25 -0800
"Brian Harring" <ferringbgmail.com> wrote:

> 2) code isn't root aware.

If root == $ROOT, then I don't see how it would affect
this. Even with
$ROOT != / we're using the tools in / for most things,
$ROOT is just
for deployment.
If root means something else here please clarify.

> 3) no snapshot integration.

Wanted to wait for feedback first before adjusting all
tools.

> 4) code invalidly assumes that all later version of the
pulled atom
> from vdb will work.

That assumption is only a fallback if not atom mapping for a
format can 
be found. You have a better fallback strategy?

> 6) This breaks _all_ syncing for users who have
overlays but lack the
> format versions file.  That's a massive no go, you
don't break
> compatibility introducing compatibility checks (nor do
you piss off
> several thousand overlay using users for questionable
gain).

Reread the code, missing format_version raises a warning,
nothing else
(for now).

> 7) even cooler, say you're running max visible
portage, and using an
> overlay that lacks a format_version file.  With the vdb
portage
> lookup, it'll tell the user that they need a version
later then the
> max version.  Nice way to get people to test package
masked portage's,
> but it still is wrong.

see above.

> 8) (minor) output of todo is going to be fugly if
anyone uses actual
> atom constructs, boolean ORs fex (the print implicitly
assumes it's
> just a list of atoms without any boolean constructs).

Yeah, if anyone want to take a shot to convert it to feed
the list into
the depresolver feel free.

> 9) the attempted check to see if a pkg is in the passed
in myfiles
> won't work if myfiles holds atoms; eg,
> myfiles, pkg =
[">=sys-apps/portage-2.0.54"],
"sys-apps/portage"
> assert pkg not in myfiles

reread the code, we're only comparing the keys (still not
perfect, but
covers your case).

> 2) It's overengineered.  There is _no_ reason to hit
up a webserver
> just to get atoms; that data can be bundled in the tree
in a seperate
> file.  As is, this breaks users who sync without a
connection without
> any gain.  Realistically, I'd be surprised if any alt
package managers
> go this route (I know I won't be hitting a webserver
up for pkgcore).

a) it's a fallback if the system and tree don't have a
depmap
b) it doesn't break
c) it doesn't matter what other package managers will do

> 3) What the portage project thinks a repo tree needs
does not map to
> what my tree may need.  Clarifying, format 1 specifies
portage xyz and
> bash-3 (ebuilds in the tree use bash regex).  My
personal tree needs
> portage xyz (manifest/layout changes), but requires
just bash-2.  With
> the central db approach, portage will assume my tree is
valid via the
> version #, and if the number differs, it'll assume
that I require
> bash3 when in reality, my tree is bash-2 and up.  This
points to why
> the format -> depends mapping should be bundled with
the tree.

See above.

> 4) portage-2.0 portage-2.1 are bad names for tree
formats.  Back to
> the EAPI int discussion, but I really don't see any
gain in string
> names, regardless the 2.0/2.1 sucks as a descriptive
name (subjective
> I realize) 

Well, in the end you need a format->feature map, and
these names simply
have that mapping implicit in a way.

> Requiring folks to jam a file with a random string in
it in
> each of their overlays also sucks, ways to make that
less sucky would
> be advisable.

s/Requiring/Recommending/
Ideally we could detect if an overlay is local or from a
third party,
and only perform checks for the latter, but that's not
doable atm.
Zac: The error-on-missing was your idea, feel free to defend
it.

Marius
-- 
gentoo-portage-devgentoo.org mailing list

On Wed, 29 Mar 2006 14:42:25 -0800
"Brian Harring" <ferringbgmail.com> wrote:

> 2) code isn't root aware.

If root == $ROOT, then I don't see how it would affect
this. Even with
$ROOT != / we're using the tools in / for most things,
$ROOT is just
for deployment.
If root means something else here please clarify.

> 3) no snapshot integration.

Wanted to wait for feedback first before adjusting all
tools.

> 4) code invalidly assumes that all later version of the
pulled atom
> from vdb will work.

That assumption is only a fallback if not atom mapping for a
format can 
be found. You have a better fallback strategy?

> 6) This breaks _all_ syncing for users who have
overlays but lack the
> format versions file.  That's a massive no go, you
don't break
> compatibility introducing compatibility checks (nor do
you piss off
> several thousand overlay using users for questionable
gain).

Reread the code, missing format_version raises a warning,
nothing else
(for now).

> 7) even cooler, say you're running max visible
portage, and using an
> overlay that lacks a format_version file.  With the vdb
portage
> lookup, it'll tell the user that they need a version
later then the
> max version.  Nice way to get people to test package
masked portage's,
> but it still is wrong.

see above.

> 8) (minor) output of todo is going to be fugly if
anyone uses actual
> atom constructs, boolean ORs fex (the print implicitly
assumes it's
> just a list of atoms without any boolean constructs).

Yeah, if anyone want to take a shot to convert it to feed
the list into
the depresolver feel free.

> 9) the attempted check to see if a pkg is in the passed
in myfiles
> won't work if myfiles holds atoms; eg,
> myfiles, pkg =
[">=sys-apps/portage-2.0.54"],
"sys-apps/portage"
> assert pkg not in myfiles

reread the code, we're only comparing the keys (still not
perfect, but
covers your case).

> 2) It's overengineered.  There is _no_ reason to hit
up a webserver
> just to get atoms; that data can be bundled in the tree
in a seperate
> file.  As is, this breaks users who sync without a
connection without
> any gain.  Realistically, I'd be surprised if any alt
package managers
> go this route (I know I won't be hitting a webserver
up for pkgcore).

a) it's a fallback if the system and tree don't have a
depmap
b) it doesn't break
c) it doesn't matter what other package managers will do

> 3) What the portage project thinks a repo tree needs
does not map to
> what my tree may need.  Clarifying, format 1 specifies
portage xyz and
> bash-3 (ebuilds in the tree use bash regex).  My
personal tree needs
> portage xyz (manifest/layout changes), but requires
just bash-2.  With
> the central db approach, portage will assume my tree is
valid via the
> version #, and if the number differs, it'll assume
that I require
> bash3 when in reality, my tree is bash-2 and up.  This
points to why
> the format -> depends mapping should be bundled with
the tree.

See above.

> 4) portage-2.0 portage-2.1 are bad names for tree
formats.  Back to
> the EAPI int discussion, but I really don't see any
gain in string
> names, regardless the 2.0/2.1 sucks as a descriptive
name (subjective
> I realize) 

Well, in the end you need a format->feature map, and
these names simply
have that mapping implicit in a way.

> Requiring folks to jam a file with a random string in
it in
> each of their overlays also sucks, ways to make that
less sucky would
> be advisable.

s/Requiring/Recommending/
Ideally we could detect if an overlay is local or from a
third party,
and only perform checks for the latter, but that's not
doable atm.
Zac: The error-on-missing was your idea, feel free to defend
it.

Marius
-- 
gentoo-portage-devgentoo.org mailing list

On Thu, 30 Mar 2006 08:30:17 +0900
Jason Stubbs <jstubbsgentoo.org> wrote:

> On Thursday 30 March 2006 01:21, Marius Mauch wrote:
> > Marius Mauch schrieb:
> > > So after manifest2 is in, I'll revive the
other issue that IMO is
> > > a requirement for 2.1: enforcing dependencies
needed to use the
> > > tree (see old threads or glep44 for
reasoning).
> 
> Can you summarise the reasoning again please?

a) avoid massive breakage when certain new features are
introduced
(past examples being cascading profiles or new-style
virtuals)
b) similar to a) allow people to use new features without
having to
wait for a year or two

> > - at sync time first check the tree format of the
remote tree, if
> > we can't handle it the sync is aborted
> 
> While I haven't checked over the code at all, the
point above above
> sounds problematic. What happens if the a user installs
the latest
> with this patch and then doesn't sync for three years?
The tree may
> then require a later portage but the user can't sync
to be able to
> emerge a later portage.

!!! This upgrade might fail, in that case please read the
!!! following documents for help:
!!! http://www.gentoo.org/proj/en/portage/doc/common-pr
oblems.xml
!!!
http://www.gentoo.org/proj/en/portage/doc/m
anually-fixing-portage.xml

These docs will then tell the user about the following
options
a) use the secret override
b) manually update the tree with wget+tar
c) perform an manual update (as in wget
portage-version.tar.bz2, ...)

That should be an extreme exception though, there will be
some rules
regarding format bumps (like "only use formats where
that haven been
supported by stable portage for n months").

Marius
-- 
gentoo-portage-devgentoo.org mailing list

On 3/29/06, Marius Mauch <genonegentoo.org> wrote:
> On Wed, 29 Mar 2006 14:42:25 -0800
> "Brian Harring" <ferringbgmail.com> wrote:
>
> > 2) code isn't root aware.
>
> If root == $ROOT, then I don't see how it would affect
this. Even with
> $ROOT != / we're using the tools in / for most things,
$ROOT is just
> for deployment.

If ROOT!="/", portage installed at ROOT=/ isn't
capable of reading the
tree, the check will invalidly assume the user is trying to
update
installed portage at ROOT=/ when in reality it's trying to
update
ROOT!="/" .

ROOT=/ should be updated prior to fooling with ROOT!=/ since
as you
said, it's just the offset the files get installed to.

And yes, my initial description sucked 


> > 4) code invalidly assumes that all later version
of the pulled atom
> > from vdb will work.
>
> That assumption is only a fallback if not atom mapping
for a format can
> be found. You have a better fallback strategy?

You've got 3 fallbacks already, plus this.  Frankly, you
can make it
only so smart- the simpler the code is, the less likely it
gets
screwed up (thus making compatibility an issue since you
have to work
around broken compatibility checks).

Telling folks to upgrade to latest portage version when the
latest may
not even support that tree format still is bad advice for a
user.


> > 6) This breaks _all_ syncing for users who have
overlays but lack the
> > format versions file.  That's a massive no go,
you don't break
> > compatibility introducing compatibility checks
(nor do you piss off
> > several thousand overlay using users for
questionable gain).
>
> Reread the code, missing format_version raises a
warning, nothing else
> (for now).

Bleh, that's what the hell that split was for...
That's a bit fugly, and is going to be extremely obnoxious
for
consumers of it (warning for each overlay isn't purdy).


> > 7) even cooler, say you're running max visible
portage, and using an
> > overlay that lacks a format_version file.  With
the vdb portage
> > lookup, it'll tell the user that they need a
version later then the
> > max version.  Nice way to get people to test
package masked portage's,
> > but it still is wrong.
>
> see above.

This case still exists; assume format_version exists and
it's unknown
to any of the 3 (!?) mappings, and that it isn't ''-
still is
misleading to users.


> > 8) (minor) output of todo is going to be fugly if
anyone uses actual
> > atom constructs, boolean ORs fex (the print
implicitly assumes it's
> > just a list of atoms without any boolean
constructs).
>
> Yeah, if anyone want to take a shot to convert it to
feed the list into
> the depresolver feel free.

Comment was in regards to the output; don't have to resolve
it to
output it properly.


> > 9) the attempted check to see if a pkg is in the
passed in myfiles
> > won't work if myfiles holds atoms; eg,
> > myfiles, pkg =
[">=sys-apps/portage-2.0.54"],
"sys-apps/portage"
> > assert pkg not in myfiles
>
> reread the code, we're only comparing the keys (still
not perfect, but
> covers your case).

Yeah, noticed that one shortly after opening the mouth
('bout the norm).

>
> > 2) It's overengineered.  There is _no_ reason to
hit up a webserver
> > just to get atoms; that data can be bundled in the
tree in a seperate
> > file.  As is, this breaks users who sync without a
connection without
> > any gain.  Realistically, I'd be surprised if any
alt package managers
> > go this route (I know I won't be hitting a
webserver up for pkgcore).
>
> a) it's a fallback if the system and tree don't have
a depmap
> b) it doesn't break

So it's a fallback (vdb lookup) with a fallback (uri
lookup) with a
fallback (bundled mapping) with a default (tree), right?

Sounds overengineered to me ;)

Realistically, the cases where the url fallback are going to
be useful
are when the tree doesn't provide it and it's an official
tree format-
in other words, *very* rarely (if ever) for folks using
rsync portdir.
 Seriously, it's only useful when their tree is partially
corrupted or
someone screws up and bumps the format version of the tree
without
updating the tree deps.

For folks _not_ using gentoo provided portdir, the portage
project
'default' is pretty much of no use at all- won't cover
any of the tree
level deps beyond stating "you need portage
xyz".  So again, what gain
beyond added complexity?


> > 3) What the portage project thinks a repo tree
needs does not map to
> > what my tree may need.  Clarifying, format 1
specifies portage xyz and
> > bash-3 (ebuilds in the tree use bash regex).  My
personal tree needs
> > portage xyz (manifest/layout changes), but
requires just bash-2.  With
> > the central db approach, portage will assume my
tree is valid via the
> > version #, and if the number differs, it'll
assume that I require
> > bash3 when in reality, my tree is bash-2 and up. 
This points to why
> > the format -> depends mapping should be bundled
with the tree.
>
> See above.

<from the patch>
+	mylocations = []
+	mylocations.append(os.path.join(os.sep,
portage_const.PRIVATE_PATH,
"format2dep.map"))
+	mylocations.append(os.path.join(tree,
"portage-format2dep.map"))
+	mylocations.append(portage_const.FORMATMAP_URL)
+	
+	formatmap = {}
+	for loc in mylocations:
+		try:
+			f = urlopen(loc)
+		except:
+			continue
<snip>
+				formatmap[l[0]] = l[1:]
+		f.close()

^^^ that looks a helluva lot like a L->R updating of the
dict, with
the rightmost overriding any previous key, rightmost in the
list being
the URL... so unless I'm on seriously good crack (literally
top of the
line), the tree bundled mapping has no say in it if portage
project
states it as xyz.

What also sucks a bit about that is that say a user has 6
overlays
(random figure), it forces 6 pulls of that file- and just to
make it
_really_ fun, lets say they're stuck behind a pissy
firewall that
requires proxying.

Because this code is using urllib instead of the portage
supplied
network downloader (fetch func), it has to pull proxy from
the env
(which current code doesn't do); if this isn't done, it's
6 timeouts
without being able to fetch the file because it's not using
the user
specified fetching command (FETCHCOMMAND).

Additional implementation note, the code above duplicates
grabdict and
doesn't handle comments well (format handling in general
doesn't
handle comments well).  I'd take a look at extending
portage_util.grab* to work with file objects if passed in
instead of
reinventing the wheel in another bit of code.

Finally... I'll note you dodged my point about shoving the
file into
metadata dir; polluting the root dir (then and now) still is
fugly
from where I'm sitting (address that point please) 
~harring

-- 
gentoo-portage-devgentoo.org mailing list

On 3/29/06, Marius Mauch <genonegentoo.org> wrote:
> On Thu, 30 Mar 2006 08:30:17 +0900
> Jason Stubbs <jstubbsgentoo.org> wrote:
>
> > On Thursday 30 March 2006 01:21, Marius Mauch
wrote:
> > > Marius Mauch schrieb:
> > > > So after manifest2 is in, I'll revive
the other issue that IMO is
> > > > a requirement for 2.1: enforcing
dependencies needed to use the
> > > > tree (see old threads or glep44 for
reasoning).
> >
> > Can you summarise the reasoning again please?
>
> a) avoid massive breakage when certain new features are
introduced
> (past examples being cascading profiles or new-style
virtuals)
> b) similar to a) allow people to use new features
without having to
> wait for a year or two

It's worth noting that the massive breakage (virtuals fex)
can be
dealt with a bit saner by writing robust code.  The virtuals
horkage
during cache transfer being the prime example of that...

> These docs will then tell the user about the following
options
> a) use the secret override

This seems like a case where --force would be a helluva lot
better
then some random nasty env var that is documented on a
webpage (assume
the user has no browser due to bootstrapping the system
without a web
connection).

> That should be an extreme exception though, there will
be some rules
> regarding format bumps (like "only use formats
where that haven been
> supported by stable portage for n months").

Knowing the rules up front would be useful to evaluate this
patch's
usefulness...

-- 
gentoo-portage-devgentoo.org mailing list