obsoleting shlibs - what's the plan

I just updated to 4.99.58 from 4.99.24 (!, long story, disk
too small to
build).  Mostly things are working fine.

I updated using BUILD-NetBSD from etcmanage, and then ran
'postinstall
fix obsolete'.  It deleted libraries that I think it
shouldn't have:

        Removed obsolete file /usr/lib/libroken.so.12.3
        Removed obsolete link /usr/lib/libroken.so.12
        Removed obsolete file /usr/lib/libkrb5.so.20.1
        Removed obsolete link /usr/lib/libkrb5.so.20
        Removed obsolete file /usr/lib/libkafs.so.7.0
        Removed obsolete link /usr/lib/libkafs.so.7
        Removed obsolete file /usr/lib/libkadm5srv.so.9.0
        Removed obsolete link /usr/lib/libkadm5srv.so.9
        Removed obsolete file /usr/lib/libkadm5clnt.so.8.0
        Removed obsolete link /usr/lib/libkadm5clnt.so.8
        Removed obsolete file /usr/lib/libhdb.so.9.0
        Removed obsolete link /usr/lib/libhdb.so.9
        Removed obsolete file /usr/lib/libgssapi.so.5.0
        Removed obsolete link /usr/lib/libgssapi.so.5
        Removed obsolete file /usr/lib/libcom_err.so.4.1
        Removed obsolete link /usr/lib/libcom_err.so.4
        Removed obsolete file /usr/lib/libasn1.so.6.1
        Removed obsolete link /usr/lib/libasn1.so.6

These are previous major versions of kerberos libraries.  My
system has
even older versions (picking one):

-r--r--r--  1 root  wheel  514864 Mar 24 21:36 libkrb5.a
lrwxr-xr-x  1 root  wheel      15 Mar 24 21:39 libkrb5.so
-> libkrb5.so.21.0
lrwxr-xr-x  1 root  wheel      15 Dec  2  2003 libkrb5.so.18
-> libkrb5.so.18.0
-r--r--r--  1 root  wheel  237892 Dec  2  2003
libkrb5.so.18.0
lrwxr-xr-x  1 root  wheel      15 Nov  3  2005 libkrb5.so.19
-> libkrb5.so.19.1
-r--r--r--  1 root  wheel  247545 Oct 25  2005
libkrb5.so.19.1
lrwxr-xr-x  1 root  wheel      15 Mar 24 21:39 libkrb5.so.21
-> libkrb5.so.21.0
-r--r--r--  1 root  wheel  402663 Mar 24 21:36
libkrb5.so.21.0
-r--r--r--  1 root  wheel  544664 Mar 24 21:36 libkrb5_p.a
-r--r--r--  1 root  wheel  564748 Mar 24 21:36
libkrb5_pic.a

This removal broke some of my programs:

> ldd /usr/pkg/sbin/cupsd
/usr/pkg/sbin/cupsd:
        -lgssapi.5 => not found
        -lkrb5.20 => not found
        -lasn1.6 => not found
        -ldes.7 => /usr/lib/libdes.so.7
        -lroken.12 => not found
        -lcom_err.4 => not found
        -lcrypt.0 => /usr/lib/libcrypt.so.0
        -lcrypto.3 => /usr/lib/libcrypto.so.3
        -lssl.4 => /usr/lib/libssl.so.4
        -lpthread.0 => /usr/lib/libpthread.so.0
        -lm.0 => /usr/lib/libm387.so.0
        -lm.0 => /usr/lib/libm.so.0
        -lz.1 => /usr/lib/libz.so.1
        -lcups.2 => /usr/pkg/lib/libcups.so.2
        -lresolv.1 => /usr/lib/libresolv.so.1
        -lslp.1 => /usr/pkg/lib/libslp.so.1
        -ldns_sd => /usr/pkg/lib/libdns_sd.so
        -lgssapi.5 => not found
        -lkrb5.20 => not found
        -lasn1.6 => not found
        -lroken.12 => not found
        -lcom_err.4 => not found
        -lc.12 => /usr/lib/libc.so.12


So, I think these shouldn't have been marked obsolete, and
that in
general that we should not obsolete previous major shlibs,
just minor
ones (e.g., if .so.3.1 arrives we can remove .so.3.0, but
not .so.2.1).

uweNetBSD.org (Valeriy E. Ushakov) writes:

>"will be used by programs that do not use
them"?  EPARSE.

I believe that programs linked against the old libraries
will not work or not work correctly when they start using
these libraries, i.e. when they call kerberos functions.
So, the only use of the old libraries will be for ballast
to keep the linker happy.

We have two possible scenarios:

- old libraries are dropped and programs linked against them
will
  fail, telling everyone that the corresponding package
probably
  needs a rebuild anyway.
  
- old libraries are kept, some programs may work without
fault,
  some may fail in subtile ways. The only way to get known
behaviour
  is to rebuild the corresponding packages, but now you have
to
  find the affected programs yourself.

How many packages are using Kerberos without specifying an
option
(which tells me that the user requires Kerberos
functionality)
and will be affected?

All this only happens, when you do an upgrade to the base
system.
In my experience I have to rebuild many packages (i.e _all_
due to
very limited update handling in pkgsrc) after I upgrade the
base
system to another release. Now you have to do it also when
you upgrade
a -current system after a large subsystem was changed. I
don't consider
this a problem or even worth this discussion.

-- 
-- 
                                Michael van Elst
Internet: mlelstvserpens.de
                                "A potential Snark may
lurk in every tree."

On Thu, Mar 27, 2008 at 12:25:24AM +0000, Michael van Elst
wrote:
 > All this only happens, when you do an upgrade to the
base system.
 > In my experience I have to rebuild many packages (i.e
_all_ due to
 > very limited update handling in pkgsrc) after I
upgrade the base
 > system to another release. Now you have to do it also
when you upgrade
 > a -current system after a large subsystem was changed.
I don't consider
 > this a problem or even worth this discussion.

I beg to differ... breaking backwards compatibility for old
binaries
*is* a problem.

-- 
David A. Holland
dhollandnetbsd.org

On Wed, Mar 26, 2008 at 10:53:02AM -0400, Greg Troxel
wrote:
  | So, I think these shouldn't have been marked obsolete,
and that in
  | general that we should not obsolete previous major
shlibs, just minor
  | ones (e.g., if .so.3.1 arrives we can remove .so.3.0,
but not .so.2.1).

Correct.  I've fixed this.

On Wed, Mar 26, 2008 at 10:39:14PM -0400, Thor Lancelot
Simon wrote:
 > > In practice, you can in some circumstances end up
with multiple major
 > > versions of the same lib loaded at once, and with
standard ELF that
 > > pretty much inevitably results in nasal demons.
 > > 
 > > This is what can happen:
 > > 
 > > Starting point:
 > > 
 > >    app uses libwowiezowie.so.1
 > >    app uses libkrb5.so.20
 > >    libwowiezowie.so.1 uses libkrb5.so.20
 > > 
 > > This works. But now suppose I recompile
libwowiezowie:
 > > 
 > >    app uses libwowiezowie.so.1
 > 
 > This cannot happen with our in-tree shared libraries,
because we always
 > bump the major number of any dependent library when
bumping the major
 > number of any library it depends upon.
 > 
 > Or, so I think, anyway.  We certainly have tried hard
to prevent this
 > situation for a long time.

Right. But libs in packages? (I suppose I should have stated
that
bit up front...)

-- 
David A. Holland
dhollandnetbsd.org

On Thu, Mar 27, 2008 at 02:22:30AM +0000, David Holland
wrote:
> On Thu, Mar 27, 2008 at 12:25:24AM +0000, Michael van
Elst wrote:
>  > All this only happens, when you do an upgrade to
the base system.
>  > In my experience I have to rebuild many packages
(i.e _all_ due to
>  > very limited update handling in pkgsrc) after I
upgrade the base
>  > system to another release. Now you have to do it
also when you upgrade
>  > a -current system after a large subsystem was
changed. I don't consider
>  > this a problem or even worth this discussion.
> 
> I beg to differ... breaking backwards compatibility for
old binaries
> *is* a problem.

Strongly seconded. I've never had issue with older package
when upgrading
a system, exept for some threaded apps when switching from a
release to
a post-netbsd-4 current which dropped SA support. But I
consider as an
exeption. 

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

mlelstvserpens.de (Michael van Elst) writes:

This is now moot, as Luke said he fixed the set lists to
remove the
obsolete tag on the old libs.

> I believe that programs linked against the old
libraries
> will not work or not work correctly when they start
using
> these libraries, i.e. when they call kerberos
functions.

Can you explain why?  Please assume that they worked
correctly before
the upgrade.

> So, the only use of the old libraries will be for
ballast
> to keep the linker happy.

That only follows from your unproven assertion above.

> We have two possible scenarios:
>
> - old libraries are dropped and programs linked against
them will
>   fail, telling everyone that the corresponding package
probably
>   needs a rebuild anyway.

This is hostile to users and not the standard NetBSD
practice.

> - old libraries are kept, some programs may work
without fault,
>   some may fail in subtile ways. The only way to get
known behaviour
>   is to rebuild the corresponding packages, but now you
have to
>   find the affected programs yourself.

If anything, this is a general problem of updating shlibs. 
Your
assertion of failure seems unfounded.  It would be nice to
have some
utility to mark packages unsafe_depends if they link against
a base
system lib for which a higher major is available, and then
pkg_rolling-replace could straigthen all that out.  But that
step can be
done separately.

> How many packages are using Kerberos without specifying
an option
> (which tells me that the user requires Kerberos
functionality)
> and will be affected?

Quite a few I think; it's pretty normal for pkgsrc programs

> All this only happens, when you do an upgrade to the
base system.
> In my experience I have to rebuild many packages (i.e
_all_ due to
> very limited update handling in pkgsrc) after I upgrade
the base
> system to another release. Now you have to do it also
when you upgrade

My experience has been that it has basically not been
necessary to
rebuild packages, with very limited exceptions.

> a -current system after a large subsystem was changed.
I don't consider
> this a problem or even worth this discussion.

Many others consider it a problem; the NetBSD way is to be
careful about
binary compatibility.  It's not just about current; had this
remained
then it would also have happened from 4 to 5.

On Thu, 27 Mar 2008, Greg Troxel wrote:

> mlelstvserpens.de (Michael van Elst) writes:
>
> This is now moot, as Luke said he fixed the set lists
to remove the
> obsolete tag on the old libs.
>
>> I believe that programs linked against the old
libraries
>> will not work or not work correctly when they start
using
>> these libraries, i.e. when they call kerberos
functions.
>
> Can you explain why?  Please assume that they worked
correctly before
> the upgrade.

I'm afraid this is the reason I've sometimes had the
"Oh, god, my system
is broken, time to wipe out pkgsrc and start over."
Fortunately a 6YO P4
system is fast enough that I don't have to wait weeks for
the recompile.

I'm not completely sure this is the reason, but once in a
while (once or
twice a year at the most?) I've had to do this.

> That only follows from your unproven assertion above.

Heh. I resemble that comment.

>> We have two possible scenarios:
>>
>> - old libraries are dropped and programs linked
against them will
>>   fail, telling everyone that the corresponding
package probably
>>   needs a rebuild anyway.
>
> This is hostile to users and not the standard NetBSD
practice.

I don't believe this is the case. The NetBSD way is to do
secret things
that break -current and point and laugh at the outsiders who
got bitten.
Wait, what were we talking about? 

>> a -current system after a large subsystem was
changed. I don't consider
>> this a problem or even worth this discussion.
>
> Many others consider it a problem; the NetBSD way is to
be careful about
> binary compatibility.  It's not just about current; had
this remained
> then it would also have happened from 4 to 5.

The NetBSD way, maybe, but not the pkgsrc way. Too many cats
to herd.
I'd say break it and make people recompile and wish we had a
way to
crosscompile so people like me who are running SPARCclassics
don't have
a month of downtime.

-- 
Hisashi T Fujinaka - htoddtwofifty.com
BSEE(6/86) + BSChem(3/95) + BAEnglish(8/95) + MSCS(8/03) +
$2.50 = latte

On Thu, Mar 27, 2008 at 08:08:46AM -0700, Hisashi T Fujinaka
wrote:
> The NetBSD way, maybe, but not the pkgsrc way. Too many
cats to herd.
> I'd say break it and make people recompile and wish we
had a way to
> crosscompile so people like me who are running
SPARCclassics don't have
> a month of downtime.

If you can be more specific of what you need / want to
cross-compile,
tell me about it. If you are on AsiaBSDCon, we can discuss
this during
or after the NetBSD discussion round.

Joerg

On Thu, Mar 27, 2008 at 08:08:46AM -0700, Hisashi T Fujinaka
wrote:
 > >>I believe that programs linked against the old
libraries
 > >>will not work or not work correctly when they
start using
 > >>these libraries, i.e. when they call kerberos
functions.
 > >
 > >Can you explain why?  Please assume that they
worked correctly before
 > >the upgrade.
 > 
 > I'm afraid this is the reason I've sometimes had the
"Oh, god, my system
 > is broken, time to wipe out pkgsrc and start
over." Fortunately a 6YO P4
 > system is fast enough that I don't have to wait weeks
for the recompile.
 > 
 > I'm not completely sure this is the reason, but once
in a while (once or
 > twice a year at the most?) I've had to do this.

If this happens as a result of changes in the base system,
it's a bug,
and should be corrected before it bites anyone else.

-- 
David A. Holland
dhollandnetbsd.org