SOC project proposal: easy updates

Hi,

How does this sound as an SoC project proposal?

Thanks,

J

  - - 8< - - - - - - - - - - - - - Cut here - - - - - - -
- - - - - - >8 - -

  Easy updates for NetBSD systems

  Project summary:
    Estimated difficulty: Medium
    Person/group of contact: Julian Coleman <jdc AT
netbsd.org>
    Person/group of contact: tech-misc mailing list
    Person/group of contact: tech-install mailing list

  Add the ability to easily update NetBSD systems with
security patches and
  OS updates.

  This project involves work in two major areas.  The first
area is the
  NetBSD build process, where the framework for updates
needs be added
  into the build process.  The second area is the client
side, where the
  mechanisms for discovering and installing updates need to
be written.

  There is unfinished work where all the installed files in
the NetBSD
  system are tagged with a system package (syspkg) name. 
This work needs
  to be finished, so that all installed system files are
recorded in a
  similar fashion to the way that pkgsrc files are recorded.
 This will
  be an additional part of the NetBSD build process.

  For system updates, for example the change from 3.0 to
3.0.1 or 3.1, not
  all the installed system files are updated.  The NetBSD
build process
  should be extended to compare the results of the
"base" system build
  (e.g. 3.0) and the results of the "update"
system build (e.g. 3.0.1).
  This comparison will generate a collection of system
packages that need
  to be installed in order to update the system to the newer
version.
  Security patches, which usually affect a small number of
system packages
  could be handled in the same way.  For ease of
installation and
  administration, each update should be a superset of the
previous ones.

  On the client side, a tool needs to be written to handle
system package
  updates.  This could be an extension to the existing
`sysinst` program.
  The tool needs to compare the list of currently installed
system packages
  with the list of newer system packages available for
download.  For
  example, the program could determine that the the base
system is 3.0 and
  discover that there are updates available to 3.0.1, 3.0.2
and 3.1, as well
  as security patches for 3.0.2 and 3.1.  The way in which
this information
  is generated for the client tool needs to be determined,
as it needs to an
  integral part of the build process, above.

  The client tool needs to be able to safely backup the
existing system
  packages and install the new versions.  When the user is
happy with the
  new packages, the previous versions can be removed.

  Project deliverables:
    build system: "database" of installed system
packages, with versions
    build system: ability to compare two NetBSD builds to
determine which
      system packages are changed
    build system: creation of downloadable system package
"update list" for
      updated packages
    client tool: query and compare installed system package
list and update
      list(s) from server(s)
    client tool: backup obsoleted system packages, download
and install
      updated system packages

  Background investigations:
    NetBSD build process
    NetBSD syspkg infrastructure
    pkgsrc databse of installed files and packages
    Other OS update mechanisms (e.g. FreeBSD)

  Knowledge required:
    C programming
    Makefile programming
    Shell script programming

  - - 8< - - - - - - - - - - - - - Cut here - - - - - - -
- - - - - - >8 - -

-- 
  My other computer also runs NetBSD    /        Sailing at
Newbiggin
        http://www.netbsd.org/    
   /   http://www.newbi
gginsailingclub.org/

Julian Coleman <jdccoris.org.uk> writes:
> How does this sound as an SoC project proposal?

I'd say that the project should be re-written to propose a
port of the
existing FreeBSD tools for doing the same thing. There is no
good
reason to re-invent the wheel.

Perry

> I'd say that the project should be re-written to
propose a port of the
> existing FreeBSD tools for doing the same thing. There
is no good
> reason to re-invent the wheel.

Do you mean for both the build side and for the client side?
 If so, do
we need syspkgs - what purpose would they serve?  Or would
this be a
combination of syspkgs and the FreeBSD tools?

J

-- 
  My other computer also runs NetBSD    /        Sailing at
Newbiggin
        http://www.netbsd.org/    
   /   http://www.newbi
gginsailingclub.org/

On Thu, 15 Mar 2007, Julian Coleman wrote:

> > I'd say that the project should be re-written to
propose a port of the
> > existing FreeBSD tools for doing the same thing.
There is no good
> > reason to re-invent the wheel.
> 
> Do you mean for both the build side and for the client
side?  If so, do
> we need syspkgs - what purpose would they serve?  Or
would this be a
> combination of syspkgs and the FreeBSD tools?

This was discussed a lot recently. Some suggested we should
just provide 
complete files (such as included in syspkgs) and not binary
diffs (like 
FreeBSD's binary updates).

Your proposal says:

   The NetBSD build process should be extended to compare
the results of
   the "base" system build (e.g. 3.0) and the
results of the "update" 
   system build (e.g. 3.0.1).

Do you have further details on how you'd like this to be
done?

As mentioned in the FreeBSD binary updates whitepaper, my
Puget Sound 
Technology system just created MD5 checksums for all files
of resulting 
install and then compared after patching. Then I manually
reviewed the 
changed files to determine if there was any noise. (FreeBSD
automates 
this.)

I tarred up the new files and appended to a self-extracting
sh script 
that backed up previous files, can roll back changes, list
files, and 
give info about the update.

I think we don't need to do binary diffs. But just use
syspkgs. Tell the 
users that new syspkgs are available (even in database, like

pkg_summary(5), or in security announcements) and let the
user upgrade 
with pkg_add. (By the way, I have a pkg_add that overwrites
files instead 
of deleting existing package and removing files first; I
have used it 
hundreds of times for over a year on various NetBSD and
Linux systems but 
not recently; most of it is in pkgsrc-wip.)

  Jeremy C. Reed

On Mar 15, 2007, at 10:04 AM, Jeremy C. Reed wrote:

> I think we don't need to do binary diffs. But just use
syspkgs.

I agree 100%.

-- thorpej

> This was discussed a lot recently. Some suggested we
should just provide 
> complete files (such as included in syspkgs) and not
binary diffs (like 
> FreeBSD's binary updates).

My preference is for complete syspkgs.  Cutting down the
amount of data to
be transferred seems a worthy aim, but I thought that making
a syspkg the
smallest unit that is transferred is more manageable (but
see below).

> Do you have further details on how you'd like this to
be done?
> 
> As mentioned in the FreeBSD binary updates whitepaper,
my Puget Sound 
> Technology system just created MD5 checksums for all
files of resulting 
> install and then compared after patching. Then I
manually reviewed the 
> changed files to determine if there was any noise.
(FreeBSD automates 
> this.)

Some sort of checksum per syspkg is what I was thinking. 
However, it
might be easier to do one per file and then convert the list
of changed
files into a list of changed syspkgs.  I was hoping that
this could be
automated - it could then be added to the build process for
an update
release.

> I tarred up the new files and appended to a
self-extracting sh script 
> that backed up previous files, can roll back changes,
list files, and 
> give info about the update.

I was thinking that the syspkgs would be available to
download individually.
However, it probably makes more sense to have them in one
file for release
updates (e.g. 3.0 to 3.0.1).  Security fixes could also be
done as a single
download containing all the updated syspkgs.  I was thinking
that each new
version would contain all the changed syspkgs from the base
version, so
that it's possible to go from 3.0 or 3.0.1 to 3.0.2.  For
the 3.0.1 to
3.0.2 update, one would download more than is necessary, but
at least this
would be manageable on the production (releng) side.

I wasn't thinking of putting the installation mechanism in
the update file.
I was thinking that a client (e.g. an extension to sysinst)
would fetch the
current list of available updates for the current system and
then download
and install these updated syspkgs.  The client would handle
backing up the
existing syspkgs and installing the updated ones, as well as
roll back and
commit (remove backup).

I haven't considered in detail the format of the
"database" listing the
available releases/updates and security updates.  The client
needs to be
able to determine which updates are available and which
updates supercede
others.  For example, the client has 3.0.1 installed and
updates are
available for 3.0.2 and 3.1 from the same base system (3.0),
as well as
security patches for 3.0 and 3.0.1.  The client needs to
know which choices
to present to the user - maybe it just presents them all. 
How do we handle
versioning for security patches?

> I think we don't need to do binary diffs. But just use
syspkgs. Tell the 
> users that new syspkgs are available (even in database,
like 
> pkg_summary(5), or in security announcements) and let
the user upgrade 
> with pkg_add. (By the way, I have a pkg_add that
overwrites files instead 
> of deleting existing package and removing files first;
I have used it 
> hundreds of times for over a year on various NetBSD and
Linux systems but 
> not recently; most of it is in pkgsrc-wip.)

There seems to be a lot of overlap in the way that I was
envisaging this
working with how I'd like pkgsrc package updates to be done.
 However, I
decided against extending the project to include pkgsrc, as
I thought that
the scope would then be too broad.  Plus, I haven't really
been following
pkgsrc and pkgsrc-wip recently.

Do you think that there is still merit in having this as an
SoC project?
It seems that there is a fair amount of work available that
goes some of
the way to achieving the goals I had in mind.  pkg_add could
be the tool
that is used to update system packages as well as pkgsrc
packages - thinking
about this, it makes sense from a user perspective. 
Possibly the project
would be better as a pkgsrc or part pkgsrc one?

Thanks,

J

-- 
  My other computer also runs NetBSD    /        Sailing at
Newbiggin
        http://www.netbsd.org/    
   /   http://www.newbi
gginsailingclub.org/

On Thu, Mar 15, 2007 at 07:29:11PM +0000, Julian Coleman
wrote:
> 
> My preference is for complete syspkgs.  Cutting down
the amount of data to
> be transferred seems a worthy aim, but I thought that
making a syspkg the
> smallest unit that is transferred is more manageable
(but see below).

I worry that people are trying to use syspkgs for two
different purposes:
1) selective installation
2) security updates

However, although related, they are different problems and
have different
solutions.  Or at least have different constraints.

For instance you wouldn't want a securiy update to go though
'package
update hell', you just want it to overwrite the specific
files.
Indeed you probably only want the re-release the specific
files as well.
Which requires some sceheme that allows a file to belong to
more than
one package.

Also unless you sort out the way questions get asked in
sysinst (or
whatever) installing a working system becomes just too
painful.

	David

-- 
David Laight: davidl8s.co.uk

David Laight <davidl8s.co.uk> writes:
> On Thu, Mar 15, 2007 at 07:29:11PM +0000, Julian
Coleman wrote:
>> 
>> My preference is for complete syspkgs.  Cutting
down the amount of data to
>> be transferred seems a worthy aim, but I thought
that making a syspkg the
>> smallest unit that is transferred is more
manageable (but see below).
>
> I worry that people are trying to use syspkgs for two
different purposes:

I worry that by reinventing the wheel repeatedly, NetBSD
will assure
that it doesn't get a working solution. The FreeBSD stuff is
"good
enough" -- NetBSD should just take it. A working
solution that is
reasonably good is better than perfect vaporware.

I'm sure that there are all sorts of ways in which the
FreeBSD
solution violates one person or another's taste, but it is
finished
and actually runs.

Perry