compressed vnd, cgd and encryption

I've a need to have compressed vnds which aren't easily
readable except by 
the target system. The target system will vary in spec, but
could be as 
low as P100, 64MB RAM. I can think of a few scenarios:

1) makefs > vncompress > bdes encrypt. On target, bdes
decrypt to mfs 
and run from there (currently using this)

Pros:	easy to generate
 	one-time decryption (so no per-access overhead)
 	No kernel size overhead
Cons:	Expensive on RAM

2) makefs (to get image size for cgd) > make cgd in vnd
> copy data > 
vncompress. Run in-situ on target

Pros:	No RAM overhead
Cons: 	cgd data won't compress well
 	kernel overhead for cgd
 	Per-access overhead
 	Potentially computationally heavy on low-end machine

3) As 2), but attempt to reverse order of cgd and vncompress

Pros:	As 2), but should compress much better.
Cons:	As 2), plus is it even possible?

4) makefs, vncompress, some form of basic encryption. Extend
vnd to be 
able to decrypt on fly before passing to uncompression.

Somewhere in compstrategy() in dev/vnd.c looks like the
right place, just 
before the /* uncompress the buffer */ line (row 1663).

Encryption could be similar to bdes or perhaps something
more simple (even 
EORing a repeated key).

Pros:	Could hardwire key at compile time in kernel
 	Lightweight encryption not a big computational overhead
 	Little kernel size overhead
 	Will compress well
 	No RAM overhead
Cons:	Requires kernel hacking
 	Does vnd's file access mechanisms allow this to be
slotted in?

Option 4) seems best to me for my particular purposes.
Thoughts?

(Please CC: me in on replies)

-- 
Stephen

On Fri, 23 Jun 2006, Stephen Borrill wrote:
> 3) As 2), but attempt to reverse order of cgd and
vncompress
>
> Pros:	As 2), but should compress much better.
> Cons:	As 2), plus is it even possible?

I'd expect cgd to work as source for a compressed vnd
image.
If not that should be fixed.


  - Hubert

On Fri, Jun 23, 2006 at 01:36:50PM +0200, Hubert Feyrer
wrote:
> On Fri, 23 Jun 2006, Stephen Borrill wrote:
> >3) As 2), but attempt to reverse order of cgd and
vncompress
> >
> >Pros:	As 2), but should compress much better.
> >Cons:	As 2), plus is it even possible?
> 
> I'd expect cgd to work as source for a compressed vnd
image.
> If not that should be fixed.

cgd on vnd works, and is useful for just this sort of thing
- think
about the vnd file on a remote not-entirely-trusted
fileserver/NAS.

Note that vncompress means your vnd will be readonly, if
that matters
to your needs, but you probably already discovered this.

--
Dan.

On Fri, Jun 23, 2006 at 09:47:13PM +1000, Daniel Carosone
wrote:
> cgd on vnd works

but you wanted to know about vnd on cgd - this certainly
also works.

--
Dan.

On Fri, Jun 23, 2006 at 09:54:05PM +1000, Daniel Carosone
wrote:
> On Fri, Jun 23, 2006 at 09:47:13PM +1000, Daniel
Carosone wrote:
> > cgd on vnd works
> 
> but you wanted to know about vnd on cgd - this
certainly also works.

and finally I get what you were concerned about: will this
work
directly with vnd on the cgd device raw, without a
filesystem in
between?  no. vnd will need a filesystem to hold the file,
cgd or not.

However, you could use a pretty light-weight and simple
filesystem in
the cgd, especially given you're going to be read-only. I
suggest
cd9660 with the single vnd file inside.

Of course, you wanted all of this to start with a file on
the client,
and you're still going to need that for the cgd, which
means another
vnd.

So you'll wind up with:

 - makefs -t ffs
 - vndcompress
 - makefs -t cd9660
 - dd same file size
 - vnconfig
 - cgdconfig
 - dd cd9660 image into cgd

and on the client machine,
ffs-on-compressedvnd-on-cd9660-on-cgd-on-vnd.  Which will
work, but
will probably also drive you batty.  What you really want is
a
compressed filesystem you can mount from the cgd directly. 
I don't
have any good suggestions for that, sorry.

What are you storing in here? Can you really not do the
decompression
in userland, even counting tricks like gzexe?

--
Dan.

On Fri, 23 Jun 2006, Daniel Carosone wrote:
> On Fri, Jun 23, 2006 at 09:54:05PM +1000, Daniel
Carosone wrote:
>> On Fri, Jun 23, 2006 at 09:47:13PM +1000, Daniel
Carosone wrote:
>>> cgd on vnd works
>>
>> but you wanted to know about vnd on cgd - this
certainly also works.
>
> and finally I get what you were concerned about: will
this work
> directly with vnd on the cgd device raw, without a
filesystem in
> between?  no. vnd will need a filesystem to hold the
file, cgd or not.
>
> However, you could use a pretty light-weight and simple
filesystem in
> the cgd, especially given you're going to be
read-only. I suggest
> cd9660 with the single vnd file inside.
>
> Of course, you wanted all of this to start with a file
on the client,
> and you're still going to need that for the cgd, which
means another
> vnd.
>
> So you'll wind up with:
>
> - makefs -t ffs
> - vndcompress
> - makefs -t cd9660
> - dd same file size
> - vnconfig
> - cgdconfig
> - dd cd9660 image into cgd
>
> and on the client machine,
> ffs-on-compressedvnd-on-cd9660-on-cgd-on-vnd.  Which
will work, but
> will probably also drive you batty.

I could do with remaining sane 

I'm also concerned about overhead here on low-end CPU
machines.

> What you really want is a
> compressed filesystem you can mount from the cgd
directly.  I don't
> have any good suggestions for that, sorry.

ISTR cgd being CPU intensive hence me thinking about a
lightweight 
encryption built into vnd at the point of decompression. I
think I'll do a 
proof of concept say just EORing all the compressed vnd with
a single 
value and seeing what I achieve.

> What are you storing in here? Can you really not do the
decompression
> in userland, even counting tricks like gzexe?

I'm storing chunks of software, libraries and data (e.g.
linux emulation 
or X). It's not sensitive data, it's more for copy
protection.

-- 
Stephen

Daniel Carosone <dangeek.com.au> writes:

> and finally I get what you were concerned about: will
this work
> directly with vnd on the cgd device raw, without a
filesystem in
> between?  no. vnd will need a filesystem to hold the
file, cgd or not.

This has always seemed like a bug or weakness in the
"vndcompress"
implementation. vnd(4) provides a file-to-device mapping;
having the
decompression support in there means that we only support
compressed-file-to-device. In contrast, if the decompression
layer
were a separate device-to-device mapping (zd?), we could
support both
modes (file->vnd->zd->filesystem and
(xxxd->zd->filesystem). This would
also be good for, say, images in flash in an embedded
system, where
you want to drop the image into some address block of flash,
but
there's not much value in a "wrapper"
filesystem.

        - Nathan

On Mon, Jun 26, 2006 at 10:04:40AM -0400, Nathan J. Williams
wrote:
> Daniel Carosone <dangeek.com.au> writes:
> 
> > and finally I get what you were concerned about:
will this work
> > directly with vnd on the cgd device raw, without a
filesystem in
> > between?  no. vnd will need a filesystem to hold
the file, cgd or not.
> 
> This has always seemed like a bug or weakness in the
"vndcompress"
> implementation.

Absolutely.

> In contrast, if the decompression layer were a separate
> device-to-device mapping (zd?), we could support both
modes
> (file->vnd->zd->filesystem and
(xxxd->zd->filesystem). 

Yah.  Alternately/additionally, supporting a filesystem that
allowed
compressed files would be good.  The typical usage of this
tool is for
compressed liveCD images - and yet apparently there are
compressed
file extensions for iso9660.  It's easier to support
compressed files
because of the read-only, prepared in advance way the
filesystem image
is written.

--
Dan.