: r604 - docs/nutissues.txt

Author: michael
Date: Tue Feb 12 00:33:21 2008
New Revision: 604

Log:
3 more issues which have come up in the past but have IIRC
never
been resolved.


Modified:
   docs/nutissues.txt

Modified: docs/nutissues.txt
============================================================
==================
--- docs/nutissues.txt	(original)
+++ docs/nutissues.txt	Tue Feb 12 00:33:21 2008
 -110,3
+110,42  Solutions:
 A. Store such alternative playlists of scenes in info
packets somehow.
 B. Design a separate layer for it.
 C. Do not support this.
+
+
+Issue header-compression
+------------------------
+Headers of codec frames often contain low entropy
information or things
+we already know like the frame size.
+
+A. Store header bytes and length in the framecode table.
+B. Leave things as they are.
+
+
+Issue small-frames
+------------------
+The original intent of nut frames was that 1 container
frame == 1 codec
+frame. Though this does not seem to be explicitly written
in nut.txt.
+Also it is inefficient for very small frames, AMR-NB for
example has 6-32
+bytes per frame.
+
+Solutions:
+A. Enforce 1 container frame == 1 codec frame even if it
causes 10% overhead.
+B. Allow multiple frames as long as the whole packet is
less than some
+   fixed minimum in bytes (like 256byte)
+C. Allow multiple frames as long as the whole packet is
less than some
+   fixed minimum in bytes (like 256byte) and the codec uses
a constant
+   framesize in samples.
+D. Use header compression, that is allow to store the first
(few) bytes
+   of a codec frame together with its size in the framecode
table. This
+   would allow us to store the size of a frame without any
redundancy.
+   Thus effectivly avoiding the overhead small frames
cause.
+
+
+Issue pcm-frames
+----------------
+No word is said about how many or few PCM samples should be
in a frame.
+
+Solutions:
+A. Define an maximum number of samples (like 512)
+B. Define an maximum timespam (like 0.1 sec)
+C. Define an maximum number of bytes (like 1024)
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 12:33:22AM +0100, michael wrote:
> Author: michael
> Date: Tue Feb 12 00:33:21 2008
> New Revision: 604
> 
> Log:
> 3 more issues which have come up in the past but have
IIRC never
> been resolved.
> 
> 
> Modified:
>    docs/nutissues.txt
> 
> Modified: docs/nutissues.txt
>
============================================================
==================
> --- docs/nutissues.txt	(original)
> +++ docs/nutissues.txt	Tue Feb 12 00:33:21 2008
>  -110,3 +110,42  Solutions:
>  A. Store such alternative playlists of scenes in info
packets somehow.
>  B. Design a separate layer for it.
>  C. Do not support this.
> +
> +
> +Issue header-compression
> +------------------------
> +Headers of codec frames often contain low entropy
information or things
> +we already know like the frame size.
> +
> +A. Store header bytes and length in the framecode
table.
> +B. Leave things as they are.

I think this one was resolved strongly as a leave-it-alone.
I'm
absolutely against any proposal that precludes
implementations from
performing zero-copy decoding from the stream buffer.

> +Issue small-frames
> +------------------
> +The original intent of nut frames was that 1 container
frame == 1 codec
> +frame. Though this does not seem to be explicitly
written in nut.txt.

It is.

> +Also it is inefficient for very small frames, AMR-NB
for example has 6-32
> +bytes per frame.

I don't think anyone really cares that much about efficiency
when
storing shit codecs in NUT. Obviously any good codec will
use large
frame sizes or compression will not be good.

> +Solutions:
> +A. Enforce 1 container frame == 1 codec frame even if
it causes 10% overhead.

Yes.

> +B. Allow multiple frames as long as the whole packet
is less than some
> +   fixed minimum in bytes (like 256byte)

Very very bad. Demuxing requires a codec-specific framer.

> +C. Allow multiple frames as long as the whole packet
is less than some
> +   fixed minimum in bytes (like 256byte) and the codec
uses a constant
> +   framesize in samples.

This does not help.

> +D. Use header compression, that is allow to store the
first (few) bytes
> +   of a codec frame together with its size in the
framecode table. This
> +   would allow us to store the size of a frame without
any redundancy.
> +   Thus effectivly avoiding the overhead small frames
cause.

At the cost of efficient decoding... If such a horrid
proposal were
accepted, it would have to be restricted to frames smaller
than ~256
bytes. Otherwise the buffer requirements for reconstructing
the
complete frame would be troublesome and for very large
frames it would
make a huge performance difference.

> +Issue pcm-frames
> +----------------
> +No word is said about how many or few PCM samples
should be in a frame.
> +
> +Solutions:
> +A. Define an maximum number of samples (like 512)
> +B. Define an maximum timespam (like 0.1 sec)
> +C. Define an maximum number of bytes (like 1024)

PCM is already a much bigger question due to sample format
and
interleaving issues. Should there be a new fourcc for each
combination
of PCM properties, or a single fourcc with extradata? If the
latter,
the number of samples per frame would probably be coded in
the
extradata or something. I'm open to ideas.

Rich
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, Feb 11, 2008 at 10:41:51PM -0500, Rich Felker
wrote:
> On Tue, Feb 12, 2008 at 12:33:22AM +0100, michael
wrote:
> > Author: michael
> > Date: Tue Feb 12 00:33:21 2008
> > New Revision: 604
> > 
> > Log:
> > 3 more issues which have come up in the past but
have IIRC never
> > been resolved.
> > 
> > 
> > Modified:
> >    docs/nutissues.txt
> > 
> > Modified: docs/nutissues.txt
> >
============================================================
==================
> > --- docs/nutissues.txt	(original)
> > +++ docs/nutissues.txt	Tue Feb 12 00:33:21 2008
> >  -110,3 +110,42  Solutions:
> >  A. Store such alternative playlists of scenes in
info packets somehow.
> >  B. Design a separate layer for it.
> >  C. Do not support this.
> > +
> > +
> > +Issue header-compression
> > +------------------------
> > +Headers of codec frames often contain low entropy
information or things
> > +we already know like the frame size.
> > +
> > +A. Store header bytes and length in the framecode
table.
> > +B. Leave things as they are.
> 
> I think this one was resolved strongly as a
leave-it-alone.

resolved by whom? 


> I'm
> absolutely against any proposal that precludes
implementations from
> performing zero-copy decoding from the stream buffer.

Well if you have a buffer in which the frame is, just write
the header
before it. If not just write the header and then (f)read()
the rest.


> 
> > +Issue small-frames
> > +------------------
> > +The original intent of nut frames was that 1
container frame == 1 codec
> > +frame. Though this does not seem to be explicitly
written in nut.txt.
> 
> It is.

I thought so too but i searched, and didnt find it so where
is it?


> 
> > +Also it is inefficient for very small frames,
AMR-NB for example has 6-32
> > +bytes per frame.
> 
> I don't think anyone really cares that much about
efficiency when
> storing 

I do, others likely care about 10% overhead as well.


> shit codecs

I think this applies more to audio codes, if it limited to
coding shit
i wont bother and gladly leave it alone. ;)


> in NUT. Obviously any good codec will use large
> frame sizes 

no


> or compression will not be good.

also not true


> 
> > +Solutions:
> > +A. Enforce 1 container frame == 1 codec frame
even if it causes 10% overhead.
> 
> Yes.

Vote noted. And my veto as well unless option D is selected
which would
also include A.


> 
> > +B. Allow multiple frames as long as the whole
packet is less than some
> > +   fixed minimum in bytes (like 256byte)
> 
> Very very bad. Demuxing requires a codec-specific
framer.
> 
> > +C. Allow multiple frames as long as the whole
packet is less than some
> > +   fixed minimum in bytes (like 256byte) and the
codec uses a constant
> > +   framesize in samples.
> 
> This does not help.

help what?


> 
> > +D. Use header compression, that is allow to store
the first (few) bytes
> > +   of a codec frame together with its size in the
framecode table. This
> > +   would allow us to store the size of a frame
without any redundancy.
> > +   Thus effectivly avoiding the overhead small
frames cause.
> 
> At the cost of efficient decoding... If such a horrid
proposal were

Id guess the overhead of working with 6 byte frames and
decoding
a framecode before each takes more time than copying 60
bytes. Also depending
on the media this comes from 10% extra disk IO should beat 1
copy pass in
slowness.

And i dont give a damn about zero copy for a 400-1000 byte /
sec codec!
also so much for bad compression ...

[...]
-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

No human being will ever know the Truth, for even if they
happen to say it
by chance, they would not even known they had done so. --
Xenophanes

_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 05:51:44AM +0100, Michael
Niedermayer wrote:
> > I'm
> > absolutely against any proposal that precludes
implementations from
> > performing zero-copy decoding from the stream
buffer.
> 
> Well if you have a buffer in which the frame is, just
write the header
> before it. If not just write the header and then
(f)read() the rest.

I'm envisioning a model where the demuxer directly exports
frames from
their original location in the stream buffer. It's likely
that this
buffer needs to be treated as read-only. It might be mmapped
from a
file (in which case writing would invoke COW) or the
previous frame
might not yet be released by the caller (in which case,
writing the
header would clobber the contents of the previous frame) --
imagine
for instance that we need to decode the next audio frame to
buffer it,
but aren't ready to decode the previous video frame yet.

The reason I'm against this whole header compression thing
is that it
puts lots of nasty restraints on the demuxer implementation
that
preclude efficient design. The nasty cases can be
special-cased to
minimize their cost, but the tradeoff is implementation size
and
complexity. The only options I'm willing to consider (and
which I
still dislike) are:

A. Only allow header elision (a more correct name than
compression)
for frames below a very small size limit (e.g. small enough
that the
compression makes at least a 1% difference to space
requirements).

B. Require that the elided header fit in the same number of
bytes (or
fewer) than the minimal NUT frame header size for the
corresponding
framecode. This would allow overwrite-in-place without
clobbering
other frames, but still the COW issue remains which I don't
like...

> > > +Issue small-frames
> > > +------------------
> > > +The original intent of nut frames was that 1
container frame == 1 codec
> > > +frame. Though this does not seem to be
explicitly written in nut.txt.
> > 
> > It is.
> 
> I thought so too but i searched, and didnt find it so
where is it?

It's at least in nut-english.txt 

> > > +Also it is inefficient for very small
frames, AMR-NB for example has 6-32
> > > +bytes per frame.
> > 
> > I don't think anyone really cares that much about
efficiency when
> > storing 
> 
> I do, others likely care about 10% overhead as well.

Care about it for what applications? For storing a master
file in some
odd barely-compressed audio form, I wouldn't care. For
streaming radio
or movie rips, of course it would matter, but I think
someone would
choose an appropriate codec for such applications..

> > shit codecs
> 
> I think this applies more to audio codes, if it limited
to coding shit
> i wont bother and gladly leave it alone. ;)
> 
> > in NUT. Obviously any good codec will use large
> > frame sizes 
> 
> no

Is there any audio codec of real interest with such small
frame sizes?

> > > +Solutions:
> > > +A. Enforce 1 container frame == 1 codec
frame even if it causes 10% overhead.
> > 
> > Yes.
> 
> Vote noted. And my veto as well unless option D is
selected which would
> also include A.

It's frustrating how you intend to "veto"
decisions that were made 4
years ago now when we're trying to finally get NUT to see
the light.

> > > +B. Allow multiple frames as long as the
whole packet is less than some
> > > +   fixed minimum in bytes (like 256byte)
> > 
> > Very very bad. Demuxing requires a codec-specific
framer.
> > 
> > > +C. Allow multiple frames as long as the
whole packet is less than some
> > > +   fixed minimum in bytes (like 256byte) and
the codec uses a constant
> > > +   framesize in samples.
> > 
> > This does not help.
> 
> help what?

I meant that I don't see how it's any less offensive than
option B.
The fundamental problem is still there: inability to demux
and seek to
individual frames without codec incest. It's sad that we've
gotten to
the point where 1 byte of overhead per frame is too much..
Even OGG
and MKV have at least that much!

> > > +D. Use header compression, that is allow to
store the first (few) bytes
> > > +   of a codec frame together with its size
in the framecode table. This
> > > +   would allow us to store the size of a
frame without any redundancy.
> > > +   Thus effectivly avoiding the overhead
small frames cause.
> > 
> > At the cost of efficient decoding... If such a
horrid proposal were
> 
> Id guess the overhead of working with 6 byte frames and
decoding
> a framecode before each takes more time than copying 60
bytes. Also depending
> on the media this comes from 10% extra disk IO should
beat 1 copy pass in
> slowness.
> 
> And i dont give a damn about zero copy for a 400-1000
byte / sec codec!
> also so much for bad compression ...

I'm not worried about zero copy for such a low bitrate
stream. My
concern is that idiots could use this nasty header
compression for
20mbit/sec h264 for no good reason, just because it
"sounds good".

Rich
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, Feb 11, 2008 at 10:41:51PM -0500, Rich Felker
wrote:
> On Tue, Feb 12, 2008 at 12:33:22AM +0100, michael
wrote:
> > 
> > Log:
> > 3 more issues which have come up in the past but
have IIRC never
> > been resolved.
> > 
> > --- docs/nutissues.txt	(original)
> > +++ docs/nutissues.txt	Tue Feb 12 00:33:21 2008
> > +Issue small-frames
> > +------------------
> > +The original intent of nut frames was that 1
container frame == 1 codec
> > +frame. Though this does not seem to be explicitly
written in nut.txt.
> 
> It is.
> 
> > +Also it is inefficient for very small frames,
AMR-NB for example has 6-32
> > +bytes per frame.
> 
> I don't think anyone really cares that much about
efficiency when
> storing shit codecs in NUT. Obviously any good codec
will use large
> frame sizes or compression will not be good.

Somebody correct me if I am wrong, but I think AMR currently
is the best
quality speech codec...

Diego
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 12:12:52AM -0500, Rich Felker
wrote:
> On Tue, Feb 12, 2008 at 05:51:44AM +0100, Michael
Niedermayer wrote:
> > > I'm
> > > absolutely against any proposal that
precludes implementations from
> > > performing zero-copy decoding from the stream
buffer.
> > 
> > Well if you have a buffer in which the frame is,
just write the header
> > before it. If not just write the header and then
(f)read() the rest.
> 
> I'm envisioning a model where the demuxer directly
exports frames from
> their original location in the stream buffer. It's
likely that this
> buffer needs to be treated as read-only. It might be
mmapped from a
> file (in which case writing would invoke COW) 

COW would be invoked only if the kernel tries to keep the
pages in some
sort of cache, otherwise there is absolutely no need for
COW.


> or the previous frame
> might not yet be released by the caller (in which case,
writing the
> header would clobber the contents of the previous
frame) -- imagine
> for instance that we need to decode the next audio
frame to buffer it,
> but aren't ready to decode the previous video frame
yet.

I wonder if anyone will ever implement such a mmap system,
at least it
will fail with files larger than 2gb on x86-linux and other
sizes for other
platforms. Also such a system would need carefull use of
madvise() to ensure
the kernel doesnt read pages through (slow) page faults or
uses random
guessing of what pages can be discarded.


> 
> The reason I'm against this whole header compression
thing is that it
> puts lots of nasty restraints on the demuxer
implementation that
> preclude efficient design. 

The header compression only puts restraints on a AFAIK
hypothetic design
not one actually existing?
And its not just the demuxer but the whole surrounding API
which would
have to be designed for such mmap zero copy.


> The nasty cases can be special-cased to
> minimize their cost, but the tradeoff is implementation
size and
> complexity. The only options I'm willing to consider
(and which I
> still dislike) are:
> 

> A. Only allow header elision (a more correct name than
compression)
> for frames below a very small size limit (e.g. small
enough that the
> compression makes at least a 1% difference to space
requirements).

That would be a compromise though iam not happy about it at
all, its another
one of these idiotic philosophical restrictions. If the
person muxing
decides that 0.5% overhead is more important than 0.5% time
spend in
an HYPOHETICAL demuxer implementation in his specific
situation. Than
this is his decission and i wonder from where you take the
arogance to
think you know better what is best for everyone else. And
these are
people who might want to apply nut in situations you havent
thought about.
The 0.5% time might be available, the 0.5% bandwidth maybe
is not, and
again this are 0.5% time for an implementation you have
envisioned not one
actually existing.


> 
> B. Require that the elided header fit in the same
number of bytes (or
> fewer) than the minimal NUT frame header size for the
corresponding
> framecode. This would allow overwrite-in-place without
clobbering
> other frames, but still the COW issue remains which I
don't like...
> 
> > > > +Issue small-frames
> > > > +------------------
> > > > +The original intent of nut frames was
that 1 container frame == 1 codec
> > > > +frame. Though this does not seem to be
explicitly written in nut.txt.
> > > 
> > > It is.
> > 
> > I thought so too but i searched, and didnt find it
so where is it?
> 
> It's at least in nut-english.txt 

It should be in nut.txt as well ...


> 
> > > > +Also it is inefficient for very small
frames, AMR-NB for example has 6-32
> > > > +bytes per frame.
> > > 
> > > I don't think anyone really cares that much
about efficiency when
> > > storing 
> > 
> > I do, others likely care about 10% overhead as
well.
> 
> Care about it for what applications? For storing a
master file in some
> odd barely-compressed audio form, I wouldn't care. For
streaming radio
> or movie rips, of course it would matter, but I think
someone would
> choose an appropriate codec for such applications..

Which codec would that be? (we are talking about
5-10kbit/sec).


> 
> > > shit codecs
> > 
> > I think this applies more to audio codes, if it
limited to coding shit
> > i wont bother and gladly leave it alone. ;)
> > 
> > > in NUT. Obviously any good codec will use
large
> > > frame sizes 
> > 
> > no
> 
> Is there any audio codec of real interest with such
small frame sizes?

AMR-NB has 6-32 byte frames see libavcodec/libamr.c
qcelp has 4-35 byte frames see soc/qcelp/qcelp_parser.c


> 
> > > > +Solutions:
> > > > +A. Enforce 1 container frame == 1 codec
frame even if it causes 10% overhead.
> > > 
> > > Yes.
> > 
> > Vote noted. And my veto as well unless option D is
selected which would
> > also include A.
> 
> It's frustrating how you intend to "veto"
decisions that were made 4
> years ago now when we're trying to finally get NUT to
see the light.

Iam not vetoing the decisson of 1frame=1frame for normal
sized frames,
which was what we decided upon. I dont think tiny frames
have been considered.
And we never planned to enforce 1frame=1frame for PCM. which
is maybe
the reason why its not in the spec? A carelessly written
rule would have
fatal consequences for PCM.


> 
> > > > +B. Allow multiple frames as long as the
whole packet is less than some
> > > > +   fixed minimum in bytes (like
256byte)
> > > 
> > > Very very bad. Demuxing requires a
codec-specific framer.
> > > 
> > > > +C. Allow multiple frames as long as the
whole packet is less than some
> > > > +   fixed minimum in bytes (like
256byte) and the codec uses a constant
> > > > +   framesize in samples.
> > > 
> > > This does not help.
> > 
> > help what?
> 
> I meant that I don't see how it's any less offensive
than option B.

So it is offensive.


> The fundamental problem is still there: inability to
demux and seek to
> individual frames without codec incest. It's sad that
we've gotten to
> the point where 1 byte of overhead per frame is too
much.. Even OGG
> and MKV have at least that much!

try mpeg-ps ;)


> 
> > > > +D. Use header compression, that is
allow to store the first (few) bytes
> > > > +   of a codec frame together with its
size in the framecode table. This
> > > > +   would allow us to store the size of
a frame without any redundancy.
> > > > +   Thus effectivly avoiding the
overhead small frames cause.
> > > 
> > > At the cost of efficient decoding... If such
a horrid proposal were
> > 
> > Id guess the overhead of working with 6 byte
frames and decoding
> > a framecode before each takes more time than
copying 60 bytes. Also depending
> > on the media this comes from 10% extra disk IO
should beat 1 copy pass in
> > slowness.
> > 
> > And i dont give a damn about zero copy for a
400-1000 byte / sec codec!
> > also so much for bad compression ...
> 
> I'm not worried about zero copy for such a low bitrate
stream. My
> concern is that idiots could use this nasty header
compression for
> 20mbit/sec h264 for no good reason, just because it
"sounds good".

And if they did, what bad would happen? COW copying every
20th page?
what overhead does that cause?
Also even if decoding such a file were 2% slower, its one
file which
has been muxed by someone intentionally ignoring
recommanditions
and defaults. Such a person will make many more such
decissions and
likely just use WMV or something. I think the header elision
would be your
least concern with such a file.

[...]
-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

I have never wished to cater to the crowd; for what I know
they do not
approve, and what they approve I do not know. -- Epicurus

_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, 11 Feb 2008 22:41:51 -0500
Rich Felker <daliasaerifal.cx> wrote:

> I don't think anyone really cares that much about
efficiency when
> storing shit codecs in NUT. Obviously any good codec
will use large
> frame sizes or compression will not be good.

Ever heard of VoIP? In such applications it is essential to
have as
little latency as possible. Do the maths and you will see
that you need
a codec frame size between 10 and 30ms to get a decent
experience. At
8KHz that's just a couple 100 samples.

	Albeu

PS: I don't dispute the fact that putting such codec in nut
doesn't
make much sense, just that these codec are not useless shit
as you
seems to imply.

_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 09:01:04AM +0100, Diego Biurrun
wrote:
> > > +Also it is inefficient for very small
frames, AMR-NB for example has 6-32
> > > +bytes per frame.
> > 
> > I don't think anyone really cares that much about
efficiency when
> > storing shit codecs in NUT. Obviously any good
codec will use large
> > frame sizes or compression will not be good.
> 
> Somebody correct me if I am wrong, but I think AMR
currently is the best
> quality speech codec...

No, PCM is the best quality speech codec.   
AMR is good for shit-quality applications like telephone,
but not for
anything where you want the audio to actually be
recognizable.

Rich
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 01:42:33PM +0100, Michael
Niedermayer wrote:
> On Tue, Feb 12, 2008 at 12:12:52AM -0500, Rich Felker
wrote:
> > On Tue, Feb 12, 2008 at 05:51:44AM +0100, Michael
Niedermayer wrote:
> > > > I'm
> > > > absolutely against any proposal that
precludes implementations from
> > > > performing zero-copy decoding from the
stream buffer.
> > > 
> > > Well if you have a buffer in which the frame
is, just write the header
> > > before it. If not just write the header and
then (f)read() the rest.
> > 
> > I'm envisioning a model where the demuxer directly
exports frames from
> > their original location in the stream buffer. It's
likely that this
> > buffer needs to be treated as read-only. It might
be mmapped from a
> > file (in which case writing would invoke COW) 
> 
> COW would be invoked only if the kernel tries to keep
the pages in some
> sort of cache, otherwise there is absolutely no need
for COW.

Even if not a copy, there'd be a page fault, which is
expensive in
itself.

> > or the previous frame
> > might not yet be released by the caller (in which
case, writing the
> > header would clobber the contents of the previous
frame) -- imagine
> > for instance that we need to decode the next audio
frame to buffer it,
> > but aren't ready to decode the previous video
frame yet.
> 
> I wonder if anyone will ever implement such a mmap
system, at least it
> will fail with files larger than 2gb on x86-linux and
other sizes for other
> platforms.

Nope, you only map the section of the stream that's still in
use.
Normally this will be several megs at most.

> Also such a system would need carefull use of madvise()
to ensure
> the kernel doesnt read pages through (slow) page faults
or uses random
> guessing of what pages can be discarded.

Indeed.

> > The reason I'm against this whole header
compression thing is that it
> > puts lots of nasty restraints on the demuxer
implementation that
> > preclude efficient design. 
> 
> The header compression only puts restraints on a AFAIK
hypothetic design
> not one actually existing?
> And its not just the demuxer but the whole surrounding
API which would
> have to be designed for such mmap zero copy.

Precluding the optimal design is bad practice even if the
optimal
design has not yet been implemented anywhere. Also keep in
mind that
what I said does NOT only apply to the mmap design but to a
nice clean
read-based design with low copying that probably IS similar
to various
current implementations.

> > The nasty cases can be special-cased to
> > minimize their cost, but the tradeoff is
implementation size and
> > complexity. The only options I'm willing to
consider (and which I
> > still dislike) are:
> > 
> 
> > A. Only allow header elision (a more correct name
than compression)
> > for frames below a very small size limit (e.g.
small enough that the
> > compression makes at least a 1% difference to
space requirements).
> 
> That would be a compromise though iam not happy about
it at all, its another
> one of these idiotic philosophical restrictions. If the
person muxing
> decides that 0.5% overhead is more important than 0.5%
time spend in
> an HYPOHETICAL demuxer implementation

No, I already told you the issue here is not just time but
buffer
size. Requiring an unboundedly large buffer in addition to
the main
stream buffer is a potentially huge burden leading to all
sorts of
robustness issues. As long as the buffer size is bounded and
small, no
one really cares.

> in his specific situation. Than
> this is his decission and i wonder from where you take
the arogance to
> think you know better what is best for everyone else.
And these are

It's NOT the business of the person making the file to put
restrictions on the process reading the file. This has been
one of my
key issues for NUT since the beginning. Just because
idiot_who_makes_file thinks the only purpose of the file is
to watch
it with player_x doesn't mean they're exempt from the
features of NUT
that make the file nicely usable for editing, optimal
seeking, etc.

> > > > > +Issue small-frames
> > > > > +------------------
> > > > > +The original intent of nut frames
was that 1 container frame == 1 codec
> > > > > +frame. Though this does not seem
to be explicitly written in nut.txt.
> > > > 
> > > > It is.
> > > 
> > > I thought so too but i searched, and didnt
find it so where is it?
> > 
> > It's at least in nut-english.txt 
> 
> It should be in nut.txt as well ...

If it's not feel free to add it, but I think it belongs in
the
information about codecs, since the definition of a frame is
pretty
codec-specific. Expressing the abstract idea in the
semantic
requirements section of nut.txt would be nice though!

> > > > > +Also it is inefficient for very
small frames, AMR-NB for example has 6-32
> > > > > +bytes per frame.
> > > > 
> > > > I don't think anyone really cares that
much about efficiency when
> > > > storing 
> > > 
> > > I do, others likely care about 10% overhead
as well.
> > 
> > Care about it for what applications? For storing a
master file in some
> > odd barely-compressed audio form, I wouldn't care.
For streaming radio
> > or movie rips, of course it would matter, but I
think someone would
> > choose an appropriate codec for such
applications..
> 
> Which codec would that be? (we are talking about
5-10kbit/sec).

Vorbis.

> > > > shit codecs
> > > 
> > > I think this applies more to audio codes, if
it limited to coding shit
> > > i wont bother and gladly leave it alone. ;)
> > > 
> > > > in NUT. Obviously any good codec will
use large
> > > > frame sizes 
> > > 
> > > no
> > 
> > Is there any audio codec of real interest with
such small frame sizes?
> 
> AMR-NB has 6-32 byte frames see libavcodec/libamr.c
> qcelp has 4-35 byte frames see
soc/qcelp/qcelp_parser.c

And is there any interest in these codecs aside from
watching files
generated by camera phones and transcoding them to something
sane?

> Iam not vetoing the decisson of 1frame=1frame for
normal sized frames,
> which was what we decided upon. I dont think tiny
frames have been considered.
> And we never planned to enforce 1frame=1frame for PCM.
which is maybe
> the reason why its not in the spec? A carelessly
written rule would have
> fatal consequences for PCM.

For PCM, there's no seeking issue because one always knows
how to seek
to an arbitrary sample in PCM 'frames'. In this case I would
just
recommend having a SHOULD clause that PCM frames SHOULD be
same or
shorter duration than the typical frame durations for other
streams in
the file (as a way of keeping interleaving clean and aiding
players
that don't want to do sample-resolution seeking inside PCM
frames).

> > I'm not worried about zero copy for such a low
bitrate stream. My
> > concern is that idiots could use this nasty header
compression for
> > 20mbit/sec h264 for no good reason, just because
it "sounds good".
> 
> And if they did, what bad would happen? COW copying
every 20th page?
> what overhead does that cause?
> Also even if decoding such a file were 2% slower, its
one file which

COW is not the worst-case. If you read my original post, the
common
and worst case is NOT necessarily mmap but rather when you
can't
clobber the source stream because the calling process still
has a
valid reference to previous frames in the stream which
cannot be
clobbered. This would require a memcpy of the ENTIRE FRAME
which would
easily make performance 10% slower or much worse.

If you really want this header-elision, I'm willing to
consider it for
SMALL frames. But please don't flame about wanting to
support it for
large frames where it's absolutely useless and has lots of
practical
problems for efficient implementations!

Rich
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Tue, Feb 12, 2008 at 03:24:34PM +0100, Alban Bedel
wrote:
> On Mon, 11 Feb 2008 22:41:51 -0500
> Rich Felker <daliasaerifal.cx> wrote:
> 
> > I don't think anyone really cares that much about
efficiency when
> > storing shit codecs in NUT. Obviously any good
codec will use large
> > frame sizes or compression will not be good.
> 
> Ever heard of VoIP? In such applications it is
essential to have as
> little latency as possible. Do the maths and you will
see that you need
> a codec frame size between 10 and 30ms to get a decent
experience. At
> 8KHz that's just a couple 100 samples.

At 8khz you might as well send the contents of /dev/urandom
instead of
a signal...

> PS: I don't dispute the fact that putting such codec in
nut doesn't
> make much sense, just that these codec are not useless
shit as you
> seems to imply.

=P

Rich
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu
https://lists.mplayerhq.hu/mailman/listinfo/nut-devel