Info packets in NUT stream (spec bugs?)

On Thu, Nov 16, 2006 at 12:12:53AM +0100, Michael
Niedermayer wrote:
> On Wed, Nov 15, 2006 at 11:29:12PM +0100, Michael
Niedermayer wrote:
> > also you dont need syncpoints over TCP, its a
waste of bandwidth ...
> 
> heres a simple nut streaming spec proposal (whats not
explicitly overriden
> is as defined by the current nut spec)
> 
> streaming in general:
> * nut is transmitted raw (no extra headers)
> * info may change and be transmitted anywhere
> * the client can (if supported by the server do seeks,
ask for header or
>   info retransmitts, and ask for intra refresh
(keyframe), and request
>   a syncpoint)
> 
> streaming over error free channels:
> * main, stream and info headers MUST NOT be repeateded
unless requested by
>   the client and supported by the server
> * there is only 1 syncpoint after the headers unless
server side seeking
>   is supported and used
> -----
> streaming over channels with packet loss:
> * packet boundaries are aligned with nut frame starts
where possible
> * packet boundaries are aligned with slice starts where
possible
> * FLAG_CODED_PTS SHOULD be set for all frames which use
a frame from
>   a previous packet to calculate the pts
> * if the packet starts with a framecode then it MUST
have FLAG_CHECKSUM
>   set (that way packets which continue past packets can
be distinguished
>   from packets starting new frames)


This doesn't seem like a streaming protocol, more like a
butchering of the 
nut spec (most obvious, allowing syncpoints not be written -
this is an 
obvious violation of max_distance). This doesn't necessarily
make it a bad 
idea (if you are completely error free, and not seeking,
syncpoints are 
redundant and non-trivial to calculate for their back_ptr),
but it does 
not seem like something to be written in a seperate document
and being 
called a "streaming protocol" - it's a modified
NUT format spec for 
streaming... And I think it's probably a bad idea to have
several modified 
specs (it's somewhat like mpeg-ts, mpeg-ps, mpeg-bla).

- ods15
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

Hi

On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded Shimon wrote:
> On Thu, Nov 16, 2006 at 12:12:53AM +0100, Michael
Niedermayer wrote:
> > On Wed, Nov 15, 2006 at 11:29:12PM +0100, Michael
Niedermayer wrote:
> > > also you dont need syncpoints over TCP, its a
waste of bandwidth ...
> > 
> > heres a simple nut streaming spec proposal (whats
not explicitly overriden
> > is as defined by the current nut spec)
> > 
> > streaming in general:
> > * nut is transmitted raw (no extra headers)
> > * info may change and be transmitted anywhere
> > * the client can (if supported by the server do
seeks, ask for header or
> >   info retransmitts, and ask for intra refresh
(keyframe), and request
> >   a syncpoint)
> > 
> > streaming over error free channels:
> > * main, stream and info headers MUST NOT be
repeateded unless requested by
> >   the client and supported by the server
> > * there is only 1 syncpoint after the headers
unless server side seeking
> >   is supported and used
> > -----
> > streaming over channels with packet loss:
> > * packet boundaries are aligned with nut frame
starts where possible
> > * packet boundaries are aligned with slice starts
where possible
> > * FLAG_CODED_PTS SHOULD be set for all frames
which use a frame from
> >   a previous packet to calculate the pts
> > * if the packet starts with a framecode then it
MUST have FLAG_CHECKSUM
> >   set (that way packets which continue past
packets can be distinguished
> >   from packets starting new frames)
> 
> 
> This doesn't seem like a streaming protocol, more like
a butchering of the 
> nut spec (most obvious, allowing syncpoints not be
written - this is an 
> obvious violation of max_distance). This doesn't
necessarily make it a bad 
> idea (if you are completely error free, and not
seeking, syncpoints are 
> redundant and non-trivial to calculate for their
back_ptr), but it does 
> not seem like something to be written in a seperate
document and being 
> called a "streaming protocol" - it's a
modified NUT format spec for 
> streaming... And I think it's probably a bad idea to
have several modified 
> specs (it's somewhat like mpeg-ts, mpeg-ps, mpeg-bla).

no mpeg-ts and mpeg-ps are much more different then what i
proposed

the problem simply is that different environments/systems
have different
requirements, if there is packet loss then some additional
things are needed
if not they are a waste, surely some minimum error recovery
stuff can always
be added but then again you where the one saying you will
ignore the spec
and not repeat headers as its not needed in your case! so
what do you
argue against? seems like its what you yourself wanted?
if you buther the spec by omiting repeated header then
butcher it enough
so it cannot be played if dumped raw otherwise such broken
streams will
spread not neccessarily originating from stream dumps but
maybe rather
people who like to safe 0.1% space

and then info packets, i will not accpet info to be
unfindable, that is
files stored on disk must have at least

O(number of distinct info packets) to enumerate all distinct
info packets
O(file size) is not ok
simply adding a single back pointer to midstream info
packets would help
alot here

also very nice would be
O(number of apllicable info packets * log n) to find info
for a specific time
though we currently dont gurantee that

[...]
-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or
copy any book
Today you'd get arrested for mere telling someone where the
library is
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Fri, Nov 17, 2006 at 01:02:22PM +0100, Michael
Niedermayer wrote:
> On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded Shimon
wrote:
> > This doesn't seem like a streaming protocol, more
like a butchering of the 
> > nut spec (most obvious, allowing syncpoints not be
written - this is an 
> > obvious violation of max_distance). This doesn't
necessarily make it a bad 
> > idea (if you are completely error free, and not
seeking, syncpoints are 
> > redundant and non-trivial to calculate for their
back_ptr), but it does 
> > not seem like something to be written in a
seperate document and being 
> > called a "streaming protocol" - it's a
modified NUT format spec for 
> > streaming... And I think it's probably a bad idea
to have several modified 
> > specs (it's somewhat like mpeg-ts, mpeg-ps,
mpeg-bla).
> 
> no mpeg-ts and mpeg-ps are much more different then
what i proposed
> 
> the problem simply is that different
environments/systems have different
> requirements, if there is packet loss then some
additional things are needed
> if not they are a waste, surely some minimum error
recovery stuff can always
> be added but then again you where the one saying you
will ignore the spec
> and not repeat headers as its not needed in your case!
so what do you
> argue against? seems like its what you yourself wanted?

I don't consider it a butchering of the spec to omit
repeated headers in 
the case of realtime streaming as it is an impossible rule
to inforce and 
illogical in the case of realtime streaming. The only way to
truely 
enforce this rule in realtime streams is by making all 3
header copies 
right at the begginning, with no frame inbetween - otherwise
the user can 
stop the dumping immediately after it begun and it would be
impossible to 
get all 3 copies even if the streamer repeated headers.
Even if they were repeated - they would not be useful for
finding in 
dumped files as they would not be in 2^n locations (unless
you expect the 
streaming server to start doing calculations for EACH
client..)

It's a butchering of the spec only because you can
semantically consider 
the file truncated and incomplete - in your suggest of
max_distance 
butchering, you're explicitly butchering the spec to the
point that 
compliant demuxers will not be able to play it without
different modes for 
streaming and files.

> if you buther the spec by omiting repeated header then
butcher it enough
> so it cannot be played if dumped raw otherwise such
broken streams will
> spread not neccessarily originating from stream dumps
but maybe rather
> people who like to safe 0.1% space

Well, I somewhat doubt that will actually happen, but I
accept the
argument...

> and then info packets, i will not accpet info to be
unfindable, that is
> files stored on disk must have at least
> 
> O(number of distinct info packets) to enumerate all
distinct info packets
> O(file size) is not ok
> simply adding a single back pointer to midstream info
packets would help
> alot here

I have no idea for a solution. Your proposed solution
doesn't work as 
there is no way to find the last info packet. even with your
stuffing 
info packet in headers idea: 1. how would you find the
headers? 2. who 
said the headers are necessarily after the last info packet
in the file? 
Our main assumption is that in realtime stream dumping, they
can be 
cut/truncated at absoloutely any arbitrary point in the
file.



BTW, if we ultimately decide on disallowing mid-stream info
packets, we 
need to remove these 2 parts from the spec:
---
If a demuxer has seen several info packets with the same
chapter_id and
stream_id then it MUST ignore all but the one with the
highest position in
the file

demxuxers SHOULD NOT search the whole file for info packets
---

As they make no sense without them (all headers must be
repeated 
identically after all main headers)

BTW2, If we decide we do allow mis-stream info packets, then
this rule 
somewhat fails:
---
    chapter_id n MUST not be used unless there are at least
n chapters in the
    file
---

As a streamer would start at chapter_id=1, and linearly grow
- you can 
join at any arbitrary point, even when chapter_id=5926.

- ods15
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Sun, Nov 19, 2006 at 06:58:59PM +0200, Oded Shimon wrote:
> On Fri, Nov 17, 2006 at 01:02:22PM +0100, Michael
Niedermayer wrote:
> > On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded
Shimon wrote:
> > > This doesn't seem like a streaming protocol,
more like a butchering of the 
> > > nut spec (most obvious, allowing syncpoints
not be written - this is an 
> > > obvious violation of max_distance). This
doesn't necessarily make it a bad 
> > > idea (if you are completely error free, and
not seeking, syncpoints are 
> > > redundant and non-trivial to calculate for
their back_ptr), but it does 
> > > not seem like something to be written in a
seperate document and being 
> > > called a "streaming protocol" -
it's a modified NUT format spec for 
> > > streaming... And I think it's probably a bad
idea to have several modified 
> > > specs (it's somewhat like mpeg-ts, mpeg-ps,
mpeg-bla).
> > 
> > no mpeg-ts and mpeg-ps are much more different
then what i proposed
> > 
> > the problem simply is that different
environments/systems have different
> > requirements, if there is packet loss then some
additional things are needed
> > if not they are a waste, surely some minimum error
recovery stuff can always
> > be added but then again you where the one saying
you will ignore the spec
> > and not repeat headers as its not needed in your
case! so what do you
> > argue against? seems like its what you yourself
wanted?
> 
> I don't consider it a butchering of the spec to omit
repeated headers in 
> the case of realtime streaming as it is an impossible
rule to inforce and 
> illogical in the case of realtime streaming. The only
way to truely 
> enforce this rule in realtime streams is by making all
3 header copies 
> right at the begginning, with no frame inbetween -
otherwise the user can 
> stop the dumping immediately after it begun and it
would be impossible to 
> get all 3 copies even if the streamer repeated headers.
> Even if they were repeated - they would not be useful
for finding in 
> dumped files as they would not be in 2^n locations
(unless you expect the 
> streaming server to start doing calculations for EACH
client..)
> 
> It's a butchering of the spec only because you can
semantically consider 
> the file truncated and incomplete - in your suggest of
max_distance 
> butchering, you're explicitly butchering the spec to
the point that 
> compliant demuxers will not be able to play it without
different modes for 
> streaming and files.

You realize, right, that the issue here is that a truncated
nut file
is not a valid nut file, according to the spec? If we
acknowledge the
existence of truncated files but note that they do not
constitute
"complete" files (by virtue of missing header
copies and missing
index) the problem goes away, I think.

> > if you buther the spec by omiting repeated header
then butcher it enough
> > so it cannot be played if dumped raw otherwise
such broken streams will
> > spread not neccessarily originating from stream
dumps but maybe rather
> > people who like to safe 0.1% space
> 
> Well, I somewhat doubt that will actually happen, but I
accept the
> argument...

It's quite easy to say that incomplete streams are valid but
incomplete, and further to require that IF the stream
contains and
index, it MUST be complete (i.e. include header copies).

> BTW, if we ultimately decide on disallowing mid-stream
info packets, we 
> need to remove these 2 parts from the spec:
> ---
> If a demuxer has seen several info packets with the
same chapter_id and
> stream_id then it MUST ignore all but the one with the
highest position in
> the file
> 
> demxuxers SHOULD NOT search the whole file for info
packets
> ---

I'm find with keeping this possibility as long as the last
condition
(don't search) is made abundantly clear. But really I don't
care. It's
a stupid bikeshed issue.

> BTW2, If we decide we do allow mis-stream info packets,
then this rule 
> somewhat fails:
> ---
>     chapter_id n MUST not be used unless there are at
least n chapters in the
>     file
> ---
> 
> As a streamer would start at chapter_id=1, and linearly
grow - you can 
> join at any arbitrary point, even when chapter_id=5926.

IMO this should remain illegal. You can use negative
chapter_id or
something.. In any case the info system is probably not
ideal for the
sort of streaming uses you're describing. There are much
better out of
band ways to accomplish this info delivery..

Rich

_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

Hi

On Sun, Nov 19, 2006 at 06:58:59PM +0200, Oded Shimon wrote:
> On Fri, Nov 17, 2006 at 01:02:22PM +0100, Michael
Niedermayer wrote:
> > On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded
Shimon wrote:
> > > This doesn't seem like a streaming protocol,
more like a butchering of the 
> > > nut spec (most obvious, allowing syncpoints
not be written - this is an 
> > > obvious violation of max_distance). This
doesn't necessarily make it a bad 
> > > idea (if you are completely error free, and
not seeking, syncpoints are 
> > > redundant and non-trivial to calculate for
their back_ptr), but it does 
> > > not seem like something to be written in a
seperate document and being 
> > > called a "streaming protocol" -
it's a modified NUT format spec for 
> > > streaming... And I think it's probably a bad
idea to have several modified 
> > > specs (it's somewhat like mpeg-ts, mpeg-ps,
mpeg-bla).
> > 
> > no mpeg-ts and mpeg-ps are much more different
then what i proposed
> > 
> > the problem simply is that different
environments/systems have different
> > requirements, if there is packet loss then some
additional things are needed
> > if not they are a waste, surely some minimum error
recovery stuff can always
> > be added but then again you where the one saying
you will ignore the spec
> > and not repeat headers as its not needed in your
case! so what do you
> > argue against? seems like its what you yourself
wanted?
> 
> I don't consider it a butchering of the spec to omit
repeated headers in 
> the case of realtime streaming as it is an impossible
rule to inforce and 
> illogical in the case of realtime streaming. The only
way to truely 
> enforce this rule in realtime streams is by making all
3 header copies 
> right at the begginning, with no frame inbetween -
otherwise the user can 
> stop the dumping immediately after it begun and it
would be impossible to 
> get all 3 copies even if the streamer repeated headers.
> Even if they were repeated - they would not be useful
for finding in 
> dumped files as they would not be in 2^n locations
(unless you expect the 
> streaming server to start doing calculations for EACH
client..)

well, if you want to be able to start playing in the middle
(your comment
strongly suggests that) you either need to repeat headers
often or transmit
them out of band, or allow clients to seeks to past data
somehow ...

also i think we once said that timestamps must start at 0,
you seem to break
that too, iam not sure if theres anything in the spec which
forbids it or
if theres anything which would break, iam just mentioning it
...

and anyway the not duplicated headers are a minor issue
about which i dont
really care as long as its clear that this is not ok in
normal nut files
it of course can happen in truncated or otherwise damaged
files and a
demuxer still should handle that sanely ...


[...]
> > if you buther the spec by omiting repeated header
then butcher it enough
> > so it cannot be played if dumped raw otherwise
such broken streams will
> > spread not neccessarily originating from stream
dumps but maybe rather
> > people who like to safe 0.1% space
> 
> Well, I somewhat doubt that will actually happen, but I
accept the
> argument...

you want to omit them in your streams IIRC because of the
space saving so
why shouldnt others too?


> 
> > and then info packets, i will not accpet info to
be unfindable, that is
> > files stored on disk must have at least
> > 
> > O(number of distinct info packets) to enumerate
all distinct info packets
> > O(file size) is not ok
> > simply adding a single back pointer to midstream
info packets would help
> > alot here
> 
> I have no idea for a solution. Your proposed solution
doesn't work as 
> there is no way to find the last info packet. even with
your stuffing 
> info packet in headers idea: 1. how would you find the
headers? 2. who 
> said the headers are necessarily after the last info
packet in the file? 
> Our main assumption is that in realtime stream dumping,
they can be 
> cut/truncated at absoloutely any arbitrary point in the
file.

1. every midstream info packet (and only them) MUST have a
pointer to the
   previous non redundant/repeated midstream info packet
2. the distance between midstream info packets MUST be <=
C*max_distance
   unless that is impossible (too large stream header /
frame) in which
   case the distance MUST be as small as possible for the
specific case
3. a info packet MUST either be part of the normal info
packets or it
   MUST be repeated like described in 2. within the area to
which it applies
   and it MUST be in that area at least once


> 
> 
> 
> BTW, if we ultimately decide on disallowing mid-stream
info packets, we 
> need to remove these 2 parts from the spec:
> ---
> If a demuxer has seen several info packets with the
same chapter_id and
> stream_id then it MUST ignore all but the one with the
highest position in
> the file
> 
> demxuxers SHOULD NOT search the whole file for info
packets
> ---
> 
> As they make no sense without them (all headers must be
repeated 
> identically after all main headers)

leaving them does no harm droping them changes the frozen
spec again ...
it also makes adding mid stream info packets at a later time
much
harder if the rules are droped now


> 
> BTW2, If we decide we do allow mis-stream info packets,
then this rule 
> somewhat fails:
> ---
>     chapter_id n MUST not be used unless there are at
least n chapters in the
>     file
> ---
> 
> As a streamer would start at chapter_id=1, and linearly
grow - you can 
> join at any arbitrary point, even when chapter_id=5926.

is a arbitrary cut part (both begin and end cut) always a
valid nut file?
if yes then every file is a valid nut file as it could be
the data of a
single frame

i would rather say that the whole stream from its true begin
forms a valid
nut file and when you start cutting it becomes a damaged
stream which
must be dealt with correctly by demuxers if all headers and
one syncpoint is
available


[...]

-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or
copy any book
Today you'd get arrested for mere telling someone where the
library is
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, Nov 20, 2006 at 03:29:29PM +0100, Michael
Niedermayer wrote:
> well, if you want to be able to start playing in the
middle (your comment
> strongly suggests that) you either need to repeat
headers often or transmit
> them out of band, or allow clients to seeks to past
data somehow ...

His idea is that the server would semi-dynamically generate
a stream
by first writing the headers when a client connects, then
copying
everything else from disk. This is what I had in mind too.

> also i think we once said that timestamps must start at
0, you seem to break

Absolutely not! If this were required then you could not
split long
movies into multiple files to split across media without
losing
information (and mildly breaking a/v sync since it's
impossible for
all streams to start exactly at 0 after splitting a file
unless you
choose a cut point divisible by all frame durations...).

> that too, iam not sure if theres anything in the spec
which forbids it or
> if theres anything which would break, iam just
mentioning it ...

Starting at arbitrary timestamp is totally legal and it was
always my
intent for it to be legal. Obviously streaming is not
possible either
if you don't allow for this..

> and anyway the not duplicated headers are a minor issue
about which i dont
> really care as long as its clear that this is not ok in
normal nut files
> it of course can happen in truncated or otherwise
damaged files and a
> demuxer still should handle that sanely ...

Agree.

> > > if you buther the spec by omiting repeated
header then butcher it enough
> > > so it cannot be played if dumped raw
otherwise such broken streams will
> > > spread not neccessarily originating from
stream dumps but maybe rather
> > > people who like to safe 0.1% space
> > 
> > Well, I somewhat doubt that will actually happen,
but I accept the
> > argument...
> 
> you want to omit them in your streams IIRC because of
the space saving so
> why shouldnt others too?

Well in Oded's case it happens that they're utterly useless,
but as
long as we follow what you and I said above I don't think
there's a
problem.

> > I have no idea for a solution. Your proposed
solution doesn't work as 
> > there is no way to find the last info packet. even
with your stuffing 
> > info packet in headers idea: 1. how would you find
the headers? 2. who 
> > said the headers are necessarily after the last
info packet in the file? 
> > Our main assumption is that in realtime stream
dumping, they can be 
> > cut/truncated at absoloutely any arbitrary point
in the file.
> 
> 1. every midstream info packet (and only them) MUST
have a pointer to the
>    previous non redundant/repeated midstream info
packet
> 2. the distance between midstream info packets MUST be
<= C*max_distance
>    unless that is impossible (too large stream header /
frame) in which
>    case the distance MUST be as small as possible for
the specific case
> 3. a info packet MUST either be part of the normal info
packets or it
>    MUST be repeated like described in 2. within the
area to which it applies
>    and it MUST be in that area at least once

Seems like the sanest proposal for this so far, but....
introduces
huge ugly complexity to implementation.

> > BTW, if we ultimately decide on disallowing
mid-stream info packets, we 
> > need to remove these 2 parts from the spec:
> > ---
> > If a demuxer has seen several info packets with
the same chapter_id and
> > stream_id then it MUST ignore all but the one with
the highest position in
> > the file
> > 
> > demxuxers SHOULD NOT search the whole file for
info packets
> > ---
> > 
> > As they make no sense without them (all headers
must be repeated 
> > identically after all main headers)
> 
> leaving them does no harm droping them changes the
frozen spec again ...
> it also makes adding mid stream info packets at a later
time much
> harder if the rules are droped now

Yes, let's not harm syntactic extensibility. I doubt the
sanity of
this mid-stream info at all, but...

> > BTW2, If we decide we do allow mis-stream info
packets, then this rule 
> > somewhat fails:
> > ---
> >     chapter_id n MUST not be used unless there are
at least n chapters in the
> >     file
> > ---
> > 
> > As a streamer would start at chapter_id=1, and
linearly grow - you can 
> > join at any arbitrary point, even when
chapter_id=5926.
> 
> is a arbitrary cut part (both begin and end cut) always
a valid nut file?
> if yes then every file is a valid nut file as it could
be the data of a
> single frame
> 
> i would rather say that the whole stream from its true
begin forms a valid
> nut file and when you start cutting it becomes a
damaged stream which
> must be dealt with correctly by demuxers if all headers
and one syncpoint is
> available

An arbitrary dd-cut nut file is not a valid complete not
file, and not
valid at all if the beginning has been cut. On the other
hand, a file
cut with a proper cutting tool SHOULD be valid, e.g. if
you're cutting
a movie to split it across undersized media (cdr). Requiring
the
chapters to be renumbered after cutting is particularly ugly
and
destructive of semantic information. I'd like to request a
minor
revision: instead of requiring positive chater id's to start
from 1,
just require them to be contiguous. This keeps
implementation simple
(array storage can be used), but the first element need not
reflect
"chapter 1" in a user sense.

Alternatively you can just keep all chapters stored in
both/all files
after split, pointing to timestamp ranges outside of the
current file.
I have no idea if this is valid or not... 

Rich

_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, Nov 20, 2006 at 12:37:22PM -0500, Rich Felker wrote:
> On Mon, Nov 20, 2006 at 03:29:29PM +0100, Michael
Niedermayer wrote:
> > well, if you want to be able to start playing in
the middle (your comment
> > strongly suggests that) you either need to repeat
headers often or transmit
> > them out of band, or allow clients to seeks to
past data somehow ...
> 
> His idea is that the server would semi-dynamically
generate a stream
> by first writing the headers when a client connects,
then copying
> everything else from disk. This is what I had in mind
too.

Not necessarily from disk. All it has to buffer is the main
headers once, 
then it just has to wait for it's lead muxer to give a
syncpoint, and 
then give everything to the client like any other client...

> > that too, iam not sure if theres anything in the
spec which forbids it or
> > if theres anything which would break, iam just
mentioning it ...
> 
> Starting at arbitrary timestamp is totally legal and it
was always my
> intent for it to be legal. Obviously streaming is not
possible either
> if you don't allow for this..

I've never actually tested it, but AFAIK libnut is
completely safe and 
non-breaking on this issue.

> > you want to omit them in your streams IIRC because
of the space saving so
> > why shouldnt others too?
> 
> Well in Oded's case it happens that they're utterly
useless, but as
> long as we follow what you and I said above I don't
think there's a
> problem.

I assume you mean what you said regarding "complete
file" semantics and 
index. In which case, yes, I agree too.

> > > I have no idea for a solution. Your proposed
solution doesn't work as 
> > > there is no way to find the last info packet.
even with your stuffing 
> > > info packet in headers idea: 1. how would you
find the headers? 2. who 
> > > said the headers are necessarily after the
last info packet in the file? 
> > > Our main assumption is that in realtime
stream dumping, they can be 
> > > cut/truncated at absoloutely any arbitrary
point in the file.
> > 
> > 1. every midstream info packet (and only them)
MUST have a pointer to the
> >    previous non redundant/repeated midstream info
packet
> > 2. the distance between midstream info packets
MUST be <= C*max_distance
> >    unless that is impossible (too large stream
header / frame) in which
> >    case the distance MUST be as small as possible
for the specific case
> > 3. a info packet MUST either be part of the normal
info packets or it
> >    MUST be repeated like described in 2. within
the area to which it applies
> >    and it MUST be in that area at least once
> 
> Seems like the sanest proposal for this so far, but....
introduces
> huge ugly complexity to implementation.

It's actually not that bad in implementation - just keep
last_info and 
last_info_redundant (to know if to place another one now), a
few more if's 
together with syncpoint writing, and you're done.

Or do you mean complexity in demuxer? In which case, yes, it
is somewhat 
ugly... But I don't really agree in demuxer searching for
info packets 
anywhere after main headers anyway.

I have two questions though:
How do you know if to search for these redundant packets or 
not? I assume you do NOT write them at all if there are no
mid-stream info 
packets at all, in which case, when looking for them, how do
you know if 
you simply haven't scanned enough of C*max_distance from
EOF, or if there 
aren't mid-stream info packets at all?
Also, for the begginning of file, do you write these
info_redundant until 
you have one info packet or not?
Sounds like this would need another flag in the main header.

> > leaving them does no harm droping them changes the
frozen spec again ...
> > it also makes adding mid stream info packets at a
later time much
> > harder if the rules are droped now
> 
> Yes, let's not harm syntactic extensibility. I doubt
the sanity of
> this mid-stream info at all, but...

It does feel like a bikeshed issue, but it is very important
to some 
people (not me BTW).

[...]
> I'd like to request a minor
> revision: instead of requiring positive chater id's to
start from 1,
> just require them to be contiguous. This keeps
implementation simple
> (array storage can be used), but the first element need
not reflect
> "chapter 1" in a user sense.

I preffer this.

- ods15
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

Hi

On Mon, Nov 20, 2006 at 07:41:48PM +0200, Oded Shimon wrote:
> On Mon, Nov 20, 2006 at 12:37:22PM -0500, Rich Felker
wrote:
[...]
> > > that too, iam not sure if theres anything in
the spec which forbids it or
> > > if theres anything which would break, iam
just mentioning it ...
> > 
> > Starting at arbitrary timestamp is totally legal
and it was always my
> > intent for it to be legal. Obviously streaming is
not possible either
> > if you don't allow for this..

IIRC someone originally wanted timestamps to start from 0,
we then realized
thats not possible and then we agreed that it should be
close to 0 or
something (and no iam not at all trying to say that i am
against arbitrary
start timestamps ...)


> 
> I've never actually tested it, but AFAIK libnut is
completely safe and 
> non-breaking on this issue.

theres at least one issue with random start timestamps
try 1e9999 as start timestamp and tell me if that worked

while the fileformat of course has no problem with arbitrary
integers,
implementations will ...
making it clear that 0 should be used as start where
possible reduces
the issue but doesnt solve it

something like:
timestamps SHOULD start at or close to 0, and MUST fit into
63bit, note
this limit may be raised in the (distant) future
should solve the issue ...

this should be enough for a few hundread years at nanosecond
resolution


[...]
> > > > I have no idea for a solution. Your
proposed solution doesn't work as 
> > > > there is no way to find the last info
packet. even with your stuffing 
> > > > info packet in headers idea: 1. how
would you find the headers? 2. who 
> > > > said the headers are necessarily after
the last info packet in the file? 
> > > > Our main assumption is that in realtime
stream dumping, they can be 
> > > > cut/truncated at absoloutely any
arbitrary point in the file.
> > > 
> > > 1. every midstream info packet (and only
them) MUST have a pointer to the
> > >    previous non redundant/repeated midstream
info packet
> > > 2. the distance between midstream info
packets MUST be <= C*max_distance
> > >    unless that is impossible (too large
stream header / frame) in which
> > >    case the distance MUST be as small as
possible for the specific case
> > > 3. a info packet MUST either be part of the
normal info packets or it
> > >    MUST be repeated like described in 2.
within the area to which it applies
> > >    and it MUST be in that area at least once
> > 
> > Seems like the sanest proposal for this so far,
but.... introduces
> > huge ugly complexity to implementation.
> 
> It's actually not that bad in implementation - just
keep last_info and 
> last_info_redundant (to know if to place another one
now), a few more if's 
> together with syncpoint writing, and you're done.
> 
> Or do you mean complexity in demuxer? In which case,
yes, it is somewhat 
> ugly... But I don't really agree in demuxer searching
for info packets 
> anywhere after main headers anyway.

thats perfectly fine, i never proposed that demuxers should
search for them
its just that if a demuxer wants to or needs to then it
should be technically
possible to find them
also remuxing such streamdumps into midstream info less nut
files benefits
from the ability to qickly enumerate all midstream info
packets ...


> 
> I have two questions though:
> How do you know if to search for these redundant
packets or 
> not? I assume you do NOT write them at all if there are
no mid-stream info 
> packets at all, in which case, when looking for them,
how do you know if 
> you simply haven't scanned enough of C*max_distance
from EOF, or if there 
> aren't mid-stream info packets at all?
> Also, for the begginning of file, do you write these
info_redundant until 
> you have one info packet or not?

dunno, probably yes


> Sounds like this would need another flag in the main
header.

yes


> 
[...]
> [...]
> > I'd like to request a minor
> > revision: instead of requiring positive chater
id's to start from 1,
> > just require them to be contiguous. This keeps
implementation simple
> > (array storage can be used), but the first element
need not reflect
> > "chapter 1" in a user sense.
> 
> I preffer this.

same issue as with timestamps, may i limit that to 31 bit or
more
precissely int32_t?

and is it really that simple? i mean with the start from 0
its a
count=MAX(count, id); realloc(count); store
with the contiguous ones lets say you receive
100,101,102,103,104,105,99,100,101,...

here the first info packet with 99 was damaged ...

then what about mixing that with normal info packets? i mean
info packets
with chapter id 1,2,3
and then midstream info packets 4,5,6,7, ...
and if you now cut the file, you loose contiguousity ...

i think these things need to be resolved before we can think
about allowing
chapter ids to start at >0

[...]

-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or
copy any book
Today you'd get arrested for mere telling someone where the
library is
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

On Mon, Nov 20, 2006 at 11:32:37PM +0100, Michael
Niedermayer wrote:
> Hi
> 
> On Mon, Nov 20, 2006 at 07:41:48PM +0200, Oded Shimon
wrote:
> > On Mon, Nov 20, 2006 at 12:37:22PM -0500, Rich
Felker wrote:
> [...]
> > > > that too, iam not sure if theres
anything in the spec which forbids it or
> > > > if theres anything which would break,
iam just mentioning it ...
> > > 
> > > Starting at arbitrary timestamp is totally
legal and it was always my
> > > intent for it to be legal. Obviously
streaming is not possible either
> > > if you don't allow for this..
> 
> IIRC someone originally wanted timestamps to start from
0, we then realized
> thats not possible and then we agreed that it should be
close to 0 or
> something (and no iam not at all trying to say that i
am against arbitrary
> start timestamps ...)

I never agreed to such a thing because it was always my
intent that
you could split and recombine files across media without
loss of
information. Also there's no reasonable way to specify
"close to 0".

> > I've never actually tested it, but AFAIK libnut is
completely safe and 
> > non-breaking on this issue.
> 
> theres at least one issue with random start timestamps
> try 1e9999 as start timestamp and tell me if that
worked 
> while the fileformat of course has no problem with
arbitrary integers,
> implementations will ...
> making it clear that 0 should be used as start where
possible reduces
> the issue but doesnt solve it

I think it's clear that if you use idiotic time values
you'll have
problems with implementation support. IMO it's fine to say
just that
implementations SHOULD NOT go out of their way to support
excessively
large values for any field in a NUT file.

> > It's actually not that bad in implementation -
just keep last_info and 
> > last_info_redundant (to know if to place another
one now), a few more if's 
> > together with syncpoint writing, and you're done.
> > 
> > Or do you mean complexity in demuxer? In which
case, yes, it is somewhat 
> > ugly... But I don't really agree in demuxer
searching for info packets 
> > anywhere after main headers anyway.

Demuxer of course..

> thats perfectly fine, i never proposed that demuxers
should search for them
> its just that if a demuxer wants to or needs to then it
should be technically
> possible to find them

It is always possible via linear search. If the demuxer
SHOULD NOT
search for them then we should not go out of our way to make
it easy
to search... Just my 2¢...

> > > I'd like to request a minor
> > > revision: instead of requiring positive
chater id's to start from 1,
> > > just require them to be contiguous. This
keeps implementation simple
> > > (array storage can be used), but the first
element need not reflect
> > > "chapter 1" in a user sense.
> > 
> > I preffer this.
> 
> same issue as with timestamps, may i limit that to 31
bit or more
> precissely int32_t?

I don't even care if you limit it to int16_t...

> and is it really that simple? i mean with the start
from 0 its a
> count=MAX(count, id); realloc(count); store
> with the contiguous ones lets say you receive
> 100,101,102,103,104,105,99,100,101,...
> 
> here the first info packet with 99 was damaged ...

Yes, I see. You can remap them, I suppose it's a little
painful tho..

> then what about mixing that with normal info packets? i
mean info packets
> with chapter id 1,2,3
> and then midstream info packets 4,5,6,7, ...
> and if you now cut the file, you loose contiguousity
...
> 
> i think these things need to be resolved before we can
think about allowing
> chapter ids to start at >0

Bleh..

Rich


_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel

Hi

On Mon, Nov 20, 2006 at 08:57:36PM -0500, Rich Felker wrote:
[...]
> > > I've never actually tested it, but AFAIK
libnut is completely safe and 
> > > non-breaking on this issue.
> > 
> > theres at least one issue with random start
timestamps
> > try 1e9999 as start timestamp and tell me if that
worked 
> > while the fileformat of course has no problem with
arbitrary integers,
> > implementations will ...
> > making it clear that 0 should be used as start
where possible reduces
> > the issue but doesnt solve it
> 
> I think it's clear that if you use idiotic time values
you'll have
> problems with implementation support. IMO it's fine to
say just that
> implementations SHOULD NOT go out of their way to
support excessively
> large values for any field in a NUT file.

what is excessively large? whats idiotic? thats not a good
way to specify
the valid range of a value
>32bit is idiotic for many people iam pretty sure, still
its not enough
if your input data is in nanosecond precission ...

and its neither reasonable to assume that everyone has to
spend an hour
per field to guess what range of values would have to be
supported to handle
all non idiotic cases


> 
> > > It's actually not that bad in implementation
- just keep last_info and 
> > > last_info_redundant (to know if to place
another one now), a few more if's 
> > > together with syncpoint writing, and you're
done.
> > > 
> > > Or do you mean complexity in demuxer? In
which case, yes, it is somewhat 
> > > ugly... But I don't really agree in demuxer
searching for info packets 
> > > anywhere after main headers anyway.
> 
> Demuxer of course..
> 
> > thats perfectly fine, i never proposed that
demuxers should search for them
> > its just that if a demuxer wants to or needs to
then it should be technically
> > possible to find them
> 
> It is always possible via linear search. If the demuxer
SHOULD NOT
> search for them then we should not go out of our way to
make it easy
> to search... Just my 2¢...

well there really are 2 cases IMHO
A. midstream info packets are not allowed in normal nut
files
B. midstream info packets are allowed in normal nut files

for A i agree that the pointers and repeating shouldnt be
required, there may
be other reasons though why repeating the info makes sense
...

for B i dont agree, simply because if info is there, then
there are cases
where the user will want to have that info, think of some
capture of odeds
radio stream, its not unlikely to think that the user would
want to seek to
a specific song (she knows the song title but not the time
to seek to)


[...]
-- 
Michael     GnuPG fingerprint:
9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or
copy any book
Today you'd get arrested for mere telling someone where the
library is
_______________________________________________
NUT-devel mailing list
NUT-develmplayerhq.hu

http://lists.mplayerhq.hu/mailman/listinfo/nut-devel