CR: Rebuffering of Video only presenation when video stream is running behind

Rajesh,

The base video renderer implements IHXPlaybackVelocity,
and IHXPlaybackVelocity has a method call SetKeyFrameMode().

I don't know whether mdfvideorenderer inherits from the
base video renderer or not. If it does, then the member
variable m_bKeyFrameMode will tell you if you are 
in keyframe mode or not. If medvideorender does NOT inherit
from the base video renderer, then it will need to implement
IHXPlaybackVelocity in order to be notified of
keyframe mode and/or velocity changes.

IHXPlaybackVelocity is for "trick-play" playback,
such
as 2x/4x/10x playback or reverse playback. It is not
mandatory for renderers. If the HXPlayer gets an instruction
to go into a TRickPlay mode (perhaps 2x forward playback),
then
it will query all the renderers to see if they support
IHXPlaybackVelocity. If one of the renderers does not
support
IHXPlaybackVelocity, then the call to set the playback
velocity to 2x will fail.

Eric

=============================================
Eric Hyche (ehychereal.com)
Technical Lead
RealNetworks, Inc.  

> -----Original Message-----
> From: datatype-dev-bounceshelixcommunity.org 
> [mailto:datatype-dev-bounceshelixcommunity.org] On
Behalf Of 
> rajesh.rathinasamynokia.com
> Sent: Friday, December 08, 2006 9:50 AM
> To: milkoreal.com; ldhawanreal.com
> Cc: datatype-devhelixcommunity.org
> Subject: RE: [datatype-dev] CR: Rebuffering of Video
only 
> presenation whenvideo stream is running behind
> 
> Hello All,
>   I'm trying to incorporate this change to symbian 
> mdfvideorenderer. Can someone help me clarify some
questions ?
>  
> * One of Milko's mail said about the default tolerance
value 
> should be different for diff renderers. The reason I
got from 
> his mail is Key frame mode of RV playback. Provided we
need 
> to alter the tolerance value for RV playback, how can
the 
> renderer know that it is in Key Frame mode ?
>  
> * What is the purpose of the feature 
> HELIX_FEATURE_PLAYBACK_VELOCITY ? It seemed to have
some 
> interface for key frame mode operations. Is the impl of
this 
> interface mandatory for renderers ?
>  
> Thanks,
> Rajesh.
>  
> 
> 
> ________________________________
> 
> 	From: datatype-dev-bounceshelixcommunity.org 
> [mailto:datatype-dev-bounceshelixcommunity.org] On
Behalf Of 
> ext Milko Boic
> 	Sent: Wednesday, November 29, 2006 2:20 PM
> 	To: Lovish Dhawan
> 	Cc: datatype-devhelixcommunity.org
> 	Subject: Re: [datatype-dev] CR: Rebuffering of Video 
> only presenation when video stream is running behind
> 	
> 	
> 
> 	Lovish,
> 	
> 	This looks good, except that I had an error in one 
> patch I gave you for cayenne.
> 	The following patch should be changed.  The if logic 
> should be inverted as m_bFirstFrane stands for
"still 
> expecting first frame" and not "having seen
first frame" as I 
> assumed when submitting the patch.
> 	
> 	          // If first frame has not been displayed
yet, 
> set active video time to the first
> 	          // OnTimeSync supplied time that moves us 
> into the playing state.
> 	          // This will allow us to properly measure 
> amount of time passed without
> 	          // any frame blts from the start of
playback.
> 	-         if (!m_bFistFrame)
> 	+        if (m_bFistFrame)
> 	          {
> 	               m_ulActiveVideoTime = ulTime;
> 	          }
> 	
> 	After this change, please commit.  You need not wait 
> for further approvals.
> 	Please make sure your submission to HEAD does not 
> contain any cayenne specific patches.
> 	
> 	Thanks,
> 	Milko
> 	
> 	At 06:03 AM 11/29/2006, Lovish Dhawan wrote:
> 	
> 
> 
> 		Hi Milko,
> 		 
> 		Please find the updated diffs.
> 		 
> 		Thanks & Regards
> 		Lovish
> 		
> 
> 			----- Original Message ----- 
> 			
> 			From: Milko Boic <mailto:milkoreal.com>  
> 			
> 			To: Lovish Dhawan <mailto:ldhawanreal.com>  
> 			
> 			Cc: datatype-devhelixcommunity.org 
> 			
> 			Sent: Wednesday, November 29, 2006 4:16 AM
> 			
> 			Subject: Re: [datatype-dev] CR: 
> Rebuffering of Video only presenation when video stream
is 
> running behind
> 			
> 			
> 			
> 			Lovish,
> 			
> 			
> 			Few more things:
> 			
> 			
> 			1.)
> 			
> 			The following include mixes upper/lower 
> case which will break case sensitive platforms.
> 			
> 			Also, this include should be in .cpp 
> file header as it is not needed in .h:
> 			
> 			RCS file: 
> /cvsroot/datatype/common/vidrend/pub/vidrend.h,v
> 			
> 			retrieving revision 1.27.2.9
> 			
> 			diff -u -w -r1.27.2.9 vidrend.h
> 			
> 			--- vidrend.h   24 Oct 2006 05:30:33 
> -0000      1.27.2.9
> 			
> 			+++ vidrend.h   28 Nov 2006 09:47:09 -0000
> 			
> 			 -53,6 +53,7 
> 			
> 			 
>
/***********************************************************
**
> ***************
> 			
> 			  *  Includes
> 			
> 			  */
> 			
> 			+#include "Hxausvc.h"
> 			
> 			 #include "vidrendf.h"
> 			
> 			
> 			2.)
> 			
> 			In cayenne branch only, the code to 
> insure m_ulActiveVideoTime is properly initialized on
start 
> and after seek needs to be added.  HEAD has code that
handles 
> this already but it cannot be merged to cayenne as
client 
> core behavior needed for this code to function
correctly is 
> missing in cayenne client core.
> 			
> 			The following code should thus be 
> applied to cayenne family of branches to ensure 
> m_ulActiveVideoTIme is properly initialized on start
and after seek:
> 			
> 			STDMETHODIMP 
> CVideoRenderer::OnTimeSync(ULONG32 ulTime)
> 			
> 			{
> 			
> 			...
> 			
> 			        if (m_PlayState == PlayStarting)
> 			
> 			        {
> 			
> 			            m_PlayState = Playing;
> 			
> 			
> 			+          // If first frame has not 
> been displayed yet, set active video time to the first
> 			
> 			+          // OnTimeSync supplied time 
> that moves us into the playing state.
> 			
> 			+          // This will allow us to 
> properly measure amount of time passed without
> 			
> 			+          // any frame blts from the 
> start of playback.
> 			
> 			+          if (!m_bFistFrame)
> 			
> 			+          {
> 			
> 			+               m_ulActiveVideoTime = ulTime;
> 			
> 			+          }
> 			
> 			
> 			            BltIfNeeded();
> 			
> 			
> 			            StartSchedulers();
> 			
> 			
> 			            if (m_pBltrPump)
> 			
> 			            {
> 			
> 			                m_pBltrPump->Signal();
> 			
> 			            }
> 			
> 			        }
> 			
> 			
> 			
> 			void 
> CVideoRenderer::SchedulerCallback(BOOL bIsScheduled,
> 			
> 			                                       
> BOOL bResched,
> 			
> 			                                       
> BOOL bIsVS2Call,
> 			
> 			                                       
> BOOL bProcessUndisplayableFramesOnly)
> 			
> 			{
> 			
> 			...
> 			
> 			                if (pPkt->m_pData && 
> bDisplayFrame && m_bFirstFrame)
> 			
> 			                {
> 			
> 			                    HXLOGL2(HXLOG_VIDE, 
> "Displaying first frame");
> 			
> 			                    m_bFirstFrame = FALSE;
> 			
> 			 +                 m_ulActiveVideoTime 
> = pPkt->m_ulTime;
> 			
> 			                }
> 			
> 			...
> 			
> 			}
> 			
> 			
> 			
> 			void CVideoRenderer::CheckForBufferingStart()
> 			
> 			{
> 			
> 			    LONG32 lTimeDelta = 0;
> 			
> 			    if (m_ulVideoBufferingPreference == 0)
> 			
> 			    {
> 			
> 			        // Video Buffering disabled
> 			
> 			        return;
> 			
> 			    }
> 			
> 			    else
> 			
> 			    {
> 			
> 			-       if (m_pVideoFormat && 
> !m_pVideoFormat->AreRawPacketsDone())
> 			
> 			+      if ((m_PlayState == Playing) && 
> m_pVideoFormat &&
!m_pVideoFormat->AreRawPacketsDone())
> 			
> 			        {
> 			
> 			            lTimeDelta = 
> ComputeTimeAhead( m_ulActiveVideoTime, 0);
> 			
> 			
> 			            if 
> ((m_ulAbsPlaybackVelocity !=
HX_PLAYBACK_VELOCITY_NORMAL) && 
> 			
> 			                (m_ulAbsPlaybackVelocity != 0))
> 			
> 			            {
> 			
> 			                lTimeDelta = 
> ((lTimeDelta * HX_PLAYBACK_VELOCITY_NORMAL) / 
> 			
> 			                                
> ((INT32) m_ulAbsPlaybackVelocity));
> 			
> 			            }
> 			
> 			
> 			            if (((INT32) (lTimeDelta + 
> GetLateFrameRebufferingTolerance())) < 0)
> 			
> 			            {
> 			
> 			                if 
> ((m_ulVideoBufferingPreference == 2) &&
!IsVideoOnlyPlayback())
> 			
> 			                {
> 			
> 			                    // Video buffering 
> disabled for non-video only playback
> 			
> 			                    return;
> 			
> 			                }
> 			
> 			                RequestBuffering();
> 			
> 			            }
> 			
> 			        }
> 			
> 			    }
> 			
> 			}
> 			
> 			
> 			Thanks,
> 			
> 			Milko
> 			
> 			
> 			At 07:05 AM 11/28/2006, Lovish Dhawan wrote:
> 			
> 
> 				Hi Milko,
> 				
> 				
> 				  
> 				Please find the diffs 
> incorporating your suggestions.
> 				
> 				
> 				  
> 				Thanks & Regards,
> 				
> 				Lovish 
> 
> 					----- Original Message ----- 
> 					From: Milko Boic 
> <mailto:milkoreal.com>  
> 					To: Lovish Dhawan 
> <mailto:ldhawanreal.com>  
> 					Cc: 
> datatype-devhelixcommunity.org 
> 					Sent: Tuesday, November 
> 28, 2006 12:37 AM 
> 					Subject: Re: 
> [datatype-dev] CR: Rebuffering of Video only
presenation when 
> video stream is running behind
> 					
> 					At 06:32 AM 11/23/2006, 
> Lovish Dhawan wrote: 
> 
> 						Hi Milko, 
> 						
> 						One more 
> question about reuse of
m_pTimeSyncSmoother->GetBaseTime() 
> instead of adding another set of member variables to
track 
> last packet time. 
> 						
> 						Logic in 
> IsBufferingComplete() is to buffer atleast
"preroll" amount 
> of packets since the re-buffering start, to mark end of

> buffering. m_pTimeSyncSmoother->GetBaseTime()
returns the 
> packet time for the very first packet at the start of 
> playback since it does not get reset at the
re-buffering event. 
> 
> 					Base time in 
> TimeSyncSmoother gets reset with every on time sync
(unless 
> the OnTimeSync sample is discarded if determied to be 
> especially discontinouous).
> 					
> 					
> 
> 						So, we can 
> either "reset" the TimeSyncSmoother base time
using the 
> timestamp of the first packet received after start of 
> re-buffering event OR use a separate set of member
variables 
> to keep track of last packet time. 
> 
> 					No - simply use 
> ComputeTimeAhead(pPacket->GetTime(), 0) >
m_ulPreroll .
> 					
> 					
> 
> 						My 
> understanding is that TimeSyncSmoother uses base time
to 
> smooth the playback timeline and the base time is set
only at 
> the start of buffering (either at start or after
seeking), 
> 
> 					No - on every OnTimeSync.
> 					
> 					
> 
> 						so resetting 
> this value at rebuffering event is likely not the
correct 
> approach. I think we should use 
> m_ulPacketTimeAtStartOfBuffering instead. 
> 
> 					No.
> 					
> 					
> 
> 						Attached are 
> the new diffs incorporating your suggestions with use
of 
> m_ulPacketTimeAtStartOfBuffering instead of 
> m_pTimeSyncSmoother->GetBaseTime(). 
> 
> 					-> Assert is nice, but 
> check the pointer in the if too.  Render's can run in 
> environments other the client-core.  Do not make an 
> assumption that interfaces will be there. 
> 					+            
> HX_ASSERT(pPlayer); 
> 					+            if 
> (pPlayer->QueryInterface(IID_IHXAudioPlayer, 
> (void**)&pAudioPlayer) == HXR_OK) 
> 					+            {
> 					
> 					-> pPlayer is leaked 
> below.  Need to HX_RELEASE it. 
> 					+        
> if(pSource->GetPlayer(pPlayer) == HXR_OK) 
> 					+        { 
> 					+            
> HX_ASSERT(pPlayer); 
> 					+            if 
> (pPlayer->QueryInterface(IID_IHXAudioPlayer, 
> (void**)&pAudioPlayer) == HXR_OK) 
> 					+            { 
> 					+                
> if(pAudioPlayer->GetAudioStreamCount() == 0) 
> 					+                { 
> 					+                    
> bIsVideoOnlyPlayback = TRUE; 
> 					+                } 
> 					+                
> HX_RELEASE(pAudioPlayer); 
> 					+            } 
> 					+        }
> 					
> 					
> 					->  Below return code 
> is too harsh.  If we code it such that we consider it
OK to 
> be called when video buffering is called, we should not

> return a failure.
> 					
> 					
> 					+    if 
> (m_ulVideoBufferingPreference == 0) 
> 					+    { 
> 					+        // Video 
> Buffering disabled 
> 					+        return 
> HXR_INVALID_OPERATION;
> 					
> 					
> 					Return HXR_OK instead.
> 					
> 					-> Per coding spec, 
> always put space between if and conditional
expressions: 
> 					+        
> if(pSource->GetPlayer(pPlayer) == HXR_OK)
> 					
> 					
> 					
> 
> 						Please let me know. 
> 						
> 						Thanks & Regards 
> 						Lovish 
> 						  
> 
> 							----- 
> Original Message ----- 
> 							From: 
> Milko Boic <mailto:milkoreal.com>  
> 							To: 
> Lovish Dhawan <mailto:ldhawanreal.com>  ; 
> datatype-devhelixcommunity.org 
> 							Sent: 
> Saturday, November 18, 2006 10:55 AM 
> 							
> Subject: Re: [datatype-dev] CR: Rebuffering of Video
only 
> presenation when video stream is running behind
> 							
> 							Forgot 
> to scale the LateFrameRebufferingTolerance by playback 
> acceleration factors (trick-play). 
> 							Here is 
> corrected CheckForBufferingStart: 
> 							void 
> CVideoRenderer::CheckForBufferingStart() 
> 							{ 
> 							        
> if (m_ulVideoBufferingPreference == 0) 
> 							        { 
> 							        
>          // Video Buffering disabled 
> 							        
>          return; 
> 							        } 
> 							        else 
> 							        { 
> 							        
>          if (m_pVideoFormat && 
> 							        
>               !m_pVideoFormat->OnRawPacketsEnded()) 
> 							        
>          { 
> 							        
>                   lTimeDelta = ComputeTimeAhead( 
> m_ulActiveVideoTime, 0); 
> 							        
>                    if ((m_ulAbsPlaybackVelocity != 
> HX_PLAYBACK_VELOCITY_NORMAL) && 
> 							        
>                        (m_ulAbsPlaybackVelocity != 0)) 
> 							        
>                   { 
> 							        
>                            lTimeDelta = ((lTimeDelta * 
> HX_PLAYBACK_VELOCITY_NORMAL) / 
> 							        
>                                     ((INT32) 
> m_ulAbsPlaybackVelocity)); 
> 							        
>                   }
> 							
> 							
> 							        
>                   if (((INT32) (lTimeDelta + 
> GetLateFrameRebufferingTolerance())) < 0) 
> 							        
>                   { 
> 							        
>                            if
((m_ulVideoBufferingPreference 
> == 2) && !IsVideoOnlyPlayback()) 
> 							        
>                            { 
> 							        
>                                     // Video buffering 
> disabled for non-video only playback 
> 							        
>                                     return; 
> 							        
>                            } 
> 							        
>                             RequestBuffering(); 
> 							        
>                   } 
> 							        
>          } 
> 							        } 
> 							}
> 							
> 							
> --------------------- 
> 							The 
> implementation has a few issues: 
> 							1.) 
> 							
> REBUFFER_ON_VIDEO should be enabled and controlled by a

> preference: "RebufferOnVideo". 
> 							Default 
> should be 0 (do not rebuffer on video). 
> 							You can 
> retrieve the preference value at the bottom of
::OnHeader method. 
> 							Values: 
> 							0 = do 
> not rebuffer on video 
> 							1 = 
> rebuffer on video 
> 							2 = 
> rebuffer if video only playback 
> 							2.) 
> 							+UINT16 
> 							
> +CVideoRenderer::CheckForNumStreams() 
> 							+{ 
> 							The 
> above should be changed to
CVideoRenderer::IsVideoOnlyPlayback() . 
> 							This 
> routine should obtain IHXAudioPlayer from the source
and 
> check for pAudioPlayer->GetAudioStreamCount() != 0 .

> 							The 
> NULL pointers must be checked.  We should not assert.  
> Renderer component may be run in environments where
audio 
> player is not available.  We need to handle such case
for 
> defaulting to default case of not being a video only
playback. 
> 							That 
> is, we should assume it is not video only playback
unless 
> confirmed otherwise.
> 							
> 							
> 							3.) 
> 							Remove 
> all added member vars - they are not needed: 
> 							
> ulBaseTime - available from 
> m_pTimeSyncSmoother->GetBaseTime() and not needed 
> 							
> m_bIsBaseTimeSet - not needed 
> 							
> m_ulLastFrameTime - not needed, m_ulActiveVideoTime is 
> available (merge from HEAD if needed)
> 							
> 							
> 							4.) 
> 							+    
> if(pPacket && !(pPacket->IsLost())) 
> 							+    { 
> 							+       
>  m_ulLastFrameTime = pPacket->GetTime(); 
> 							+       
>  m_lTimeOffset = lTimeOffset; 
> 							+    } 
> 							Setting 
> the frame time in OnPacket call is conceptually wrong
and 
> inaccurate. Packet is delivered well ahead of playback
time 
> and such check does not capture freezing of video due
to 
> either packets consistently arriving late or due to CPU

> constraints making the decoded frames late.  What needs
to be 
> compared is last renderer frame displayed and current 
> playback time.  More precisely, we need to see how far
behind 
> the playtime currently displayed frame is.  In general,
it is 
> difficult to use a constant to check against since
video 
> frame rate can vary and it is also possible for a video

> stream to enter a key-frame only mode (RealVideo)
during low 
> bandwidth conditions.  In such mode it is not unusual
for 
> key-frames to be 10 seconds or more apart.  It is
precisely 
> for this reason that video is really a
pseudo-contiguous 
> media and could not be reliably used for re-buffering. 
To 
> mitigate this situation, a new virtual method 
> GetLateFrameRebufferingTolerance() be called from the
base 
> video renderer that can be overriden in the derived
renderers 
> (e.g. RealVideo or Mpeg4Video) to provide proper time 
> interval threshold of un-displayed frames to cause 
> re-buffering.  This way the value can be set
appropriately 
> for different data-types. 
> 							This 
> check should be placed in ::SchedulerCallback() method
in 
> couple of places where we : 
> 							a.) 
> 							        
>     else 
> 							            { 
> 							+       
>         CheckForBufferingStart(); 
> 							        
>         if (bResched) 
> 							        
>         { 
> 							        
>            
ScheduleCallback(m_ulNoFramesPollingInterval); 
> 							        
>         } 
> 							        
>         DisplayMutex_Unlock(); 
> 							        
>           if (m_pVideoFormat && 
> m_pVideoFormat->IsFillbackDecodeNeeded()) 
> 							        
>          { 
> 							        
>               m_pVideoFormat->DecodeFrame(); 
> 							        
>          } 
> 							        
>         return; 
> 							            } 
> 							b.) 
> 							    if 
> ((m_PlayState != Stopped) && 
> 							        
> bResched) 
> 							    { 
> 							+      
> CheckForBufferingStart(); 
> 							        
> ScheduleCallback(NO_FRAMES_POLLING_INTERVAL); 
> 							        
> DisplayMutex_Unlock(); 
> 							        
>  if (m_pVideoFormat && 
> 							        
>      m_pVideoFormat->IsFillbackDecodeNeeded()
&& 
> 							        
>      (!m_bServicingFillbackDecode)) 
> 							        { 
> 							        
>      m_bServicingFillbackDecode = TRUE; 
> 							        
>      m_pVideoFormat->DecodeFrame(); 
> 							        
>      m_bServicingFillbackDecode = FALSE; 
> 							        } 
> 							        return; 
> 							    }
> 							
> 							void 
> CVideoRenderer::CheckForBufferingStart() 
> 							{ 
> 							        
> if (m_ulVideoBufferingPreference == 0) 
> 							        { 
> 							        
>          // Video Buffering disabled 
> 							        
>          return; 
> 							        } 
> 							        else 
> 							        { 
> 							        
>          if (m_pVideoFormat && 
> 							        
>               !m_pVideoFormat->OnRawPacketsEnded()
&& 
> 							        
>               ComputeTimeAhead(m_ulActiveVideoTime, 
> GetLateFrameRebufferingTolerance()) < 0) 
> 							        
>          { 
> 							        
>                   if ((m_ulVideoBufferingPreference ==
2) && 
> !IsVideoOnlyPlayback()) 
> 							        
>                   { 
> 							        
>                            // Vide buffering disabled
for 
> non-video only playback 
> 							        
>                            return; 
> 							        
>                   } 
> 							        
>                    RequestBuffering(); 
> 							        
>          } 
> 							        } 
> 							} 
> 							5.) 
> 			

CR: Rebuffering of Video only presenation when video stream is running behind

Thread: CR: Rebuffering of Video only presenation when video stream is running behind