Chapter 10. Synchronization

This chapter describes DM support for synchronizing digital media streams. The described techniques are designed to enable accurate synchronization even when there are large (and possibly unpredictable) processing delays.

UST

To timestamp each media stream, some convenient representation for time is needed. In DM, time is represented by the value of the Unadjusted System Time (UST) counter. That counter starts 0 when the system is reset, and increases continuously (without any adjustment) while the system is running.

Each process and/or piece of hardware may have its own view of the system UST counter. That view is an approximation to the real system UST counter. The difference between any two views is bounded for any implementation.

Each UST timestamp is a signed 64-bit integer value with units of nanoseconds representing a recent view of the system UST counter. A current view of the system UST is obtained by using the dmGetSystemUST function call.

DMstatus dmGetSystemUST(DMint64 systemId, DMint64* ust);

Currently systemId must be DM_SYSTEM_LOCALHOST, otherwise the status DM_STATUS_INVALID_ID is returned. The resulting UST value is placed at the address UST. The status DM_STATUS_INVALID_ARGUMENT is returned if UST is invalid. The status DM_STATUS_NO_ERROR is returned on a successful execution.

Get System UST

DMstatus dmGetSystemUST( systemId,  );

Use to obtain the current UST (Universal System Time) on a particular system. At this time, the only legal system id is DM_SYSTEM_LOCALHOST.

STATUS RETURN

This function returns one of the following:

DM_STATUS_NO_ERROR

The system UST was obtained successfully.

DM_STATUS_INVALID_ID

The specified systemid is invalid.

DM_STATUS_INVALID_ARGUMENT

The UST was not returned successfully (perhaps an invalid pointer?).

UST/MSC/ASC Parameters

Basic support for synchronization requires that the application know exactly when video or audio buffers passed through a jack. In DM this is achieved with the UST/MSC buffer parameters:

DM_AUDIO_UST_INT64, DM_VIDEO_UST_INT64

The unadjusted system time (UST) is the timestamp for the most recently processed slot in the audio/video stream. For video devices, the UST time corresponds to the time at which the field/frame starts to pass through the jack. For audio devices, the UST time corresponds to the time at which the first sample in the buffer passed through the jack.

Typically, an application will pass dmSendBuffers a video message containing a DM_IMAGE_BUFFER, a DM_VIDEO_MSC and a DM_VIDEO_UST (and possibly an DM_VIDEO_ASC - see below), or an audio message containing a DM_AUDIO_IMAGE_POINTER, a DM_AUDIO_UST, and a DM_AUDIO_MSC. In some cases, a message can contain both audio and video parameters.

Each message is processed as a single unit, and a reply is returned to the application via dmReceiveMessage. That reply will contain the completed buffer and the UST/MSC(/ASC) corresponding to the time at which the data in the buffers passed in or out of the jack. Note that, due to hardware buffering on some cards, it is possible to receive a reply message before the data has finished flowing through an output jack.

DM_AUDIO_MSC_INT64, DM_VIDEO_MSC_INT64

The media stream count (MSC) is the most recently processed slot in the audio/video stream. This is snapped at the same instant as the UST time described above. Note that MSC increases by one for each potential slot in the media stream through the jack. For interlaced video timings, each slot contains one video field, for progressive timings, each slot contains one video frame. This means that when 2 fields are interlaced into one frame and sent as one buffer, then the MSC will increment by 2 (one for each field). Furthermore, the system guarantees that the least significant bit of the MSC will reflect the state of the Field Bit, being 0 for Field 1 and 1 for Field 2. For audio, each slot contains one audio frame.

DM_AUDIO_ASC_INT64, DM_VIDEO_ASC_INT64

The application stream count (ASC) is provided to aid the developer in predicting when the audio or video data will pass through an output jack. See the “UST/MSC for Output” section below for further information on the use of the ASC parameter.

UST/MSC Example

For example, here we send an audio buffer and video buffer to an I/O path and request both UST and MSC stamps:

DMpv message[7]; 
message[0].param = DM_IMAGE_BUFFER_POINTER; 
message[0].value.pByte = someImageBuffer; 
message[0].length = sizeof(someImageBuffer); 
message[0].maxLength = sizeof(someImageBuffer); 
message[1].param = DM_VIDEO_UST_INT64; 
message[2].param = DM_VIDEO_MSC_INT64; 
message[3].param = DM_AUDIO_BUFFER_POINTER; 
message[3].value.pByte = someAudioBuffer; 
message[3].length = sizeof(someAudioBuffer); 
message[3].maxLength = sizeof(someAudioBuffer); 
message[4].param = DM_AUDIO_UST_INT64; 
message[5].param = DM_AUDIO_MSC_INT64; 
message[6].param = DM_END; 
dmSendBuffers( device, message); 

After the device has processed the buffers, it will enqueue a reply message back to the application. That reply will be an exact copy of the message sent in, with the exception that the MSC and UST values will be filled in. (For input, the buffer parameter length will also be set to the number of bytes written into it). Note that a dmSendBuffers call can only have one DM_IMAGE_BUFFER_POINTER.

UST/MSC For Input

On input the application can detect if any data is missing by looking for breaks in the MSC sequence. This could happen if an application did not provide buffers fast enough to capture all of the signal which arrived at the jack. (An alternative to looking at the MSC numbers, is to turn on the events DM_AUDIO_SEQUENCE_LOST or DM_VIDEO_SEQUENCE_LOST. Those will fire whenever the queue from application to device overflows.)

Given the UST/MSC stamps for two different buffers (UST1,MSC1) and (UST2,MSC2), the input sample rate in samples per nanosecond can be computed as:

Equation 10-1.

One common technique for synchronizing different input streams is to start recording early, stop recording late, and then use the UST/MSC stamps in the recorded data to find exact points for trimming the input data.

An alternative way to start recording several streams simultaneously is to use predicate controls (see later).

UST/MSC For Output

On output, the actual output sample rate can be computed in exactly the same way as the input sample rate:

Equation 10-2.

Some applications must determine exactly when the next buffer sent to the device will actually go out the jack. Doing this requires two steps. First, the application must maintain its own field/frame count. This parameter is called the ASC. The ASC may start at any desired value and should increase by one for every audio frame or video field enqueued. (For convenience, the application may wish to associate the ASC with the buffer by embedding it in the same message. The parameters DM_AUDIO_ASC_INT32 and DM_VIDEO_ASC_INT32 are provided for this use.)

Now, assume the application knows the (UST,MSC,ASC) for two previously-output buffers, then the application can detect if there was any underflow by comparing the number of slots the application thought it had output, with the number of slots which the system actually output.

	if (ASC2 - ASC1) == (MSC2 - MSC1) then all is well.

Assuming all is well, and that the application knows the current ASC, then the next data the application enqueues may be predicted to have a system sequence count of:

Equation 10-3.

and may be predicted to hit the output jack at time:

Equation 10-4.

Note that the application should periodically recompute the actual sample rate based on measured MSC/UST values. It is not sufficient to rely on a nominal sample rate since the actual rate may drift over time.

So, in summary: given the above mechanism, the application knows the UST/MSC pair for every processed buffer. Using the UST/MSC's for several processed buffers we can compute the frame rate. Given a UST/MSC pair in the past, a prediction of the current MSC, and the frame rate, the application can predict the UST at which the next buffer to be enqueued will hit the jack.

Predicate Controls

Predicate controls allow an application to insert conditional commands into the queue to the device. Using these we can pre-program actions, allowing the device to respond immediately, without needing to wait for a round-trip through the application.

Unlike the UST/MSC timestamps, predicate controls are not required to be supported on all audio/video devices. To see if they are supported on any particular device, look for the desired parameter in the list of supported parameters on each path (see dmGetCapabilities). The simplest predicate controls are:

DM_WAIT_FOR_AUDIO_MSC_INT64and

DM_WAIT_FOR_VIDEO_MSC_INT64

When the message containing this control reaches the head of the queue it causes the queue to stall until the specified MSC value has passed. Then that message, and subsequent messages, are processed as normal.

For example, here is code that uses WAIT_FOR_AUDIO_MSC to send a particular buffer out after a specified stream count:

DMpv message[3]; 
message[0].param = DM_WAIT_FOR_AUDIO_MSC_INT64; 
message[0].value.int64 = someMSCInTheFuture; 
message[1].param = DM_AUDIO_BUFFER_POINTER; 
message[1].value.pByte = someBuffer; 
message[1].value.length = sizeof(someBuffer); 
message[2].param = DM_END; 
dmSendBuffers( someOpenPath, message); 

This places a message on the queue to the path and then immediately returns control to the application. As the device processes that message, it will pause until the specified media MSC value has passed before allowing the buffer to flow through the jack.

Using this technique an application can program several media streams to start in-sync by simply choosing some MSC count to start in the future.


Note: If both DM_IMAGE_DOMINANCE and DM_WAIT_FOR_VIDEO_MSC controls are set and do not correspond to the same starting output field order, the DM_WAIT_FOR_VIDEO_MSC_INT64 control will override DM_IMAGE_DOMINANCE_INT32 control settings.


Another set of synchronization predicate controls are:

DM_WAIT_FOR_AUDIO_UST_INT64 and DM_WAIT_FOR_VIDEO_UST_INT64

When the message containing this control reaches the head of the queue it causes the queue to stall until the specified UST value has passed. Then that message, and subsequent messages, are processed as normal. Note that the accuracy with which the system is able to implement the WAIT_FOR_UST command is device-dependent - see device-specific documentation for limitations. For example, here is code that uses WAIT_FOR_AUDIO_UST to send a particular buffer out after a specified time:

MLpv message[3]; 
message[0].param = ML_WAIT_FOR_AUDIO_UST_INT64; 
message[0].value.int64 = someUSTtimeInTheFuture; 
message[1].param = ML_AUDIO_BUFFER_POINTER; 
message[1].value.pByte = someBuffer; 
message[1].value.length = sizeof(someBuffer); 
message[2].param = ML_END; 
mlSendBuffers( someOpenPath, message); 

This places a message on the queue to the path and then immediately returns control to the application. As the device processes that message, it will pause until the specified video UST time has passed before allowing the buffer to flow through the jack.

Using this technique an application can program several media streams to start in-sync by simply choosing some UST time in the future and program each to start at that time.

DM_IF_VIDEO_UST_LT or

DM_IF_AUDIO_UST_LT

When included in a message, this control will cause the following logical test: if the UST is less than the specified time, then the entire message is processed as normal. Otherwise, the entire message is simply skipped.

Regardless of the outcome, any following messages are processed as normal. Skipping over a message takes time, so there is a limit to how many messages a device can skip before the delay starts to become noticeable. All media devices will support skipping at least one message without noticeable delay.