Chapter 7. Image Parameters

This chapter describes in detail the dmSDK image parameters and gives examples of the resulting in-memory pixel formats.

Introduction

An image buffer is memory allocated for a frame or field of pixels. Since the dmSDK itself does not allocate memory for buffers, the application must do the allocation. This means that each buffer requires a dedicated memory allocation call (malloc, for example.)

Buffers must be in contiguous virtual memory and should be pinned in memory for optimum performance. Once a buffer has been created, the pointer to the buffer is passed to the dmSDK with the parameter DM_IMAGE_BUFFER_POINTER. Pointer to the first byte of an image buffer in memory. The buffer address must comply with the alignment constraints for buffers on the particular path or transcoder to which it is being sent. See dmGetCapabilities for details on determining alignment requirements with DM_PATH_BUFFER_ALIGNMENT_INT32. For example, if DM_PATH_BUFFER_ALIGNMENT_INT32 is 8, this means that the value of the buffer pointer must be a multiple of 8 bytes. The same applies to DM_PATH_COMPONENT_ALIGNMENT_INT32 where the beginning of each line (the first pixel of each line) must be a multiple of the value of the DM_PATH_COMPONENTALIGNMENT_INT32 parameter.

Figure 7-1. General Image Buffer Layout

General Image Buffer Layout

In Figure 7-1 an image is mapped into a image buffer in a very general form.

Figure 7-2. Simple Image Buffer Layout

Simple Image Buffer Layout

Figure 7-2 shows the more common simple image buffer layout.

Image Buffer Parameters

The following subsections list and describe all image parameters.

DM_IMAGE_BUFFER_POINTER

Pointer to the first byte of an image buffer in memory. The buffer address must comply with the alignment constraints for buffers on the particular path or transcoder to which it is being sent. See dmGetCapabilities for details on determining alignment requirements with DM_PATH_BUFFER_ALIGNMENT_INT32. For example, if DM_PATH_BUFFER_ALIGNMENT_INT32 is 8, this means that the value of the buffer pointer must be a multiple of 8 bytes. The same applies to DM_PATH_COMPONENT_ALIGNMENT_INT32 where the beginning of each line (the first pixel of each line) must be a multiple of the value of the DM_PATH_COMPONENT_ALIGNMENT_INT32 parameter.

DM_IMAGE_WIDTH_INT32

The width of the image in pixels.

DM_IMAGE_HEIGHT_1_INT32

For progressive or interleaved buffers (depending on parameter DM_IMAGE_INTERLEAVE_MODE_INT32), this represents the height of each frame. For interlaced and non-interleaved signals, this represents the height of each F1 field. Measured in pixels.

DM_IMAGE_HEIGHT_2_INT32

The height of each F2 field in an interlaced non-interleaved signal. Otherwise it has value 0.

DM_IMAGE_ROW_BYTES_INT32

The number of bytes along one row of the image buffer. If this value is 0, each row is exactly DM_IMAGE_WIDTH_INT32 pixels wide. Default is 0.


Note: In physical memory there is no notion of two dimensions, the end of the first row continues directly at the start of the second row. An image buffer contains either one frame or one field. For interlaced image data the two fields can be stored in two separate image buffers or they can be stored in interleaved form in one image buffer.


DM_IMAGE_SKIP_PIXELS_INT32

The number of pixels to skip at the start of each line in the image buffer. Default is 0. Must be 0 if DM_IMAGE_ROW_BYTES_INT32 is 0. Default is 0.

DM_IMAGE_SKIP_ROWS_INT32

The number of rows to skip at the start of each image buffer. Default is 0.

DM_IMAGE_TEMPORAL_SAMPLING_INT32

Specifies whether the image source is progressive or interlaced. Set to DM_TEMPORAL_SAMPLING_FIELD_BASED or DM_TEMPORAL_SAMPLING_PROGRESSIVE. Default is device-dependent. If the image data is field based, the parameter DM_IMAGE_INTERLEAVE_MODE_INT32 defines how the two fields are stored in an image buffer.

DM_IMAGE_INTERLEAVE_MODE_INT32

Only used in interlaced images. This parameter specifies whether the two fields have been interleaved into a single image (and reside in a single buffer) or are stored in two separate fields (hence in two separate buffers). Set to DM_INTERLEAVE_MODE_INTERLEAVED or DM_INTERLEAVE_MODE_SINGLE_FIELD. This is ignored for signals with progressive timing. Default is interleaved.

In DM_INTERLEAVE_MODE_INTERLEAVED each pair of fields is interleaved into a single buffer. In this case the parameter DM_IMAGE_HEIGHT_2_INT32 is set to zero.

For DM_INTERLEAVE_MODE_SINGLE_FIELD the two fields are stored separately. This means that each field has its own image buffer, use DM_IMAGE_HEIGHT_1_INT32 for the F1 buffer and DM_IMAGE_HEIGHT_2_INT32 for the F2 buffer.

DM_IMAGE_DOMINANCE_INT32

Sets the dominance of the video signal. The allowable values are DM_DOMINANCE_F1 (default), and DM_DOMINANCE_F2. Ignored for progressive signals. Field dominance defines the order of fields in a frame and can be either F1-dominant or F2-dominant. F1-dominant specifies a frame as an F1 field followed by an F2 field. F2-dominant specifies a frame as an F2 field followed by an F1 field. Notice also that for the same sequent of fields there are two valid interpretations which of the two fields belong together. Changing the field dominance is most significant when external devices (for example, a tape deck) can only operate on frame boundaries.

Figure 7-3. Field Dominance

Field Dominance

DM_IMAGE_ORIENTATION_INT32

The orientation of the image.

DM_ORIENTATION_TOP_TO_BOTTOM “natural video order” pixel [0,0] is at the top left of the image.

DM_ORIENTATION_BOTTOM_TO_TOP ”natural graphics order” pixel [0,0] is at the bottom left of the image.

DM_IMAGE_COMPRESSION_INT32

An image buffer can also store a compressed image, for example this could be the output of a codec. If the image data is compressed, then one of the following values are used:

DM_COMPRESSION_UNCOMPRESSED
DM_COMPRESSION_BASELINE_JPEG
DM_COMPRESSION_DV_625
DM_COMPRESSION_DV_525
DM_COMPRESSION_MPEG2I
DM_COMPRESSION_DVCPRO_625
DM_COMPRESSION_DVCPRO_525
DM_COMPRESSION_DVCPRO50_625
DM_COMPRESSION_DVCPRO50_525
DM_COMPRESSION_MPEG2

If the image data is in uncompressed format the value of this parameter is DM_COMPRESSION_UNCOMPRESSED.


Note: In case of a compressed bit stream, all parameters that describe the image data (that is, height, width, color space, etc.) might not be known. The only parameters that might be known are the compression type DM_IMAGE_COMPRESSION_INT32 and the size of the bit stream DM_IMAGE_SIZE_INT32. The image buffer layout parameters (DM_IMAGE_SKIP_ROWS, DM_IMAGE_SKIP_PIXELS, and DM_IMAGE_ROW_BYTES) do not apply to compressed images.


For more infomation on JPEG, refer to W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image Data Compression Standard, New York, NY: Van Nostrand Reinhold, 1993.

For more information on DV compression, refer to Specification of Consumer-Use Digital VCRs using 6.3mm magnetic tape.

For more information on DVCPRO and DVCPRO50 compression, refer to SMPTE 314M Television - Data Structure for DV-Based Audio, Data and Compressed Video - 25 and 50 Mb/s.

For more information on MPEG2, refer to ISO/IEC 13818-2 GENERIC CODING OF MOVING PICTURES AND ASSOCIATED AUDIO: SYSTEMS.

DM_IMAGE_SIZE_INT32

Size of the image buffer in bytes. This is a read-only parameter and is computed in the device using the current path control settings. This value represents the worst-case buffer size.

DM_IMAGE_COMPRESSION_FACTOR_REAL32

For compressed images only, this parameter describes desired compression factor. A value of 1 indicates no compression, a value of x indicates that approximately x compressed buffers require the same space as 1 uncompressed buffer.


Note: The size of the uncompressed buffer depends on image width, height, packing and sampling. The default value is implementation-dependent, but should represent a reasonable trade-off between compression time, quality and bandwidth. x is a number larger than 1.


DM_IMAGE_PACKING_INT32

For recommendations on packing and component ordering, see Appendix A: "Pixels in Memory.”

The image packing parameter describes the pixel storage in detail as follows:

DM_PACKING_type_size_order

  • type is the base type of each component. Leave blank for an unsigned integer, use S for a signed integer. (In the future, the dmSDK may also support R for real numbers.)

  • size defines the number of bits per component. The size may refer to simple, padded or complex packings.

    For the simplest formats every component is the same size and there is no additional space between components. Here, a single numeric value specifies the number of bits per component. The first component consumes the first size bits, the next consumes the next size bits, and so on. Within each component, the most significant bits always precede the least-significant bits. For example, a size of 12 means that the first byte in memory has the most significant 8 bits of the first component, the second byte holds the remainder of the first component and the most significant 4 bits of the second component, and so on.

    Space is only allocated for components which are in use (that depends on the sampling mode, see later). For these formats the data must always be interpreted as a sequence of bytes. For example, DM_PACKING_8 describes a packing in which each component is an unsigned 8-bit quantity. DM_PACKING_S8 describes the same packing except that each component is a signed 8-bit quantity.

    For padded formats, each component is padded and may be treated as a short 2-byte integer. When this occurs, the size takes the form: {bits}in{size}{alignment} where:

    bits 

    is the number of bits of space per component

    space 

    is the total size of each component

    alignment 

    L or R indicates, respectively, whether the information is left or right-shifted in that space

    In this case, each component in use consumes space bits and those bits must be interpreted as a short integer. (Unused components consume no space).

    For example, here are some common packings (note that the signed-ness of the component values does matter):

              15  int  short 0 
    Packing   +--------------+
    12in16R   0000iiiiiiiiiiii
    S12in16R  ssssiiiiiiiiiiii
    12in16L   iiiiiiiiiiiipppp
    S12in16L  iiiiiiiiiiiipppp
    S12in16L0 iiiiiiiiiiii0000
    

    where s indicates sign-extension, i indicates the actual component information, and p indicates padding (replicated from the most significant bits of information).


    Note: These bit locations refer to the locations when the 16-bit component has been loaded into a register as a 16-bit integer quantity.

    For the most complex formats, the size of every component is specified explicitly, and the entire pixel must be treated as a single 4-byte integer. The size takes the form size1_size2_size3_size4, where size1 is the size of component 1, size2 is the size of component 2, and so on. In this case, the entire pixel is a single 4-byte integer of length equal to the sum of the component sizes. Any space allocated to unused components must be zero-filled. The most common complex packing occurs when 4 components are packed within a 4-byte integer. For example, DM_PACKING_10_10_10_2 is:

                31             int             0
    Packing     +------------------------------+
    10_10_10_2  11111111112222222222333333333344

    where 1 is the first component, 2 is the second component, and so on. The bit locations refer to the locations when this 32-bit pixel is loaded into a register as a 32-bit integer quantity. If only three components were in use (determined from the sampling), then the space for the fourth component would be zero-filled.

  • order is the order of the components in memory. Leave blank for natural ordering (1,2,3,4), use R for reversed ordering (4,3,2,1). For all other orderings, specify the component order explicitly. For example, 4123 indicates that the fourth component is stored first in memory, followed by the remaining three components. Here, we compare a normal, a reversed, and a 4123 packing:

                 31             int             0
    Packing      +------------------------------+
    10_10_10_2   11111111112222222222333333333344
    10_10_10_2_R 4433333333332222222222111111
    10_10_10_2_4123 44111111111122222222223333333333   
     

    where 1 is the first component, 2 is the second component, and so on. Since this is a complex packing, the bit locations refer to the locations when this entire pixel is loaded into a register as a single integer.

DM_IMAGE_COLORSPACE_INT32

The colorspace parameters describe how to interpret each component. The full colorspace parameter is:

DM_COLORSPACE_representation_standard_range

where:

  • representation is either DM_REPRESENTATION_RGB or DM_REPRESENTATION_CbYCr.

    This controls how to interpret each component. Table 7-1 shows this mapping (assuming for now that every component is sampled once per pixel):

    Table 7-1. Mapping colorspace representation parameters

    Colorspace

    Representation

    Component 1

    Component 2

    Component 3

    Component 4

    RGB

    Red

    Green

    Blue

    Alpha

    CbYCr

    Cb

    Y

    Cr

    Alpha


    Remember, the packing dictates the size and order of the components in memory, while the colorspace describes what each component represents. For example, here we show the effect of colorspace and packing combined (again assuming a 4444 sampling, see later).

    Color                    31            int              0
    Standard  Packing        +------------------------------+
    RGB       10_10_10_2     RRRRRRRRRRGGGGGGGGGGBBBBBBBBBBAA
    RGB       10_10_10_2_R   AABBBBBBBBBBGGGGGGGGGGRRRRRRRRRR
    CbYCr     10_10_10_2     bbbbbbbbbbYYYYYYYYYYrrrrrrrrrrAA
    CbYCr     10_10_10_2_R   AAbbbbbbbbbbYYYYYYYYYYrrrrrrrrrr

  • standard indicates how to interpret particular values as actual colors. Choosing a different standard alters the way the system converts between different color representations. The current standards supported are Rec. 601, Rec. 709 and SMPTE 240M.

  • range is either FULL, where the smallest and largest values are limited only by the available packing size, or HEAD, where the smallest and largest values are somewhat less than the theoretical min/max values to allow some "headroom". Full range is common in computer graphics. Headroom range is common in video, particularly when sending video signals over a wire (for example, values outside the legal component range may be used to mark the start or end of a video frame). When constructing a colorspace, you must specify a representation, a standard and a range.

    In Rec. 601 video, the black level (blackest black) is 16 for 8-bit video and 64 for 10-bit video, but in computer graphics, 0 is blackest black. If a picture with 16 for blackest black is displayed by a system that uses 0 as blackest black, the image colors are all grayed-out as a result of shifting the colors to this new scale. Similarly, the brightest level is 235 for 8-bit video and 940 for 10-bit video. The best results are obtained by choosing the correct colorspace.

    Example 7-1. DM_COLORSPACE_RGB_709_FULL

    DM_COLORSPACE_RGB_709_FULL is shorthand for the following:

    DM_REPRESENTATION_RGB
    +
    DM_STANDARD_709
    +
    DM_RANGE_FULL

    where:

    • representation is RGB

    • the standard is 709

    • full-range data is used


DM_IMAGE_SAMPLING_INT32

The sampling parameters take their names from common terminology in the video industry. They describe how often each component is sampled for each pixel. In computer graphics, its normal for every component to be sampled once per pixel, but in video that need not be the case.

For all RGB colorspaces, the only legal samplings are:

  • DM_SAMPLING_444 indicates that the R, G and B components are each sampled once per pixel, and only the first 3 channels are used. If used with an image packing that provides space for a fourth component, then those bits should have value 0 on an input path and will be ignored on an output path.

  • DM_SAMPLING_4444 indicates that the R, G, B and A components are sampled once per pixel.

For all CbYCr colorspaces, the legal samplings include:

  • DM_SAMPLING_444 indicates that Cb, Y, and Cr are each sampled once per pixel and only the first 3 channels are used. If a packing provides space for a 4th channel then those bits should have value 0.

  • DM_SAMPLING_4444 indicates that Cb, Y, Cr and Alpha are each sampled once per pixel.

  • DM_SAMPLING_422 indicates that Y is sampled once per pixel and Cb/Cr are sampled once per pair of pixels. In this case Cb and Cr are interleaved on component 1 (Cb is first, Cr is second) and the Y occupies component 2. If used with an image packing that provides space for a third or fourth component, then those bits should have value 0 on an input path and will be ignored on an output path.

  • DM_SAMPLING_4224 indicates that Y and Alpha are sampled once per pixel and Cb/Cr are sampled once per pair of pixels. In this case Cb and Cr are interleaved on component 1, Y is on component 2, component 3 contains the alpha channel and component 4 is not used (and will have value 0 if space is allocated for it in the packing).

  • DM_SAMPLING_411 indicates that Y is sampled once per pixel and Cb/Cr are sampled once per 4 pixels. In this case Cb, Y is component 2 and Cr occupies component 3. If used with an image packing that provides space for a 4th component then those bits should have value 0 on an input path and will be ignored on an output path.

  • DM_SAMPLING_420 indicates that Y is sampled once per pixel and Cb or Cr is sampled once per pair of pixels on alternate lines. In this case Cb or Cr is interleaved on component 1 and the Y occupies component 2. If used with an image packing that provides space for a 3rd or 4th component then those bits should have value 0 on an input path and will be ignored on an output path.

  • DM_SAMPLING_400 indicates that only Y is sampled per pixel (a greyscale image). Here Y is stored on component 1, all other components are unused. If used with an image packing that provides space for additional components, then those bits should have value 0 on an input path and will be ignored on an output path.

  • DM_SAMPLING_0004 indicates that only Alpha is sampled per pixel. If used with an image packing that provides space for additional components, then those bits should have value 0 on an input path and will be ignored for an output path.

Table 7-2 shows the combined effect of sampling and colorspace on the component definitions:

Table 7-2. Effect of sampling and colorspace on component definitions.

Sampling

Colorspace

Representation

Comp 1

Comp 2

Comp 3

Comp 4

4444

RGB

Red

Green

Blue

Alpha

444

RGB

Red

Green

Blue

 

0004

RGB

Alpha

Y

Cr

Alpha

444

CbYCr

Cb

Y

Cr

0

4224

CbYCr

Cb/Cr

Y

Alpha

0

422

400

CbYCr

CbYCr

Cb/Cr

Y

Y

 

 

420

CbYCr

Cb/Cr*

Y

  

411

CbYCr

Y

Cr

  

0004

CbYCr

Alpha

   

[1]

DM_SWAP_BYTES_INT32

Parameter DM_IMAGE_SWAP_BYTES may be available on some devices. When set to 0 (the default) this has no effect. When set to 1, the device reorders bytes as a first step when reading data from memory, and as a final step when writing data to memory. The exact reordering depends on the packing element size. For simple and padded packing formats (see packings, below) the element size is the size of each component. For complex packing formats, the element size is the sum of the four component sizes.

The swap-bytes parameter reorders bits as follows:

Element Size

Default ordering

Modified ordering

16 bit

[15..0]

[7..0][15..8]

32 bit

[31..0]

[7..0][15..8][23..16][31..24]

other

[n..0]

[n..0] (no change)




[1] * Cb and Cr components are multiplexed with Y on alternate lines (not pixels.)