The Video subpackage contains a collection of drivers for reading from the HDMI-In port, writing to the HDMI-Out port, transferring data, setting interrupts and manipulating video frames.
The Video hardware subsystem consists of a HDMI-In block, a HDMI-Out block, and a Video DMA. The HDMI-In and HDMI-Out blocks also support color space conversions, e.g. from YCrCb to RGB and back, and changing the number of channels in each pixel.
Video data can be captured from the HDMI-In, and streamed to DRAM using the Video DMA, or streamed from DRAM to the HDMI-Out.
The Pixel Unpack and the Color Convert block allow conversion between different color spaces at runtime.
The HDMI-Out is similar to HDMI-In. It has a Pixel Pack block (instead of the Unpack block for HDMI-In) and a Color Convert block.
The video library supports two different video front-ends. The front-end is responsible for converting the signals at the pins of the device into a 24-bit-per-pixel, BGR formatted AXI stream that can be used by the rest of the pipeline. For the Pynq-Z1 and Pynq-Z2 a DVI-based front-end provides for resolutions of up to 1080p although due to the speed ratings of the differential pins only up to 720p is officially supported. This front-end is a drop-in IP included in the PYNQ IP library. For the ZCU104 the Xilinx HDMI subsystems are used to support full 4k support however recreating this bitstream will require a license. The HDMI subsystems also require connecting to a HDMI PHY responsible for driving the transceivers. This code can be seen in the ZCU104’s base.py. All custom overlays needing to use the video subsystem should use this setup code.
There are 3 main ways that some processing could be carried out on a Video pipeline.
- Read, write and process frames in Python on the PS
- Modify the overlay to insert an IP into the video pipeline
- Modify the overlay to connect an IP to an available interface in the overlay
Input and output frames are represented as numpy arrays frames are read from the HDMI input and written to the HDMI output. The HDMI input will return the most recent complete frame received or block if necessary until the frame is ready. The HDMI output will continue outputting the last frame written until a new frame is provided.
As frames are numpy arrays they can be directly used in other Python libraries including the Python OpenCV libraries.
Examples of using the video API with OpenCV can be found in the video notebooks.
Processing video in the PS will be relatively slow. Where possible low level processing should be carried out in the PL. The video subsystem supports basic color space and pixel type conversions in hardware before passing a frame to the Python environment to improve performance of OpenCV and processing by other libraries.
The HDMI signal blocks are AXI-stream. A custom AXI-stream IP with an input stream and output stream could be inserted into the video pipeline, either on the HDMI-In side, or HDMI-Out side. However, usually IP to process the video stream will assume a fixed color space/pixel format and this will determine where an IP should be connected in the Video pipeline.
It is usually appropriate to insert the IP after the pixel_pack block on the HDMI-In block, or before the pixel_unpack block on the HDMI-Out side. This gives flexibility to use the video subsystem color space conversion blocks before and after the custom IP.
The video pipelines of the Pynq-Z1 and Pynq-Z2 boards run at 142 MHz with one pixel-per-clock, slightly below the 148.5 MHz pixel clock fo 1080p60 video but sufficient once blanking intervals are taken into account. for the ZCU104 board the pipeline runs at 300 MHz and two pixels-per-clock to support 4k60 (2160p) video.
An IP block can be added to an overlay and connected to an available interface. Usually a DMA would be used to stream the input frame buffer to the IP, and send the processed data back to the output frame buffer in DRAM.
Note that the DRAM is likely to be a bottleneck for video processing. The Video data is written to DRAM, then read from DRAM and send to the custom IP and is written back to DRAM, where it is read by the HDMI out.
For the Pynq-Z1 which has a 16-bit DRAM, up to 1080p cwgraysc (8-bits per pixel) can be processed at ~60fps alongside the framebuffer memory bandwidth, but this is very close to the total memory bandwidth of the system. The ZCU104 with its much larger memory bandwidth can support 4k video at 60 FPS at 24-bit colour.
More information about the Video subpackage, its components, and its their APIs can be found in the pynq.lib.video Module section.
For more examples, see the Video Notebooks folder on the Pynq-Z1, Pynq-Z2 or ZCU104 board in the following directory:
Set up an instance of the HDMI-in, and HDMI-out.
from pynq import Overlay from pynq.lib.video import * base = Overlay('base.bit') hdmi_in = base.video.hdmi_in hdmi_out = base.video.hdmi_out
The HDMI-in interface is enabled using the
configure function which can
optionally take a colorspace parameter. If no colorspace is specified then
24-bit BGR is used by default. The HDMI-in mode can be used to configure the
HDMI-out block. This specifies the output color space and resolution.
Once the HDMI controllers have been configured, they can be started:
To connect a simple stream from HDMI-in to HDMI-out, the two streams can be tied together.
This takes the unmodified input stream and passes it directly to the
output. While the input and output are tied frames can still be read from the
input but any call to
hdmi_out.writeframe will end the tie.
frame = hdmi_in.readframe() ... hdmi_out.writeframe(frame)
This would allow some processing to be carried out on the HDMI-in frame before writing it to the HDMI-out.
Color Space Conversion¶
The video subsystem supports general color space conversions so that frames in DRAM are in a format appropriate for any subsequent processing. The default color space is BGR(24-bit) with RGB (24-bit), RGBA (32-bit), BGR (24-bit), YCbCr (24-bit), and grayscale (8-bit) provided as built-in options.
The colorspace converter operates on each pixel independently using a 3x4 matrix to transform the pixels. The converter is programmed with a list of twelve coefficients in the following order:
Each coefficient should be a floating point number between -2 and +2.
The pixels to and from the HDMI frontends are in BGR order so a list of coefficients to convert from the input format to RGB would be:
[0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0]
reversing the order of the channels and not adding any bias.
The driver for the colorspace converters has a single property that contains the list of coefficients.
colorspace_in = base.video.hdmi_in.color_convert colorspace_out = base.video.hdmi_out.color_convert bgr2rgb = [0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0] colorspace_in.colorspace = bgr2rgb colorspace_out.colorspace = bgr2rgb
The default pixel format for the HDMI frontends is 24-bit - that is three 8-bit channels. This can be converted to 8, 16, 24 or 32 bits.
8-bit mode selects the first channel in the pixel (and drops the other two) 16-bit mode can either select the first two channels or select the first and performs chroma resampling of the other two resulting in 4:2:2 formatted frames. 24-bit mode is pass-through, and doesn’t change the format 32-bit mode pads the stream with additional 8-bits.
pixel_in = base.video.hdmi_in.pixel_pack pixel_out = base.video.hdmi_out.pixel_unpack pixel_in.bits_per_pixel = 8 pixel_out.bits_per_pixel = 16 pixel_out.resample = False
hdmi_out.writeframe functions may
potentially block if a complete frame has not yet been read or written,
_async versions of these functions also exist. One use for the asynchronous
versions is if frames are being transferred to a separate accelerator using a
DMA engine. The DMA driver is also asyncio aware so the computation can be
written as two tasks. One to retrieve frames from the Video DMA and forward them
to the accelerator and a second task to bring frames back from the accelerator.
async def readframes(): while True: frame = await hdmi_in.readframe_async() dma.sendchannel.transfer(frame) await dma.sendchannel.wait_async() frame.freebuffer() async def writeframes(): while True: frame = hdmi_out.newframe() dma.recvchannel.transfer(frame) await dma.recvchannel.wait() await hdmi_out.writeframe_async(frame)
Zynq Ultrascale+ DisplayPort¶
On Zynq Ultrascale+ devices there is a hardened DisplayPort interface that may
be exposed on the board. On all supported boards the PYNQ environment will
bring up a Fluxbox-based desktop environment with the Chromium browser to allow
easy access to Jupyter directly on the board. For high-performance video output
over DisplayPort the PYNQ environment offers a
DisplayPort class that
offers a similar API to the HDMI output. The only change between the
DisplayPort and HDMI outputs is that the colourspace cannot be changed
dynamically and frames are not interoperable between the two subsystems. While
PYNQ is using the DisplayPort output it will replace the desktop environment.