Video using the Base Overlay

The PYNQ-Z1 board contains a HDMI input port, and a HDMI output port connected to the FPGA fabric of the Zynq® chip. This means to use the HDMI ports, HDMI controllers must be included in a hardware library or overlay.

The base overlay contains a HDMI input controller, and a HDMI Output controller, both connected to their corresponding HDMI ports. A frame can be captured from the HDMI input, and streamed into DDR memory. The frames in DDR memory, can be accessed from Python.

A framebuffer can be shared between HDMI in and HDMI out to enable streaming.

Video IO

The overlay contains two video controllers, HDMI in and out. Both interfaces can be controlled independently, or used in combination to capture an image from the HDMI, process it, and display it on the HDMI out.

There is also a USB controller connected to the Zynq PS. A webcam can also be used to capture images, or video input, that can be processed and displayed on the HDMI out.

The HDMI video capture controller

To use the HDMI in controller, connect the on-board HDMI In port to a valid video source. E.g. your laptop can be used if it has HDMI out. Any HDMI video source can be used up to 1080p.

To use the HDMI in, ensure you have connected a valid HDMI source and execute the next cell. If a valid HDMI source is not detected, the HDMI in controller will timeout with an error.

In [1]:
from pynq import Overlay
from pynq.drivers.video import HDMI

# Download bitstream
Overlay("base.bit").download()

# Initialize HDMI as an input device
hdmi_in = HDMI('in')

The HDMI() argument ‘in’ indicates that the object is in capture mode.

When a valid video input source is connected, the controller should recognize it and start automatically. If a HDMI source is not connected, the code will time-out with an error.

Starting and stopping the controller

You can manually start/stop the controller

In [2]:
hdmi_in.start()
In [3]:
hdmi_in.stop()

Readback from the controller

To check the state of the controller:

In [4]:
state = hdmi_in.state()
print(state)
2

The state is returned as an integer value, with one of three possible values:

  • 0 if disconnected
  • 1 if streaming
  • 2 if paused

You can also check the width and height of the input source (assuming a source is connected):

In [5]:
hdmi_in.start()

width = hdmi_in.frame_width()
height = hdmi_in.frame_height()
print('HDMI is capturing a video source of resolution {}x{}'\
      .format(width,height))
HDMI is capturing a video source of resolution 1920x1080

HDMI Frame list

The HDMI object holds a frame list, that can contain up to 3 frames, and is where the controller stores the captured frames. At the object instantiation, the current frame is the one at index 0. You can check at any time which frame index is active:

In [6]:
hdmi_in.frame_index()
Out[6]:
0

The frame_index() method can also be used to set a new index, if you specify an argument with the method call. For instance:

In [7]:
index = hdmi_in.frame_index()
hdmi_in.frame_index(index + 1)

This will set the current frame index to the next in the sequence. Note that, if index is 2 (the last frame in the list), (index+1) will cause an exception.

If you want to set the next frame in the sequence, use:

In [8]:
hdmi_in.frame_index_next()
Out[8]:
2

This will loop through the frame list and it will also return the new index as an integer.

Access the current frame

There are two ways to access pixel data: hdmi.frame() and hdmi.frame_raw().

In [9]:
from IPython.display import Image

frame = hdmi_in.frame()
orig_img_path = '/home/xilinx/jupyter_notebooks/Getting_Started/images/hdmi_in_frame0.jpg'
frame.save_as_jpeg(orig_img_path)
Image(filename=orig_img_path)
Out[9]:
_images/9_base_overlay_video_20_0.jpg

This will dump the frame as a list _frame[height, width][rgb]. Where rgb is a tuple (r,g,b). If you want to modify the green component of a pixel, you can do it as shown below. In the example, the top left quarter of the image will have the green component increased.

In [10]:
for x in range(int(width/2)):
    for y in range(int(height/2)):
        (red,green,blue) = frame[x,y]
        green = green*2
        if(green>255):
            green = 255
        frame[x,y] = (red, green, blue)

new_img_path = '/home/xilinx/jupyter_notebooks/Getting_Started/images/hdmi_in_frame1.jpg'
frame.save_as_jpeg(new_img_path)
Image(filename=new_img_path)
Out[10]:
_images/9_base_overlay_video_22_0.jpg

This frame() method is a simple way to capture pixel data, but processing it in Python will be slow. If you want to dump a frame at a specific index, just pass the index as an argument of the frame() method:

In [11]:
# dumping frame at index 2
frame = hdmi_in.frame(2)

If higher performance is required, the frame_raw() method can be used:

In [12]:
# dumping frame at current index
frame_raw = hdmi_in.frame_raw()

# dumping frame at index 2
frame_raw = hdmi_in.frame_raw(2)

This method will return a fast memory dump of the internal frame list, as a mono-dimensional list of dimension frame[1920*1080*3] (This array is of fixed size regardless of the input source resolution). 1920x1080 is the maximum supported frame dimension and 3 separate values for each pixel (Blue, Green, Red).

When the resolution is less than 1920x1080, the user must manually extract the correct pixel data.

For example, if the resolution of the video input source is 800x600, meaningful values will only be in the range frame_raw[1920*i*3] to frame_raw[(1920*i + 799)*3] for each i (rows) from 0 to 599. Any other position outside of this range will contain invalid data.

In [13]:
# printing the green component of pixel (0,0)
print(frame_raw[1])

# printing the blue component of pixel (1,399)
print(frame_raw[1920 + 399 + 0])

# printing the red component of the last pixel (599,799)
print(frame_raw[1920*599 + 799 + 2])
175
227
249

Frame Lists

To draw or display smooth animations/video, note the following:

Draw a new frame to a frame location not currently in use (an index different to the current hdmi.frame_index()) . Once finished writing the new frame, change the current frame index to the new frame index.

The HDMI out controller

Using the HDMI output is similar to using the HDMI input. Connect the HDMI OUT port to a monitor, or other display device.

To instantiate the HDMI controller:

In [14]:
from pynq.drivers import HDMI

hdmi_out = HDMI('out')

For the HDMI controller, you have to start/stop the device explicitly:

In [15]:
hdmi_out.start()
In [16]:
hdmi_out.stop()

To check the state of the controller:

In [17]:
state = hdmi_out.state()
print(state)
0

The state is returned as an integer value, with 2 possible values:

  • 0 if stopped
  • 1 if running

After initialization, the display resolution is set at the lowest level: 640x480 at 60Hz.

To check the current resolution:

In [18]:
print(hdmi_out.mode())
640x480@60Hz

This will print the current mode as a string. To change the mode, insert a valid index as an argument when calling mode():

In [19]:
hdmi_out.mode(4)
Out[19]:
'1920x1080@60Hz'

Valid resolutions are:

  • 0 : 640x480, 60Hz
  • 1 : 800x600, 60Hz
  • 2 : 1280x720, 60Hz
  • 3 : 1280x1024, 60Hz
  • 4 : 1920x1080, 60Hz

Input/Output Frame Lists

To draw or display smooth animations/video, note the following:

Draw a new frame to a frame location not currently in use (an index different to the current hdmi.frame_index()) . Once finished writing the new frame, change the current frame index to the new frame index.

Streaming from HDMI Input to Output

To use the HDMI input and output to capture and display an image, make both the HDMI input and output share the same frame list. The frame list in both cases can be accessed. You can make the two object share the same frame list by a frame list as an argument to the second object’s constructor.

In [20]:
from pynq.drivers.video import HDMI

hdmi_in = HDMI('in')
hdmi_out = HDMI('out', frame_list=hdmi_in.frame_list)
hdmi_out.mode(4)
Out[20]:
'1920x1080@60Hz'

To start the controllers:

In [21]:
hdmi_out.start()
hdmi_in.start()

The last step is always to stop the controllers and delete HDMI objects.

In [22]:
hdmi_out.stop()
hdmi_in.stop()
del hdmi_out
del hdmi_in