Using ISI (Image Sensor Interface) in Linux4SAM 6.0 and later

---

Introduction

This page is mainly about how to enable and configure the ISI in AT91SAM SoCs for different image sensors, in our Linux4SAM 6.0 and later releases.
For older releases and older kernel, check this page.

The Image Sensor Interface (ISI) connects a CMOS-type image sensor to the processor and provides image capture in various formats from sensor side.
ISI uses H/VSYNC signal for synchronization or EAV/SAV.
ISI supports the following sensor input formats : YCbCr422, RGB565, RGB888 and grayscale raw data.

ISI supports the following output formats:

• ISI has two paths (Preview path and Codec path) for the output.
  • Preview path will output RGB data with different format.
    • Can convert YCbCr or YUV to RGB.
      • Support downscale and decimation.
      • Max output solution is 640x480.
  • Codec path will output YUV data with different orders.
    • Can convert RGB to YCrCb.
    • Max output solution is 2048x2048.

Prerequisites

• • In SAMA5D3x-EK boards:
    • J11's PIN29 need to be disconnected when insert ISI module board. Since PIN29 is connected to ISI_D11 (pin mux as TWD1).
    • TWI0 (i2c0)'s TWD0 & TWCK0 signals are using same pins as ISI_VSYNC & ISI_HSYNC. So need to disable i2c0 when you enable ISI.
    • TWI1 (i2c1)'s TWD1 & TWCK1 signals are using same pins as ISI_D11 & ISI_D10. So it cannot support 12bit data input.
    • Led d3 should be disable as well as it conflict with camera sensor's reset pin PE24.
    • These products have been deprecated in linux4sam releases.
  • The main board for ISI feature display is SAM9X60-EK with the SAM9X60 MPU.
  • Other MPUs or other boards do not support ISI or have been deprecated in linux4sam.
  • Supported CMOS sensors: Omivision OV2640 , Omnivision OV7740 and Micron/Aptina monochrome sensor MT9V022

Tips : The ISI Device Tree support is added as an overlay in our dt-overlay-at91 tree. For any questions check our DT-overlay page .

Detail description of software

ISI driver in Linux kernel

ISI driver is part of the media platform drivers, and it supports the standard v4L2 APIs.
Current supported sensors: Omivision OV2640 , Omnivision OV7740 and Micron/Aptina monochrome sensor MT9V022.
It's easy to support a sensor if the sensor is using DVP interface connection. This means it uses H/VSYNC signals. Sensors with serial interface are not supported, only parallel.
soc-camera framework was deprecated. There are still sensors in soc_camera, but most sensors were ported to the platform camera framework. Thus we do not recommend using soc_camera sensors.
Supported sensors are available in platform camera framework. You can find all camera supported platform sensors in Kernel menuconfig by:

• select the menu: "Device Drivers -> Multimedia support -> Media Controller API"
• select the menu: "Device Drivers -> Multimedia support -> V4L2 sub-device userspace API"
• deselect the menu: "Device Drivers -> Multimedia support -> Autoselect ancillary drivers (tuners, sensors, i2c, frontends)"
  • 
• Then all support sensors can be found in the menu: "Device Drivers -> Multimedia support -> I2C Encoders, decoders, sensors and other helper chips"
  • 

Tips: to add a new sensor support, you need to create a Device Tree Overlay file to add the sensor remote port. The DT-overlay page provides more information. Our public Github repository will gladly accept patches.

User Applications to try ISI functionality

fswebcam

fswebcam is a neat and simple webcam app. It captures images from a V4L1/V4L2 compatible device or file, averages them to reduce noise and draws a caption using the GD Graphics Library which also handles compressing the image to PNG or JPEG. The resulting image is saved to a file or sent to stdio where it can be piped to something like ncftpput or scp.

• Add fswebcam in Buildroot
  • Select "Package Selection for the target -> Graphic libraries and applications -> fswebcam".
• Use fswebcam to capture a image.

  #!/bin/sh
  
  VIDEO_DEV=/dev/video0
  SKIP_FRAMES=20
  
  # test preview channel
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p RGB565 -r 640x480 rgb565.jpg
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p RGB565 -r 320x240 rgb565_defactor.jpg
  
  # test codec channel
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p YUYV -r 640x480 yuyv.jpg
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p YUYV -r 800x600 yuyv_800x600.jpg
  
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p UYVY -r 640x480 uyvy.jpg
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p UYVY -r 800x600 uyvy_800x600.jpg
  
  # test codec channel, without any processing, GREY, or Bayer RGB.
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p BAYER -r 640x480 bayer_bggr8.jpg
  fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p SGRBG8 -r 640x480 bayer_grbg8.jpg
  
     

  • -S: frames that need to skip.
  • -d /dev/video0: specify the ISI as the input source.
  • -p: pixel format, can be RGB565, YUYV, UYVY, BAYER, SGRBGB8 and etc.
  • -r: resolution.

FFmpeg

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It supports video4linux2 in Linux.

• Add FFmpeg in Buildroot (It's already included in the Linux4SAM buildroot demo).
  • Select "Package Selection for the target -> Audio and video applications -> ffmpeg".
• Use FFmpeg to capture a mpeg4 video clip.

     ffmpeg -r 25 -s vga -t 20 -pix_fmt yuyv422 -f video4linux2 -i /dev/video0 video.avi
     

  • -r: frame rate
  • -s: resolution, it can be qcif, cif, qvga, vga, svga, xga, uxga.
  • -t: time duration in second.
  • -pix_fmt: pixel format, only support yuyv422.
  • -f video4linux2: specify the format. Use ffmpeg -formats will show all supported formats.
  • -i /dev/video0: specify the ISI as the input source. Run following command to check the source name:

    # cat /sys/class/video4linux/video0/name
    isi-camera
             

  • video.avi: output video file name

Tips: As no vcodec specified, it use mpeg4 as default.

Tips: Run ffmpeg -pix_fmts can show all the supported pixel formats.

• Use FFmpeg application to capture images.

     ffmpeg -f video4linux2 -r 1 -s vga -t 4 -i /dev/video0 -pix_fmt yuyv422 -f image2 -vcodec png image%d.png
     

  • -r: frame rate
  • -s: resolution, it can be qcif, cif, qvga, vga, svga, xga, uxga.
  • -t: time duration in second.
  • -i /dev/video0: specify the ISI as the input source. Run following command to check the source name:

    # cat /sys/class/video4linux/video0/name
    isi-camera
             

  • -pix_fmt: pixel format, only support yuyv422.
  • -f image2: image2 sequence format. Use ffmpeg -formats will show all supported formats.
  • -vcodec png: specify output format is png. Use ffmpeg -codecs will show all supported codec.
  • image%d.png: output file name is image1.png, image2.png and etc.

GStreamer

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

GStreamer has been ported to a wide range of operating systems, processors and compilers.

• Add GStreamer in Buildroot (It's already included in the Linux4SAM Buildroot demo).
  • Select "Package Selection for the target -> Audio and video applications -> gstreamer" and the plugins that you needed.
• Use GStreamer to preview on LCD.

     # gstreamer 1.0
     gst-launch-1.0 v4l2src device="/dev/video1" ! video/x-raw,width=640,height=480 ! videoconvert ! kmssink
     

  • v4l2src: a plugin to support v4l2 device as a source
    • device="/dev/video1": specify the ISI as the v4l2 input device.
      • You can check the device name by run command: cat /sys/class/video4linux/video1/name
    • video/x-raw,width=640,height=480: For gstreamer 1.0, specify the v4l2 output video format and size.

  *=videoconvert=

  a plugin to convert from one color space to another

  • kmssink: a plugin to render to KMS (kernel mode setting) device

Tips: run gst-inspect will show all installed plugins.

Tips: run gst-inspect [plugin name] will show all supported parameters for this plugins.

ZXing barcode reader

ZXing is an open-source, multi-format 1D/2D barcode image processing library implemented in Java, with ports to other languages. The project also includes a barcode reader example.

• Add the ZXing barcode reader application in Buildroot.
  • Add ZXing libary in Buildroot (The Linux4SAM buildroot demo already include it).
    • Select "Package Selection for the target -> Libraries -> Graphics -> zxing".
  • Apply attached patch on top of the buildroot-2012.11.1-at91.
    • This patch will change the zxing project's Makefile to generate not only zxing library but also the barcode reader example.
  • Run command make zxing to generate the barcode application: zxing_barcode.
    • zxing_barcode is located on outpout/target/use/bin/.

• Read barcode from the image by using ZXing barcode reader.
  • Get an picture which include a barcode.
    • Please refer to FFmpeg section for the image capture.
  • Run following command to reader the barcode.

    zxing_barcode *.jpg

FAQ

Q: How to check the ISI is probed in my board?

1. Check the boot message whether there is an information about sensor probe.

   
   i2c i2c-0: OV2640 Probed
   ...
   ov5642 0-003c: reg_read: i2c read error, reg: 300a
   ov5642: probe of 0-003c failed with error -121
         

   In above example, the message shows an OV2640 sensor is probed. But probe of OV5642 failed as we only support one camera module slot in the board.
2. Print all the v4L2 device in system to check that ISI device exists?

   # ls /sys/class/video4linux/video*
   /sys/class/video4linux/video0:
   debug      dev        index      name       power      subsystem  uevent
   
   /sys/class/video4linux/video1:
   debug      device     name       subsystem
   dev        index      power      uevent
   # cat /sys/class/video4linux/video0/name
   
   # cat /sys/class/video4linux/video1/name
   isi-camera
       

   In above example, we can find the video1 is the isi-camera device.

Q: If the ISI cannot be probed in my board, what should I do?

1. Check the kernel boot message to see if there is any error message about ISI and sensor
2. Following the below check list to troubleshoot above errors.
   • Check the kernel config file:
     • Is the ISI driver and the sensor driver enabled?
   • Check your board's device tree files and overlays (.dts, .dtsi .dtso):
     • Is the ISI device node is enabled?
     • Are the ISI pins are configured correctly?
     • Is the sensor's i2c info correct?
     • Is the sensor's power/reset pin correct?
     • Is the PCK correct?

Reference

• V4L2 documentation
• ZXing project page
• FFmpeg home page
• GStreamer home page