CEVA-MM3101, a Programmable, High Performance, Low Power Imaging and Computer Vision DSP IP Platform for Camera-Enabled Devices
Leading device OEMs (smartphones, tablet computers, portable media players, portable gaming consoles, etc.) are looking to differentiate their products from the competition. One way to achieve this is by providing unique and innovative applications such as image enhancement in low-light handling, sophisticated Natural User Interface (NUI) like Gesture Recognition, Augmented Reality (AR) and depth sensing.
The Automotive market is on the move. Various regulations and technology changes are pressing more and more cameras into this space and related technologies such as Advanced Driver Assistance Systems (ADAS) applications and Human Machine Interface (HMI) applications are being included in newer models.
CEVA-MM3101 applications
In the Surveillance market, while the cameras becomes more and more intelligent – most of the video analysis is done online and not on the server and requires massive computer vision support.
The home entertainment market, the emergence of Digital Televisions is driving designers to support state-of-the-art capabilities such as gesture recognition and face detection, recognition and beautification. These devices also need to provide enhanced encoders for video conferencing and special post-processing functions for display.
Other emerging markets like Wearable devices also have a requirement for low power consumption, yet look for flexibility during product development. A flexible solution that answers the power and performance requirements is crucial for the product success.
CEVA-MM3101 IP platform enables these and many other complex computer vision applications while using its extremely efficient performance and low power capabilities to execute multiple applications in parallel.
The CEVA-MM3101 offers a scalable solution that can support a wide variety of products and use-cases. CEVA-MM3101 is augmented by a robust set of full product algorithms coming both from in-house as well as partners technology. The design of imaging and vision applications is accelerated by a full set of software libraries, software tools including CEVA-CV computer vision libraries and ability to offload the CPU by a dedicated framework.
By offloading the device’s main CPU and the GPU/GP-GPU for performance-intensive imaging & computer vision processing tasks, the highly-efficient CEVA-MM3101 dramatically reduces the power consumption of the overall system, while providing complete flexibility. Algorithm developers can leverage the CEVA-MM3101’s programmable architecture to implement their own proprietary software, thereby addressing unique use-cases and providing exceptional functionality that truly differentiates their products.
CEVA-MM3101 Target Applications
The CEVA-MM3101 can be used in SoCs that offload application processors (APs) in mobile devices, automotive, surveillance applications and home entertainment systems. Computational photography is happening and requiring intensive performance with flexibility to try new algorithms and adopt them quickly and easily.
Camera enabled devices required image and video enhancement as well as support for scene analysis functions such as face tools (detection, recognition, and beautification), motion detection, gesture recognition, and object detection. The different devices equipped with more than one camera – from 3 in smartphones and more than 6 in future automobiles – requires advanced algorithms like stitching and registrations.
The CEVA-MM3101 addresses all of these functions for the most demanding applications, including rich computer vision library (based on OpenCV) with optimized kernels for CEVA-MM3101 platform. The CEVA-MM3101 can be easily integrated into any SoC using its simple interface between the host CPU and the CEVA-MM3101. For example the CEVA-MM3101 equipped with Android interface framework which enables fast and easy integration to Android-based systems.
| Features | Benefits |
|---|---|
| Extremely parallel Vector processor engine designed specifically for imaging and computer vision algorithms using low power mechanisms | Enables support for the most computation demanding CV algorithms? in low power consumption |
| Advanced IDE, optimizing C compiler, profiler, debugger and simulator, libraries, RTOS and device driver support | Eases software development, reduces resource requirements, and speeds time to market |
| Dedicated? Application Development Kit (ADK) for offloading computer vision tasks from CPU to CEVA-MM3101 | Seamless CV development in optimal performance and low power consumption in a simplified manner |
| Android Multimedia Framework (AMF) reference design | Time-To-Market with ready-to-go Android implementation and reference platform |
| CEVA-CV Extensive Library>750 optimized computer vision functions from OpenCV optimized for CEVA-MM3101. | Abstracts developments and enables development using standard widely-used OpenCV |
| Off-the-shelf pre-optimized software products from CEVA (such as Digital Video Stabilizer and Super-Resolution) and partners (such as Gesture Recognition, Face Detection & Recognition, Emotion Detection and ADAS) | Easy to build multi-purpose products. Can utilize of CEVA eco-system to expand your own eco-system |
Architectural Highlights
At the heart of the CEVA-MM3101 is an efficient & programmable Vector Processing (VP) engine. The Vector Processor performs filtering and the vector-type operations required for pixel processing. It is based on a dedicated pixel-processing VLIW/SIMD architecture with 10-stage pipeline. It contains 7 different units that can work in parallel enabling flexible combination for different type of instructions. All instructions are conditional execution using predication optimized to save code size. The VP can handle 32 byte operations in a single cycle and contains special instructions which can be configured in order to create proprietary filters for video and imaging processing. This strong processor is optimized for multimedia operations, capable of 64 SAD calculations in a single cycle as well as producing 8 results for 6 Tap filters – which make it ideal for imaging algorithms.
CEVA-MM3101 architecture
With built in techniques for reducing data bandwidth transfer from the DDR to the core and vice versa, and unique patents for data folding and processing on the fly and enhance internal memory structure, the VP can handle large amounts of data for burst mode image pipeline requirements as well as HD video encoding and decoding without hurting its performance. The VP architecture was specifically designed to meet the low-power requirements of mobile and home entertainment devices. Supplied with optimized kernels for image pre and post processing it enables CEVA customers, partners and 3rd parties to develop their own proprietary applications easily and rapidly.
Multi-Image Super-Resolution
The CEVA Super-Resolution (SR) algorithm increases quality of still images for mobile devices. In today’s world, where smartphones are the most commonly used camera devices, image quality is still limited due to slim form factor and strict power constraints. Super-Resolution is an important breakthrough, enabling systems manufacturers and OEMs to increase the quality of images on any device, regardless of these factors.
The CEVA SR creates a high resolution, high quality image using multiple consecutive low resolution images. This technique is done using the additional information within the multiple input images in order to increase the details of the output image.
The CEVA SR algorithm first handles registration between the input images and then fuses them together into a single, high quality image while supporting ghost removal caused by motion between the consecutive images.
Performance Comparison:
| SR Method | Complexity [op / pixel] | Real-time consideration |
|---|---|---|
| Technical Papers | More than 10,000 | Iterative, high BW |
| Commercial PC Application | 100-400 | |
| CEVA SR on MM3101 | Less than 100 | |
| Performance: Up to 16 cycles/pixel | Utilizes SIMD and specific instructions |
The CEVA SR algorithm includes various image enhancement and computational photography algorithmic techniques. It was developed in-house by CEVA’s software engineering experts and is optimized specifically for the CEVA-MM3101, utilizing CEVA’s advanced Compiler technology and using the CEVA-CV library functions in order to shorten development time.
Quality Comparison:?Original image up-scaled via Bi-cubic filter (on left) vs. CEVA SR (on right)
PC SR Application versus CEVA SuperRes
Bicubic interpolation versus CEVA SuperRes
Bicubic interpolation versus CEVA SuperRes
PC SR Application versus CEVA SuperRes
Bicubic interpolation versus CEVA SuperRes
Bicubic interpolation versus CEVA SuperRes
PC SR Application versus CEVA SuperRes
Bicubic interpolation versus CEVA SuperRes
| Features | Benefits |
|---|---|
| Receive multiple images and fuse them into a single high-resolution image – For example 5MPixel into 20MPixel image | Effectively upgrade existing sensors to higher quality ones |
| Ultra low power: <30mW @28nm | Extremely efficient for extending battery life |
| Up to 4x resolution increase vs. original picture | Allows high-quality digital zoom in real-time |
| No special requirements on the sensor | Could be implemented on any device |
| Configurable software-based solution | Supports various number of input imagesSupports any image size |
| Algorithm utilizes parallelism and efficiency of MM3101 architecture and ISA | Very efficient implementation, able to take multiple shots in single second |
Digital Video Stabilizer
In the ultra-competitive market for smartphones and other camera-enabled mobile devices, OEMs are looking for every advantage possible. Improved video and embedded vision performance and features are critical areas to be able to differentiate with. However, mobile devices present unique challenges to achieving optimal camera performance. For one, handheld devices incur lots of motion which degrades the quality of the recorded videos. In addition CMOS sensors today mostly use a rolling shutter which causes a video effect known as “Jello effect” which shows jerkiness of an object in the video stream. In order to reduce these effects and to achieve smoother video recording, good stabilization of the recorded video streams is required.
The CEVA Digital Video Stabilizer (DVS) is an advanced software application that is able to handle both shakiness (in multiple axes) and rolling shutter effects to output smoother video streams in any light conditions for embedded devices such as smartphone, tablets and others.
The DVS module was developed and supported by CEVA’s in-house imaging team of experts. It is fully optimized to run in real-time using minimal processing workloads and very low memory bandwidth on the CEVA-MM3000 platforms. For example, in a 28nm process, the
CEVA-MM3101 consumes less than 35mW power while running DVS on a 1080p30 fps video stream in real-time. Addressing higher resolution requirements, the DVS algorithm scales up to 4K Ultra HD video streams on a single CEVA-MM3101 platform.
Delivery is given in source code format to allow users to adapt the algorithm to their specific needs and requirements, which can be done over the existing extensive configurations parameters already available.
As a software module the DVS supports various configuration options which may be changed according to the user specific requirements for quality, performance and accuracy:
- Rolling shutter correction – should calibrate rolling shutter correction strength per specific sensor.
- Size and coordinates of stabilized area– can configure the size and coordinates (cropped area) to stabilize from in the input stream.
- Max rotation correction in angles.
- Smoothness level on panning, rotating, zooming.
The DVS can be easily coupled with additional imaging software modules, such as Super-Resolution, Color Enhancement and High Dynamic Range, on the fully programmable
CEVA-MM3000 platform.
The DVS algorithm was implemented utilizing CEVA’s new Application Developer Kit (ADK). The ADK serves to substantially simplify the overall software development experience, shortening the product design cycle and providing significant memory bandwidth and power consumption savings for imaging and vision applications.
CEVA Digital Stabilizer Quality Comparison:
Device used
Below are sample streams comparing the quality of CEVA DVS vs. few leading Smartphones. For each comparison we have filmed side-by-side using two identical phones, one phone with stabilization activated (the phone built-in stabilization) and one with stabilization disabled. Then we applied CEVA DVS over the stream without stabilization and compared the result with the original stabilized stream.
As shown, CEVA Stabilizer delivers significantly smoother videos versus leading Smartphone built-in stabilizers.
| Features | Benefits |
|---|---|
| Ultra low power: <35mW @28nm | Extremely efficient for extending battery life |
| Support for various lighting conditions | Could be utilized in any environment |
| No special requirements on the sensor | Could be implemented on any device |
| Configurable software-based solution | Supports parameter configuration to better tune to specific device and user, supports any video resolution |
| Delivered in source code format |