3D Sensors become a quasi-standard for high-end smartphones for best photography and immersive AR experiences. The latest 3D image sensor from Infineon and pmdtechnologies enables a new generation of applications. It aims to create most immersive and smarter AR experiences as well as better photography results with a faster autofocus in low-light conditions or more beautiful night mode portraits based on picture segmentation. In this interview Philipp von Schierstaedt, Senior Vice President RF & Sensors, Infineon Technologies AG shares more insights on these latest 3D sensors and its benefits.
Q: Can you please explain 3D depth sensor technology and its benefits?
The principle operating behind the REAL3 3D depth sensors is called Time of Flight (ToF).This technology enables electronic devices to acquire a 3D depth map of the scene in front of the device. The surroundings, objects and people are transformed into the digital space in real time. Algorithms use that data to measure distances and sizes, to track motions and to convert the shape of objects into 3D models. The Infineon 3D ToF sensors – developed in cooperation with pmdtechnologies –are designed to be integrated into the smallest 3D camera modules, accurately measuring depth in the short and longer range at the lowest power consumption.
Q: What are the key factors driving use of 3D sensors in smartphones?
The phone camera performance is a key purchase criteria for today’s generations of consumers. In recent years,3D image sensing technologies like ToF favored the emergence of new applications and further improvement of the camera’s capabilities. Many of these applications are today market-proven and becoming practically a standard feature of the front camera in premium phones – for example applications such as biometric facial recognition for secure phone unlock and payment, virtual reality avatar for social media interaction or photography beautification with special depth effects.
It is expected that the further growth of the 3D image sensor market will be particularly favored by the expansion of 3D cameras to the rear side of the smartphone. According to Strategy Analytics, the market for ToF rear cameras in smartphones is expected to grow above 500 million units per year worldwide within the next five years. AR/RM gaming together with a wide variety of applications in the commercial, leisure/sport, educational, and enterprise categories are expected to gain traction. All these new applications – enabled by novel 3D image sensor technologies – steer an improved user experience, and act again as driving factor for increased adoption of time-of-flight sensors in smartphones.
Q: What are the latest technology trends in 3D cameras?
In general we detected a wave of innovation enabled by 3D cameras in various markets – not only smartphone but also in other consumer markets with domestic service robots and AR/MR wearable (headset) applications as well as in the automotive market in applications supporting car’s automated driving. In addition, the evolution of smart factories is creating new opportunities for the deployment of 3D image sensors in factory robots, in monitoring and vision systems, and in automated guided vehicles. Particularly speaking about smartphones, the technology trends in 3D cameras moves towards covering the full range of distances (from just a few cm for face ID on the front camera to 5m or longer on the rear camera), near-professional photographic effects, 3D depth data acquisition and processing in real time, small camera form factor, and very importantly low power consumption for extended usage times of the device. This is absolutely relevant for always-on applications like 3D gaming.
Q: Can you please discuss the market growth of 3D ToF cameras in smartphones? What are its key features and benefits?
We see a development towards 3D ToF image sensors as a standard for high-end smartphones on the front camera – flagship phones are already proving this – and on the rear camera. The fastest growing will surely be the rear camera with above 500 million units/y over the next years. The deployment of 3D sensing technologies will follow this growth as they solve the challenges that new applications set to the rear camera and that traditional 2D technologies fail to accomplish: a real 3D depth map. Real AR/MR experiences require a full understanding of the spatial environment, with a high level of 3D immersion and ubiquitous interaction, with virtual images perfectly rendered into the real world. Still, challenges exist, for example current commercial devices deploying existing 3D image sensing solutions are not yet able to provide a reasonable resolution without significant trade-offs on extended distance ranges and on the power consumption.
Such shortcomings are now being addressed by the latest 3D ToF imager generations in the REAL3 portfolio of Infineon. Therefore we’re convinced that we will see our new 3D sensor either enabling new classes of applications or taking existing applications to a new level of quality and user experience.
I would like to explain this in three features.
Firstly, the sixth generation of our REAL3 3D ToF sensors has been particularly optimized to address the challenges of AR/MR and photography applications in smartphones. Precise depth data, which is taken super-fast, enables the scanning of a device’s environment. This lets you capture places and objects not only in 2D, but in the way they are: three dimensional. With the unique flexibility of our ToF technology, we enable picking up details with several 10k depth points as well as capturing larger rooms with a coarser resolution. In contrast, other solutions available on the market only offer a resolution of approx. 600 depth points at this distance, which means that small structures cannot be resolved.
Secondly, our lately announced imager provides a low power consumption and a miniaturized form factor, which has never been there before for this application. In fact, the sensor requires 40 percent less power than previous generations, thus saves the smartphone’s battery power consumption. This is key e.g. for AR/MR gaming, where the ToF camera is on for a long time. The camera design with this new imager can be 35 percent smaller due to high SoC integration level and so gives phones more freedom for advanced designs and hence being more cost efficient.
Thirdly, the new imager and system is designed for providing depth data from distances of up to 10m, without losing the high resolution on short range. This performance is achieved regardless of the ambient light, from dim light to bright sunlight, which is becoming very important especially in modern photography. This superior performance is the result of our collaboration with the experts of pmdtechnologies thanks to the patented SBI concept which is integrated on each pixel of all REAL3 sensors.
- Reference: Strategy Analytics experts, October 2020
