ToF cameras are commonly used in applications such as smartphones and consumer electronics, biometric authentication and security, autonomous vehicles and robotics, etc.
HBVCAM supply professional and honest service.
Based on the methods of determining the distances, ToF is of two types. Direct Time of Flight (DToF) and Indirect Time of Flight (iToF). Let us look into each of these in detail and their differences.
Direct ToF method works by scanning the scene with pulses of invisible infrared laser light. The light reflected by the objects in the scene is analyzed to measure the distance.
The distance from the dToF camera to the object can be measured by computing the time period taken by the pulse of light from the emitter to reach the object in the scene and then back again. For this, the dToF camera uses a sensing pixel called a Single Photon Avalanche Diode (SPAD) The SPAD can detect the sudden spike in photons when the pulse of light is reflected back from the object. This way, it keeps track of the intervals of the photon spike and measures the time.
dToF cameras are usually low-resolution. The advantages of dToF sensors are that they are compact and are inexpensive. dToF cameras are ideal for applications which does not require high-resolution and real-time performance.
Let us look into the details of LiDAR, a dToF technology that uses infrared laser beams for measuring distances.
LiDAR (Light Detection and Ranging) cameras use a laser emitter to project a raster light pattern on the scene being recorded and scan it back and forth.
The camera sensor records the time period it takes for the pulse of light to reach the object and reflect back to itself. This enables the LiDAR system to measure the distance to the object that reflected light based on the speed of light.
LiDAR sensors can be used to build up a 3D map of the area being recorded. These sensors can scan the scene to produce a single frame of data based on the distance measured and the direction of the light being directed. These capture frames will have data points in hundreds or even thousands. Once the LiDAR system completes the scan, it generates a point cloud based on the position of the acquired points. One added advantage of LiDAR systems is that they can be used in continuous streaming mode.
Various LiDAR systems are used for different applications. Since LiDAR systems use a collimated and focused laser beam to scan the area in a target, they prove effective even at long ranges. Low-power LiDAR sensors are also available and can be used for depth sensing in short ranges. They can also be used in varied lighting conditions but are sensitive to ambient light in outdoor lighting.
LiDAR sensors usually use infrared lasers in one of two wavelengths: 905 nanometers and nanometers. The shorter wavelength lasers are less likely to be absorbed by water in the atmosphere and are more useful for long-range surveying. In comparison, the longer wavelength infrared lasers can be used in eye-safe applications like robots operating around humans.
Indirect Time of Flight (iToF) cameras illuminate the entire scene using diffuse infrared laser light with the help of multiple emitters. This illumination is done in a series of modulated laser pulses. The light is continuously modulated by pulsing the laser emitters at a high frequency.
In iToF, the time interval between each pulse of light isn't measured directly, as in the case of dToF. The iToF camera records and compares the phase shift of the waveform, which gets recorded in the sensor pixel. A comparison is made between each pixel on how much the waveform is phase-shifted, by which the distance to the point in the scene can be calculated.
With iToF cameras, the distance to all points in the scene can be determined with a single shot.
If you are interested in knowing more about how a time-of-flight camera works:
Read: What is a ToF sensor? What are the key components of a ToF camera?
Read: How Time-of-Flight (ToF) compares with other 3D depth mapping technologies
A structured light-based depth-sensing camera uses a laser/LED light source to project light patterns (mostly a striped one) onto the target object. Based on the distortions obtained, the distance to the object can be calculated. A structured light 3D scanner is often used to reconstruct the 3D model of an object.
Structured light cameras are for applications such as 3D scanning and modeling, quality control and inspection, biometric authentication, cultural heritage preservation, retail, and e-commerce, etc.
All the 3D depth mapping cameras we discussed above come with their own pros and cons. The choice of camera used will entirely depend on the specifics of your end application. It is always recommended that you get the help of an imaging expert like e-con Systems to guide you through the camera evaluation and integration process.
The technologies can be analyzed using ten different parameters. A detailed comparison is given in the below table:
STEREO VISION STRUCTURED LIGHT dToF iToF Principle Compares disparities of stereo images from two 2D sensors Detects distortions of illuminated patterns by 3D surface Time taken by reflected light from an object to the sensor is measured Phase shift of modulated light pulses is measured Software Complexity High Medium Low Medium Material Cost Low High Low Medium Depth('z') Accuracy cm um~cm mm~cm mm~cm Depth Range Limited Scalable Scalable Scalable Low light Weak Good Good Good Outdoor Good Weak Fair Fair Response Time Medium Slow Fast Very Fast Compactness Low High High Medium Power Consumption Low Medium Medium Scalable-MediumAs mentioned before, depth sensing is required for any device that has to navigate autonomously. However, given below are some of the most popular embedded vision applications that need 3D depth cameras for their seamless functioning:
Autonomous Mobile Robots (AMR) have helped automate various tasks across industrial, retail, agricultural, and medical applications. Following are a few examples of AMRs used in warehouses, retail stores, hospitals, office buildings, agricultural fields, etc.
Whatever the type, any robot that has to move autonomously without any human supervision needs to have a 3D depth camera in it. Some robots that use a depth-sensing camera might use a combination of automated and human-aided navigation. Even in those cases, depth cameras help detect obstacles and avoid accidents. An example of this type of robot is a delivery robot.
To develop a better understanding of how depth cameras function in AMRs, please have a look at the article How does an Autonomous Mobile Robot use time of flight technology?
Over the last several articles, we have looked at different approaches to visual SLAM ' direct and indirect methods. Both methods of SLAM were described assuming using either a single camera or a stereo camera. Getting the proper depth of features and objects in the field of view of the camera drives much of how well SLAM performs. I wanted to touch on how depth cameras fit into these visual SLAM methods, and how they can improve the performance of SLAM algorithms.
If you are looking for more details, kindly visit Depth Camera Module.
There are a number of different approaches to getting depth information, and we'll take a look at three types of depth cameras. Each approach has its own advantages, and can be affected by the environment it is used in.
We discussed how stereo cameras work in a previous article. In essence, stereo cameras work by matching pixels between the images from each camera and triangulating the pixel depth using the baseline (distance between the cameras).
Two key advantages of stereo cameras are the cost of components, and how it performs outdoors in bright areas. On the other hand, the algorithmic cost in calculating the depth is relatively high, and low light performance can be poor. The accuracy of depth is driven by three factors: baseline distance, resolution, and lens focal length. A wider baseline and focal length provide better accuracy at range, but at the same time increases the minimum distance where depth can be determined. Rectification is also another important factor in stereo cameras and it is directly affected by the camera calibration. Using rectification, the corresponding scanlines of the left and right cameras are found and used for matching the featured pixel in the left and right images.
Some systems add a projector to a single or stereo camera system. This method uses the projection to either simplify the correspondence matching between the camera images, i.e., structured-light or measure the depth directly, i.e., Time-of-Flight (ToF). As a major advantage, this type of approach performs very well in low light environments, and if used in a stereo setup, the system could fall back to the standard passive stereo depth when objects are out of the projector range or if the projection is overpowered by ambient light.
Structured light and coded light cameras use a conceptually similar approach in using a single camera but introducing an IR projector paired to the camera sensor. The projector emits a known pattern (typically lines or speckled dots, see below) on the scene and analyzes the distortions of these lines or dots to determine the depth.
The first Microsoft Kinect ( ' ) used structured light developed by PrimeSense before Apple acquired them in . The Kinect brought depth sensors to the mass market (at $150, it was at least a magnitude cheaper than previously available solutions), and opened the door for a lot of research around the applications of depth sensors, including RGB-D SLAM.
Structured light sensors provided highly accurate depth data, at a relatively high-resolution, and operated well in low light environments. However, it is more expensive than a stereo camera module and requires more power to run the system (both to run the projector and compute the depth). Also, the range is limited (typically under 5 ' 7m) and doesn't work well on highly reflective surfaces, or in outdoor environments, where the sun would overpower the projector.
As the name implies, a ToF camera aims to measure the time which takes the light to be captured after bouncing off objects illuminated by a light source (typically LED, IR, or laser) paired with the camera. In pulsed light systems, the camera shutter is synchronized with the illuminator where the delay in returning light pulse is measured to calculate the distance to the reflection point. In contrast, continuous light systems will emit modulated amplitudes of light, and then measure the returning light to estimate the distance.
The Microsoft Kinect for Xbox One launched in , replaced the structured light system with a ToF system, and we see an increasing number of mobile phones with depth capabilities using ToF sensors as well.
ToF cameras provide an optimized (however far from ideal) solution compared to passive stereo cameras and structured light cameras in that they have the lowest depth computing cost, relatively high resolution, good depth accuracy (~1cm), and ToF camera modules typically cost less than structured light systems. Similar to structured light cameras, ToF cameras are affected by reflective surfaces, may have interference from other ToF cameras, and have limited range. Environments with multiple reflective surfaces will also diminish the accuracy of depth. The range can be scaled by increasing the power of the light emitter or using lasers, but at the expense of power and cost.
The table below attempts to generalize the different depth cameras characteristics based on commonly available systems we have used as part of our SLAM solution.
The mass-market availability of the Microsoft Kinect sensor renewed research activity in RGB-D SLAM, and techniques taking advantage of a high-resolution depth map created from it. For both direct and indirect methods of visual SLAM, the use of structured light and ToF cameras meant a gain in overall performance ' both in accuracy and computational load.
If you recall from previous articles on direct and indirect SLAM, both methods require accurate depth estimates. For stereo systems, pixels between the left and right images were matched and depth derived by triangulating over the known baseline. For mono camera systems, image frames over time were used to get a baseline for points of interest. This was a computationally expensive process in the overall SLAM pipeline, and in the mono camera case, acquiring an accurate measure of the baseline over time was very difficult. The introduction of structured light and ToF cameras helped alleviate both these problems by reducing the amount of computation needed, and at the same time providing accurate depth for every pixel (within range).
However, the limitations of these cameras also limited scenarios and use cases for systems using these cameras:
' Primarily indoor operations where the source projection or lighting didn't get overpowered by the ambient lighting
' Challenges in large open spaces that exceed the range of the projector or light emitter, where there are limited detectable surface or object within range at any given point
' Limit on highly reflective surfaces or transparent surfaces in the environment
' Different camera systems in each category have markedly different fall-offs in the accuracy of depth
As you can see, there isn't a single perfect solution for visual SLAM. What you end up with is a complex matrix of factors that need to be considered when choosing what type of SLAM to implement, and with what type of sensors. Some basic questions will include:
' What is the expected operating environment?
' Indoor, outdoor, open space, dynamic environment with moving objects and people, dark, etc'
' Are there a lot of features? Does lighting or the scene change over time?
' Will there be multiple devices running SLAM in the same space?
' How large is the operating environment?
' Is it a room? Multiple rooms?
' A shopping mall or conference center?
' Stadium?
' How fast are things moving?
' The device that is running SLAM
' Other objects in the environment
' People's safety concerns?
' Is this a known area where mapping can be done prior to operation?
' What are your cost constraints?
' Can you afford a LiDAR or ToF sensor in your system?
' What are your computing constraints?
' High-end computer?
' Raspberry Pi?
' What are your memory constraints?
' 16GB?
' 1GB?
I'll leave you with these questions for now. I will cover LiDAR SLAM in my next article, and then return to this question of how to decide on what kind of SLAM could be used in various use cases and scenarios.
For further reading:
Newcombe, et al, 'KinectFusion: Real-Time Dense Surface Mapping and Tracking,' (PDF)
For more information, please visit 1MP Global Shutter Camera.