RGBD Cameras: From Color to Distance

RGBD cameras can not only capture vibrant, colorful images but also perceive how far or near objects are, giving machines true stereo vision. That’s why RGBD cameras enable robots to precisely grasp a cup without bumping into a table leg.

What is an RGBD Camera?

In an RGBD camera, RGB refers to the color camera, i.e., the RGB camera. Color cameras are called RGB cameras because they synthesize full-color images by capturing the three basic color lights: Red, Green, and Blue. These three colors represent the spectral bands to which the cone cells in the human retina are most sensitive. When mixed, they can reproduce the vast majority of colors visible to the human eye.

The D in RGBD refers to the Depth Camera. Depth cameras based on different principles use various methods to measure the depth (i.e., distance) of an object from the camera. In the depth map generated by a depth camera, every point contains depth information.

The “color images (RGB)” and “depth maps (Depth)” captured by RGBD cameras serve as the fundamental input for RTG-SLAM to perform environment reconstruction. The system analyzes both types of data: on one hand, it uses depth information to fit the geometric surface of the scene; on the other hand, it uses color information to restore the appearance details of the scene. Ultimately, it achieves real-time environment reconstruction that is both “geometrically accurate” and “realistic in appearance.”

---

How Do RGBD Cameras Work?

The working principles vary depending on the depth-sensing technology adopted by different RGBD cameras. This article elaborates on each approach.

Structured Light

Structured light solutions project specially encoded patterns, spots, or stripes onto a target. When these patterns encounter height variations on the object’s surface, they become distorted. This method calculates the object’s three-dimensional structure by recognizing the degree of deformation in the projected pattern.

Structured light systems generally come in two types: monocular structured light and binocular structured light. A monocular structured light system consists of one projector and one camera. It is low-cost but also has lower performance, which is why it is less commonly used today.

Binocular structured light consists of one projector and two cameras. It calculates depth by analyzing the disparity of corresponding feature points in the images captured by the left and right cameras. Binocular structured light can be regarded as a binocular system with a built-in “texture generator,” capable of overcoming the matching difficulties encountered by RGB binocular solutions in textureless areas. By combining the advantages of binocular stereo vision and active structured light, it is currently more widely used.

Like RGB binocular systems, structured light solutions also perform ranging using the triangulation principle. This method offers high accuracy at close range but poor accuracy over long distances. Moreover, in high-ambient-light environments, the projected structured light can be easily disrupted.

---

Binocular Stereo Vision

When two cameras capture the same scene simultaneously, the difference in camera perspectives causes the same point in space to appear at different locations in the left and right images. This difference is called disparity. Disparity is directly related to the distance of an object: nearby objects exhibit large disparity, while distant objects exhibit small disparity. The RGB binocular stereo solution obtains depth information precisely through this disparity. As a passive measurement method (without actively emitting light), RGB binocular stereo vision offers low power consumption and relatively low hardware costs.

However, the algorithmic complexity of RGB binocular stereo vision is relatively high, placing significant demands on computational resources and presenting considerable challenges for real-time performance. Furthermore, this solution only achieves high accuracy at close range, with poor accuracy at long distances. Depth measurement in RGB binocular stereo vision is highly dependent on scene texture. In areas such as solid-colored walls, dark regions, or textureless surfaces, where matching points are difficult to find, the resulting depth map may exhibit numerous holes. Additionally, performance degrades significantly in low-light environments.

---

Time-of-Flight (ToF)

ToF solutions obtain distance information based on the time of flight. They can be categorized into two types: iToF (Indirect Time-of-Flight) and dToF (Direct Time-of-Flight).

dToF cameras emit short laser pulses toward the target. They perform a direct distance calculation by precisely measuring the time-of-flight from the emission of the light pulse to its return. The dToF solution offers a long measurement range, reaching tens or even hundreds of meters, and its measurement accuracy does not degrade significantly with increasing distance. It also features strong resistance to interference and can operate in high-ambient-light environments. However, dToF solutions generally have lower resolution. They also impose high requirements on chip design and manufacturing processes, making the technology challenging.

iToF cameras emit a modulated continuous light wave and indirectly calculate the distance by measuring the phase difference between the reflected light and the emitted light. iToF technology is mature, offers high resolution, benefits from a well-established industrial chain, has relatively low costs, and provides high measurement accuracy at close range (typically within 5 meters). However, its accuracy decreases over long distances. It is also susceptible to interference from ambient light, with performance degrading significantly in strong light conditions.

---

Key Specifications of RGBD Cameras

–Spatial Resolution: The number of pixels in the depth image, i.e., the width and height of the depth map, determining the level of detail in the depth image (e.g., 640×480, 1280×720).

–Field of View (FOV): The range of the scene that the RGBD camera can capture, including the horizontal FOV and vertical FOV (e.g., 87°×58°, 110°×70°). A larger FOV covers a wider scene range but may compromise some edge accuracy.

–Depth Range: The closest and farthest distances the camera can accurately capture, typically expressed in meters (m). Depth measurement ranges vary considerably depending on the depth-sensing technology employed. For example, structured light RGBD cameras generally have a range of 0.1–2m, while ToF RGBD cameras can achieve ranges from 0.1m to 70m or even further.

–Depth Accuracy: The measurement accuracy of the RGBD camera in the depth direction, typically expressed in millimeters (mm) (e.g., ±1mm, ±5mm). Higher accuracy leads to more precise measurement results.

–Frame Rate: The number of depth maps and color images captured per second by the RGBD camera, expressed in Frames Per Second (FPS). Common frame rates include 30fps, 60fps, 90fps, etc. High frame rates are suitable for dynamic scene capture and real-time applications.

–Power Consumption: The electrical power consumed by the camera during operation, expressed in Watts (W). Low power consumption extends battery life, a particularly important feature for applications such as robotics.

–Synchronization Accuracy: For RGBD cameras, the temporal-spatial synchronization accuracy between the color image and the depth image affects the precision of image fusion and 3D reconstruction. Hardware-level synchronization can typically achieve microsecond accuracy, while software synchronization generally achieves millisecond accuracy.

---

Representative Products of RGBD Cameras

Type	Product	Resolution	Frame Rate (FPS)
Structured Light	Microsoft Kinect v2	1920×1080	30
	Intel RealSense D415	1920×1080	30
	Apple PrimeSense	1280×960	60
	Xtion PRO LIVE	1280×1024	60
	Optonic ensneso	1280×1024	10
Stereo Vision	STEREOLABS ZED	3840×1080	100
	FLIR Bumblebee	1280×960	16/20/48
	DUO 3D	640×480	98
	Rubedes Viper	1280×720	40
Time-of-Flight (ToF)	PMD CamCube 3.0	–	40/60/80
	Mesa Swiss Ranger	176×144	50
	Sony DepthSense 525	320×240	–
	Terabee 3Dcam	640×480	30
	Google Phab2 Pro	640×480	–

---

Advantages of RGBD Cameras

Advantage	Description
Directly outputs a dense depth map	No algorithmic estimation needed; directly obtains physical distance per pixel
High close-range accuracy	Millimeter-level depth accuracy within a <4m range, far superior to monocular depth estimation
Active sensing is usable in low light	Built-in infrared light source; can obtain a depth map even in darkness
Simple and efficient point cloud generation	Depth map + intrinsics direct back-projection; generate point cloud with one line of code
Color + depth fusion	Natural integration of semantic recognition and spatial localization, comprehensive perception capability
Supports 3-Dimensional reconstruction	Algorithms such as KinectFusion and NeRF can directly use RGBD data

Limitations of RGBD Cameras

Limitation	Description
Higher cost than RGB cameras	Consumer-grade RGBD cameras typically cost above $70 USD.
Limited measurement range	Typically 0.2–6 m; depth accuracy drops sharply at longer distances.
Outdoor sunlight interference	For structured light and iToF solutions, the infrared signal is overwhelmed under strong sunlight, causing depth failure.
Blind spots for specular/transparent materials	Specular reflections and transparent materials prevent correct depth acquisition.
Relatively high power consumption	Requires continuous IR emission; typical power consumption is 2–5 W.
Depth maps contain holes	Depth missing in occluded areas, black materials, and specular surfaces; post-processing required.
Lower depth resolution than color	Typically 640×480 or 640×576, lower than that of comparable RGB cameras.

---

RGBD Camera Price (For Reference Only)

	Price in USD (approx.)
Entry-level	60–100
Mid-range	100–300
High-end	300–700
Professional	Above 700

Component	Cost Percentage	Major Manufacturers
Image Sensor	28-32%	Sony
VCSEL Laser	15-20%	Lumentum, II-VI
Depth Computing SoC	18-22%	TI
Optical Lens	8-10%	Sunny Optical
Near-infrared LED	4-6%	Luminus

---

Applications of RGBD Cameras

–Hand-Eye Coordination in Humanoid Robots: Humanoid robots represent the application scenario where the value of RGBD cameras is most evident. RGBD cameras integrated into the wrist or head of humanoid robots (e.g., Figure 01, Boston Dynamics’ Atlas) serve two purposes: the RGB image is responsible for identifying object type and pose (“this is a glass”), while the depth map provides the precise 3D position of the object (“the glass is 0.32m from the gripper with a height difference of 0.05m”). After fusing this information, the motion planner outputs an accurate 6-DOF grasping trajectory.

Compared to solutions using ordinary RGB cameras, where robots must estimate depth from images (monocular depth estimation) with errors typically ranging from centimeters to tens of centimeters — an accuracy level that is meaningless for grasping actions — RGBD cameras directly compress the positional error to <5mm, fundamentally solving the problem of “seeing but not reaching.”

–Obstacle Avoidance and Navigation in Service Robots: Mid-to-high-end service robots feature a forward-facing RGBD camera. The advantage over pure LiDAR solutions lies in the fact that LiDAR only scans a single horizontal plane (typically about 15–20cm from the ground), leaving obstacles above that plane invisible. In contrast, RGBD cameras cover the entire 3-Dimensional space within their field-of-view cone — table legs, chair backs, low-hanging banners, and the arms of walking people can all be detected. In practical products, RGBD cameras are often used in conjunction with LiDAR: LiDAR is responsible for long-range 2-Dimensional map building and localization, while the RGBD camera handles fine-grained 3-Dimensional obstacle perception at close range (<4m).

–Fine-Grained Close-Range Perception in AGVs: Warehouse AGVs (Automated Guided Vehicles) navigating between shelves require precise perception of: the 3D contour of shelf heights and goods, to plan forklift height; the 3D shape of scattered objects like pallets and boxes on the floor, to plan detour paths; and the precise location of barcodes/QR codes on goods (combined with RGB), to guide alignment for loading. RGBD cameras offer higher accuracy than pure 2D vision and richer semantic information than LiDAR, making them the standard choice for such close-range, fine-grained perception scenarios.

Recommended In-depth Reading from AI Robots Eidos

RGBD cameras integrate color (RGB) cameras and depth (D) sensors to simultaneously capture the color, texture, and three-dimensional spatial information of objects. In industrial scenarios, this enables machine vision systems to not only recognize the category of objects but also accurately locate their position, orientation, and size.

For readers interested in further understanding RGBD cameras, they can refer to this article on machine vision.

---

Market Size of RGBD Cameras

The data represents institutional forecasts and is provided solely for the reader’s reference.

	Data	Source
Global RGBD camera market (2024)	$ 0.7B–1.1B	Yole Group
Global RGBD camera market (2025)	$1.2B	IDC 2024
2030 Robotics 3-Dimensional perception market	$ 2.5B–3.5B	Yole Group

Image Credits: Baslerweb & Loshield & Mdpi& Researchgate & Mrdvs

Articles about other Industrial Cameras

TDI Cameras: Sharper, Faster, Brighter