I am trying to understand the depth APIs in Arcore. I am using a Google Pixel4, which I understand has a depth sensor: https://developers.google.com/ar/discover/supported-devices#google_play https://www.gsmarena.com/google_outlines_how_the_pixel_4s_dual_cameras_capture_depth_in_portrait_photos-news-40597.php
So, if I first call config.setDepthMode(Config.DepthMode.AUTOMATIC);
then frame.acquireDepthImage() is able to return an image.
However, the code below does not find a compatible sensor. If I remove the "REQUIRE_AND_USE" filter: I get 9 cameras all of which have DO_NOT_USE for depth.
private void UseADedicatedDepthSensor_NotWorkingOnPixel4() {
CameraConfigFilter cameraConfigFilter=new CameraConfigFilter(session);
cameraConfigFilter.setDepthSensorUsage(EnumSet.of( CameraConfig.DepthSensorUsage.REQUIRE_AND_USE));
List<CameraConfig> cameraConfigList = session.getSupportedCameraConfigs(cameraConfigFilter);
for (CameraConfig cameraConfig : cameraConfigList) {
String cameraString="camera:"+cameraConfig.getCameraId()+" d:"+cameraConfig.getDepthSensorUsage().toString()+ " s:"+cameraConfig.getTextureSize().toString();
System.out.println();
}
//this will crash as no camera has REQUIRE_AND_USE
CameraConfig supportedCamera=cameraConfigList.get(0);
System.out.println(supportedCamera.toString());
session.setCameraConfig(cameraConfigList.get(0));
}
So my questions are:
Summary at end of answer.
In the ARCore API there is a difference between using the depth sensor for AR tracking and being able to acquire a depth image, and the availability of these two features is completely independent:
Config.DepthMode determines the availability of a depth image for use in your AR-enabled app (e.g. for occlusion, space mapping, and so on). But ARCore makes no guarantees as to where the image returned by acquireDepthImage comes from. In particular, this image may not come from a hardware depth sensor. In many cases it is constructed in software from a combination of what is seen/tracked through the image camera and the movement of the device as reported by the device's IMU sensors. Even in cases where the depth sensor is used by ARCore, the returned depth image may be further processed using additional data from the image camera and sensors rather than being a raw copy of the data directly from the depth sensor.
CameraConfig.DepthSensorUsage determines whether ARCore will use the hardware depth sensor to complement the image camera and IMU sensors for the purposes of tracking the movement of the device in 3D space. This has no relation to whether an image will be available through acquireDepthImage or where that image will come from. The availability of a depth image through acquireDepthImage does not indicate that ARCore is necessarily using the depth sensor for tracking (or at all).
The reason why CameraConfig.DepthSensorUsage.REQUIRE_AND_USE exists is not to ensure the availability of a depth image through acquireDepthImage but to request that ARCore uses the depth sensor to improve movement tracking at the expense of increased CPU/GPU load and power usage. In some cases an app may want to give up the extra tracking accuracy in order to prolong battery life or to keep more processing power available for other tasks.
In addition, the availability of these two features on a given model of device is determined by different factors:
The availability of a depth image through acquireDepthImage is determined by the overall processing power of a device, as producing a depth image from the camera and IMU sensor data requires a significant amount of additional processing overhead than just tracking the movement of the device in 3D space. A device can support acquireDepthImage even if it does not have a depth sensor or ARCore does not support using the depth sensor at all.
The availability of CameraConfig.DepthSensorUsage.REQUIRE_AND_USE is determined by whether or not a) the device has a depth sensor, b) the depth sensor is of sufficient resolution, framerate, and accuracy to be useful for movement tracking, and c) the device has enough processing power to handle the additional processing overhead of including the depth sensor data into the movement tracking calculations.
When looking at the list of supported devices, devices that list "Supports Depth API" should provide a depth image when calling acquireDepthImage, and devices that list "Supports time-of-flight (ToF) hardware depth sensor" should support using CameraConfig.DepthSensorUsage.REQUIRE_AND_USE (although it is noteworthy that only very few devices list this so I suspect that there are more devices that support it but don't have it listed). Again, the "Supports Depth API" feature doesn't make any guarantees that ARCore will use the hardware depth sensor (even if the device has a depth sensor). It is also theoretically possible that a device may support CameraConfig.DepthSensorUsage.REQUIRE_AND_USE but does not provide a depth image through acquireDepthImage.
According to the supported devices list, the Pixel 4 does support the depth API (acquireDepthImage) but does not list hardware depth sensor support.
TL;DR acquireDepthImage and CameraConfig.DepthSensorUsage.REQUIRE_AND_USE are separate APIs that mean different things. acquireDepthImage gives you a depth image, but not necessarily one that is produced by a hardware depth sensor. CameraConfig.DepthSensorUsage.REQUIRE_AND_USE means that a hardware depth sensor will be used for tracking the movement of the device in 3D space. The availability of one of these features doesn't mean that the other is also available. In particular, the availability of acquireDepthImage does not mean that ARCore is using or supports the depth sensor.
Does Pixel4 have a an arcore supported depth sensor?
No, according to the details on the supported devices list. "Supports Depth API" indicates that you can acquire a depth image through acquireDepthImage, not that the hardware depth sensor is supported by ARCore.
If yes, then why are all my camera's depth sensors configured as: DO_NOT_USE
Because ARCore does not support your device's depth sensor.
How come I can still get a depth image?
Either the depth image is produced by software based on the other data that is available to ARCore, or ARCore is using the depth sensor to produce the depth image but does not support using the depth sensor for movement tracking (which is what you are asking for by setting CameraConfig.DepthSensorUsage.REQUIRE_AND_USE).