Configure detections-to-segments

The viam:vision:detections-to-segments vision service wraps an existing 2D detector and projects its bounding boxes into 3D point cloud objects using a depth camera’s intrinsic parameters. Each 2D detection produces a corresponding 3D segment that carries the original label and whose point cloud contains the projected pixels from inside the 2D box.

The camera you reference must provide both 2D images and point cloud data (for example, an Intel RealSense D400-series camera configured with the RealSense module). A plain 2D camera will not work.

This model previously shipped with core RDK as detector_3d_segmenter. It now ships as a module at viam-modules/detections-to-segments.

Prerequisites

• A configured 2D detector vision service (for example, an mlmodel or color_detector).
• A camera that returns both color images and a point cloud. See the RealSense and OAK-D modules in the registry.

Install the module

1. In the Viam app, open your machine’s CONFIGURE tab.
2. Click the + icon next to your machine part and select Configuration block.
3. In the search field, type detections-to-segments and select the matching result.
4. Click Add component, enter a name for the service, and click Add component again to confirm. The module is installed automatically.

The module downloads and starts automatically when you save the configuration.

Configure

"services": [
  {
    "name": "<segmenter_name>",
    "api": "rdk:service:vision",
    "model": "viam:vision:detections-to-segments",
    "attributes": {
      "detector_name": "<detector_name>",
      "camera_name": "<camera-name>",
      "mean_k": 5,
      "sigma": 1.25,
      "confidence_threshold_pct": 0.5
    }
  }
]

"services": [
  {
    "name": "my_segmenter",
    "api": "rdk:service:vision",
    "model": "viam:vision:detections-to-segments",
    "attributes": {
      "detector_name": "person_detector",
      "camera_name": "realsense",
      "mean_k": 5,
      "sigma": 1.25,
      "confidence_threshold_pct": 0.5
    }
  }
]

Attributes

Test your segmenter

The segmenter exposes its output through the vision service’s GetObjectPointClouds method.

from viam.services.vision import VisionClient

robot = await connect()
my_segmenter = VisionClient.from_robot(robot, "my_segmenter")

objects = await my_segmenter.get_object_point_clouds("realsense")
for o in objects:
    print(f"label: {o.geometries.geometries[0].label}")

await robot.close()

import (
  "go.viam.com/rdk/services/vision"
)

mySegmenter, err := vision.FromProvider(machine, "my_segmenter")
if err != nil {
  logger.Fatalf("cannot get vision service: %v", err)
}

segments, err := mySegmenter.GetObjectPointClouds(context.Background(), "realsense", nil)
if err != nil {
  logger.Fatalf("could not get segments: %v", err)
}
if len(segments) > 0 {
  logger.Info(segments[0])
}

Troubleshooting

Next steps

Measure depth

Retrieve point clouds and depth images from a depth camera, read depth at specific pixels, and measure distance to detected objects.

mlmodel

Configure the mlmodel vision service to turn a deployed ML model into a detector, classifier, or 3D segmenter.

color_detector

Configure the color_detector vision service to find regions of a specific hue in camera images. No ML model required.