I have a scene in which a Kawasaki robot is solving a problem of moving parts from one place to another.
Above the first place from the top, I want to install a camera to determine the position of objects relative to the edge of the camera image (i.e. in pixels, 2D), for example, using the HSV filter using OpenCV.
How can I simply add a statically attached camera (RGB or monochrome, not a depth camera) to the scene and output an image from it for further processing, for example using OpenCV?
And the second question, is it possible to display a window with an image from the camera in the Meshcat window, which opens in the browser?
I tried to find some examples of adding cameras to a scene, but unfortunately nothing was found. I found some information about pydrake.systems.sensors.RgbdSensor in API Documentation, but it so difficult to understang how to use it.
The pydrake.systems.sensors.CameraConfig is going to be your friend here. It works in conjunction with the pydrake.systems.sensors.ApplyCameraConfig function. The hardware sim example shows how a CameraConfig gets applied (note: the CameraConfig instances in that example actually come from an offline yaml files that get parsed). In that example, the camera mostly uses default values, however, the camera is posed in the world frame.
This will allow you to easily add cameras to your diagram. It will also configure to broadcast LCM messages containing your images (at your requested publication rate). This is one way to "output" images. Were you hoping to capture the images some other way (i.e., in process?)
There is also the possibility of connecting image writers into your diagram that will dump images to your file system, based on how you configure it. Let us know what needs you have in terms of "image output" and I can tailor this answer accordingly.
This is a feature that is currently under development but not yet ready. When it lands, you'll be able to visualize a streaming array of images in a browser window.
In the meantime, you can also turn on the show_rgb flag in your CameraConfig and your simulation process will open a window and display your rendering results. This is a quick and dirty solution and does have some traps if you're hoping to visualize multiple cameras simultaneously (you'll want to make sure that you have independent render engines).