Search code examples
javaimageopencvdjl

Crop image by polygon area or mask

How can I crop an image using a Polygon or Mask? Needed in order to detect an object only in a specific area in the image.

Example red area

eg, I think here to crop the image https://github.com/akuzyagin/video-detection-example/blob/2a92fe0a17c8c499f348fb1db8ace465c4d1e3b2/src/main/java/DetectionService.java#L45C1-L47C72:

Image image = ImageFactory.getInstance().fromImage(frame);
//TODO crop image by polygon before detect objects
DetectedObjects predict = predictor.predict(image);

Full code: https://github.com/akuzyagin/video-detection-example/

I tried to use the ai.djl.modality.cv.Image.getMask method passing an array of coordinates, but I didn't quite understand how this image cropping method works. I also tried various methods from ai.djl.modality.cv.util.NDImageUtils, but nothing seems to fit from the implemented methods.

Also tried converting to BufferedImage https://github.com/deepjavalibrary/djl/issues/1432 and further cropping the image Crop image by polygon area in Java , but in the logs there are errors by:

BufferedImage bufferedImage = (BufferedImage) result.getWrappedImage();

Stacktrace:

2023-09-06 14:30:40,104 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.ClassCastException: class org.opencv.core.Mat cannot be cast to class java.awt.image.BufferedImage (org.opencv.core.Mat is in unnamed module of loader io.quarkus.bootstrap.runner.RunnerClassLoader @45283ce2; java.awt.image.BufferedImage is in module java.desktop of loader 'bootstrap')
    at DetectionService.detect(DetectionService.java:82)
    at DetectionService_Bean.create(Unknown Source)
    at DetectionService_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
    at DetectionService_Bean.get(Unknown Source)
    at DetectionService_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:476)
    at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:489)
    at io.quarkus.arc.impl.ArcContainerImpl.instance(ArcContainerImpl.java:287)
    at DetectionService_Observer_Synthetic_ebc203dc60d4841a41795405dc16fe346fbb52eb.notify(Unknown Source)
    at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:328)
    at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:310)
    at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:78)
    at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:131)
    at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:100)
    at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
    at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:108)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
    at ApplicationMain.main(ApplicationMain.java:10)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:61)
    at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:32)
Solution

  • I found the answer to my question, the following code allows you to cut out a polygon from an image: Cropped image

    // create polygon with coordinates for mask
MatOfPoint points = new MatOfPoint(
      new Point(100, 400),
      new Point(500, 350),
      new Point(700, 300),
      new Point(1000, 500),
      new Point(1000, 800),
      new Point(700, 800),
      new Point(500, 800),
      new Point(100, 400)
);

// draw shape with size of frame from original video and black fill
final Scalar colorBlack = new Scalar(0, 0, 0);
Mat mask = new Mat(frame.rows(), frame.cols(), CvType.CV_8UC3, colorBlack);

// fill polygon of white color on the black fill
final Scalar colorWhite = new Scalar(255, 255, 255);
Imgproc.fillPoly(mask, Collections.singletonList(points), colorWhite, Imgproc.LINE_8);

// compute the bitwise_and of the frame and mask
Mat result = new Mat();
Core.bitwise_and(frame, mask, result);

Image croppedImage = ImageFactory.getInstance().fromImage(result);

    Full code: https://github.com/akuzyagin/video-detection-example/