Search code examples
androidopencvimage-processing

opencv deskewing a contour

InputImage

enter image description here

ResultImage

enter image description here

I have been able to filter the largest contour in the image to detect the token.

I have applied warp perception but it is only cropping the image at the edges of the contour, nothing else.

I want the detected token to be cropped out of the rest of the image entireley, de-skew it while keeping proportions so the result image should be upright, straight. Then I will move forward with finding the blobs in the token to detect the dates marked inside it.

    private Mat processMat(Mat srcMat) {
    Mat processedMat = new Mat();
    Imgproc.cvtColor(srcMat, processedMat, Imgproc.COLOR_BGR2GRAY);
    Imgproc.GaussianBlur(processedMat, processedMat, new Size(5, 5), 5);
    Imgproc.threshold(processedMat, processedMat, 127, 255, Imgproc.THRESH_BINARY);
    List<MatOfPoint> contours = new ArrayList<>();
    Mat hierarchy = new Mat();
    Imgproc.findContours(processedMat, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

    double maxVal = 0;
    int maxValIdx = 0;
    for (int contourIdx = 0; contourIdx < contours.size(); contourIdx++) {
        double contourArea = Imgproc.contourArea(contours.get(contourIdx));
        if (maxVal < contourArea) {
            maxVal = contourArea;
            maxValIdx = contourIdx;
        }
    }
    if (!contours.isEmpty()) {
        Imgproc.drawContours(srcMat, contours, maxValIdx, new Scalar(0,255,0),  3);
        Rect rect = Imgproc.boundingRect(contours.get(maxValIdx));
        Log.e("rect", "" + rect);
        int top = srcMat.height();
        int left = srcMat.width();
        int right = 0;
        int bottom = 0;

        if(rect.x < left) {
            left = rect.x;
        }
        if(rect.x+rect.width > right){
            right = rect.x+rect.width;
        }
        if(rect.y < top){
            top = rect.y;
        }
        if(rect.y+rect.height > bottom){
            bottom = rect.y+rect.height;
        }

        Point topLeft = new Point(left, top);
        Point topRight = new Point(right, top);
        Point bottomRight = new Point(right, bottom);
        Point bottomLeft = new Point(left, bottom);

        return warp(srcMat, topLeft, topRight, bottomLeft, bottomRight);
    }
    return null;
}

Mat warp(Mat inputMat, Point topLeft, Point topRight, Point bottomLeft, Point bottomRight) {
    int resultWidth = (int)(topRight.x - topLeft.x);
    int bottomWidth = (int)(bottomRight.x - bottomLeft.x);
    if(bottomWidth > resultWidth)
        resultWidth = bottomWidth;

    int resultHeight = (int)(bottomLeft.y - topLeft.y);
    int bottomHeight = (int)(bottomRight.y - topRight.y);
    if (bottomHeight > resultHeight) {
        resultHeight = bottomHeight;
    }

    Mat outputMat = new Mat(resultWidth, resultHeight, CvType.CV_8UC1);

    List<Point> source = new ArrayList<>();
    source.add(topLeft);
    source.add(topRight);
    source.add(bottomLeft);
    source.add(bottomRight);
    Mat startM = Converters.vector_Point2f_to_Mat(source);

    Point ocvPOut1 = new Point(0, 0);
    Point ocvPOut2 = new Point(resultWidth, 0);
    Point ocvPOut3 = new Point(0, resultHeight);
    Point ocvPOut4 = new Point(resultWidth, resultHeight);
    List<Point> dest = new ArrayList<>();
    dest.add(ocvPOut1);
    dest.add(ocvPOut2);
    dest.add(ocvPOut3);
    dest.add(ocvPOut4);
    Mat endM = Converters.vector_Point2f_to_Mat(dest);

    Mat perspectiveTransform = Imgproc.getPerspectiveTransform(startM, endM);

    Imgproc.warpPerspective(inputMat, outputMat, perspectiveTransform, new Size(resultWidth, resultHeight));
    return outputMat;
}

UPDATE 1

Replaced This:

        return warp(srcMat, topLeft, topRight, bottomLeft, bottomRight);

With This:

        return warp(srcMat, topLeft, topRight, bottomRight, bottomLeft);

Result Update 1:

Result Now

UPDATE 2

    public Mat warp(Mat inputMat, MatOfPoint selectedContour) {
    MatOfPoint2f new_mat = new MatOfPoint2f(selectedContour.toArray());
    MatOfPoint2f approxCurve_temp = new MatOfPoint2f();
    int contourSize = (int) selectedContour.total();
    Imgproc.approxPolyDP(new_mat, approxCurve_temp, contourSize * 0.05, true);

    double[] temp_double;
    temp_double = approxCurve_temp.get(0,0);
    Point p1 = new Point(temp_double[0], temp_double[1]);
    temp_double = approxCurve_temp.get(1,0);
    Point p2 = new Point(temp_double[0], temp_double[1]);
    temp_double = approxCurve_temp.get(2,0);
    Point p3 = new Point(temp_double[0], temp_double[1]);
    temp_double = approxCurve_temp.get(3,0);
    Point p4 = new Point(temp_double[0], temp_double[1]);
    List<Point> source = new ArrayList<Point>();
    source.add(p1);
    source.add(p2);
    source.add(p3);
    source.add(p4);
    Mat startM = Converters.vector_Point2f_to_Mat(source);

    int resultWidth = 846;
    int resultHeight = 2048;

    Mat outputMat = new Mat(resultWidth, resultHeight, CvType.CV_8UC4);

    Point ocvPOut1 = new Point(0, 0);
    Point ocvPOut2 = new Point(0, resultHeight);
    Point ocvPOut3 = new Point(resultWidth, resultHeight);
    Point ocvPOut4 = new Point(resultWidth, 0);
    List<Point> dest = new ArrayList<Point>();
    dest.add(ocvPOut1);
    dest.add(ocvPOut2);
    dest.add(ocvPOut3);
    dest.add(ocvPOut4);
    Mat endM = Converters.vector_Point2f_to_Mat(dest);

    Mat perspectiveTransform = Imgproc.getPerspectiveTransform(startM, endM);

    Imgproc.warpPerspective(inputMat, outputMat, perspectiveTransform, new Size(resultWidth, resultHeight),
            Imgproc.INTER_CUBIC);
    return outputMat;
}

Result Update 2:

I have changed my warp function a bit and the code is attached. However the resultant image is rotated somehow in the wrong direction. Can you guide me which is the correct way to do this.

Android device orientation is set to: portrait and the input image is in portrait as well.

result_update_2

UPDATE 3

I have managed to straighten the token by sorting the corners like so:

    List<Point> source = new ArrayList<Point>();
    source.add(p2);
    source.add(p3);
    source.add(p4);
    source.add(p1);
    Mat startM = Converters.vector_Point2f_to_Mat(source);

Result Update 3:

Result Update 3

However the resultant image is cropped from the left side which I have no idea how to tackle that. I have managed to straighten the input image if the token is tilted to the right or left and the output image is straight nonetheless. However if the input image already has the token centred and straight up. it rotates the token like so, using the same code:

Issue Update 3:

Issue Update 3

Solution

  • The transformation to deskew the ticket is close to an affine one. You can obtain it by approximating the outline with a parallelogram. You find the vertices of the parallelogram as the leftmost, topmost, rightmost and bottommost points.

    Actually, you just need three vertices (and the fourth can be recomputed from these). Maybe a least-square fitting of the parallelogram is possible, I don't know.

    Another option is to consider an homographic transform, which is defined from four points (but the computation is much more complex). It will take perspective into account. (You might get some insight here: https://www.codeproject.com/Articles/674433/Perspective-Projection-of-a-Rectangle-Homography.)

    To straighten up the image, it suffice to apply the inverse transform and retrieve a rectangle. Anyway, you will notice that the size of this rectangle is unknown, so that you can scale it arbitrarily. The hardest issue is to find a suitable aspect ratio.