largest prefix of array of vertices that forms a convex polygon

Question

Related to: Polygon Decomposition - Removing Concave Points to Form Convex Polygons

I am looking for an algorithm to do the following:

The input is an array of 2D points (P₀…P_N-1). The length N of the array varies (3 ≤ N < ∞)
For any M ≤ N there may or may not be a convex polygon whose vertices are P₀…P_M-1 in some order.

Note the edges are not necessarily adjacent pairs in the array.

What is the most efficient algorithm to find the maximum M such that this convex polygon exists?

My current algorithm is very inefficient. I test with M=3 then M=4, M=5 etc., compute the hull then test that all P₀…P_M-1 are vertices of the hull, if not then I break out of the loop and return M-1.

Example #1: [(-2,2), (2,2), (-2,-2), (-1,1)]

result: 3 (because the first three points form a triangle but adding P₃ = (-1,1) would make the polygon non-convex)

Example #2: [(-2,2), (2,2), (-2,-2), (1,-1)]

result: 4 (because a convex quadrilateral can be constructed from all 4 points in the array)

Update Example #3: [(-3,3), (3,3), (2,-1), (-3,-3), (3,-3), (-2,1)]
result: 4.

This example demonstrates why it is not sufficient to take the convex hull of all supplied points and find a prefix that is a subset of it. (3,-3) cannot be part of a convex polygon consisting of the first five points because then the previous point (2,-1) would no longer lie on the hull. But it is (3,-3) that must be rejected, even though it lies on the hull of all six points and (2,-1) does not.

Examples of invalid inputs:

• [(-1,-1), (0,0)] (too few points)
• [(-1,-1), (0,0), (1,1), (1, -1)] (first three points are colinear: I would not expect the algorithm to be able to handle this.)

Answer 1

There is a very simple O(m log m) solution to this.

Given that there are at least 3 points and the first 3 are not collinear:

1. Find a point, P, in the triangle of the first 3 points.

2. Sort the 3 points by their angle relative to P (counterclockwise). (This sorted list will be the convex hull)

3. While we aren't at the end of the list, find the next point's position in the sorted list.

4. If inserting the point will make the polygon concave, goto 6. (This can be checked by just checking the new neighboring two turns and the current turn)

5. Insert the point and goto 3.

6. Done.

The main edge case that you have to handle here is when the insertion is at one of the ends of the list, since the list is actually circular. One simple way to handle this is for each point insert it at its angle and at its angle +- 2pi.