Let x^2 + y^2 = r^2 be a circle with r a real.
First I get all integer points that are on the circle (e.g. (1, 2), (-1, 2), (1, -2), (-1, -2), (2, 1), (-2, 1), (2, -1), (-2, -1) for r=sqrt{5})
How can I get the number of quadrilaterals that are possible on with theses points ?
The only way I know is to brute force and test all possible 4-cycles and remove ones with crossing edges but it become too much large for big r. Even for r=sqrt(5) it take about 10 seconds with python.
Change approach, start from a simple problem:
Given a set of poins that lie on a circle:
How many quadrilaterals can you have if the size is 3?
How many quadrilaterals can you have if the size is 4?
now it becomes a bit harder
How many quadrilaterals can you have if the size is 5?
How many quadrilaterals can you have if the size is n?
you don't need to know what coordinates have the points, but just how many are them. Remember: Given 3 points which don't lie on the same straight line you can have only one circle. How ever you get three points from a circle they will never lie one the same straight line. This means that how ever you get 4 points from a circle you can use them to build a quadrilateral