We're looking for an algorithm to solve this problem in under O(N).
given two real numbers a and b (without loss of generality you can assume they are both between 0 and 1) Find an integer n between -N and N that minimizes the expression:
|a n - b - round(a n - b)|
We have thought that the Euclidean Algorithm might work well for this, but can't figure it out. It certainly looks like there should be much faster ways to do this than via an exhaustive search over integers n.
Note: in our situation a and b could be changing often, so fixing a and b for a lookup table is possible, it gets kind of ugly as N can vary as well. Haven't looked in detail into the lookup table yet, to see how small we can get it as a function of N.
It sounds like you may be looking for something like continued fractions...
How are they related? Suppose you can substitute b with a rational number b1/b2. Now you are looking for integers n and m such that an-b1/b2 is approximately m. Put it otherwise, you are looking for n and m such that (m+(b1/b2))/n = (mb2+b1)/nb1, a rational number, is approximately a. Set a1 = mb2+b1 and a2 = nb1. Find values for a1 and a2 from a continued fractions approximation and solve for n and m.
Another approach could be this:
I'm not too sure it would work though. The accuracy needed for a depends on n and b.