There are two polytopes A and B in R^3 with empty intersection. The polytopes are defined by its faces, i.e. there are only inequalities for its hyperspaces and vertexes are unknown. The problem is to find points a in A and b in B such that ||a-b|| = d(A,B) -- distance between A and B. Also we can formulate this problem for R^2 or R^d for d>3. What is the approach for this problem. And does this problem have some applications?
This paper formulates the problem of finding the distance between two generic convex sets.
They go on and provide plenty of applications, including the distance between two convex polytopes. The minimum distance between two polytopes is the dual of finding a maximum separating hyperplane. They provide formulations of this problem and show as an implementation a proof of Gordan's Theorem of Alternatives. It is formula (11.1) that provides the formulation you are asking, but some manipulations are required to bring the polytopes to that forms. Depending on the selected norm, the problem can be recast as a linear (L1 norm), quadratic (L2 norm) or general program.
Also, the references given therein (about finding the nearest point in the polytope) as relevant.
Abstract:
In this paper we explore the duality relations that characterize least norm problems. The paper starts by presenting a new Minimum Norm Duality (MND) theorem, one that considers the distance between two convex sets. Roughly speaking the new theorem says that the shortest distance between the two sets is equal to the maximal “separation” between the sets, where the term “separation” refers to the distance between a pair of parallel hyperplanes that separates the two sets.
The second part of the paper brings several examples of applications. The examples teach valuable lessons about the role of duality in least norm problems, and reveal new features of these problems. One lesson exposes the polar decomposition which characterizes the “solution” of an inconsistent system of linear inequalities. Another lesson reveals the close links between the MND theorem, theorems of the alternatives, steepest descent directions, and constructive optimality conditions.
I hope this helps!