Does anybody know what triangulation algorithm Maya uses? Lacking that, what would be the most probable algoritms to try? I tried a few simple off the top of my head (shortest/longest resulting edges, smallest minimum angle, smallest/biggest area), but they where all wrong. Is Delaunay the most plausible algoritm?
Edit: by the way, pseudo code on how to implement Delaunay for a 2D quad in 3D space to generate two triangles are more than welcome!
Edit 2: Unfortunately, this is not the answer in 3D-space (only applicable in 2D).
I don't like to second-guess people's intentions but if you are simply trying to get out of Maya what is shown in the viewport you can extract Maya's triangulation by starting with MItMeshPolygon::getTriangles.
(The corresponding normals and vertex colours are straightforwardly accessible. UVs require a little more effort -- I don't remember the details (all my Maya code is with my ex employer) but whilst at first glance it may seem like you don't have the data, in fact it's all there, just not conveniently.)
(One further note -- if your artists try hard enough, they can create polygons that crash Maya when getTriangles is called, even though they render OK and can be manipulated with the UI. This used to happen every few months, so it's worth bearing in mind but probably not worth worrying about too much.)
If you don't want to use the API or Python, then running polyTriangulate before exporting, then undo afterwards (to get back the original polygons) would let you examine out the triangulated mesh. (You may want or need to save the scene to a temp file, then reload it afterwards and use file to give it its old name back, if your export process does stuff that is difficult or impossible to undo.)
This is a bit hacky, but you're guaranteed to get the exact triangulation Maya is using. Rather easier than writing your own triangulation code, and almost certainly a LOT easier than trying to work out whatever Maya does internally...