I've been sifting through the Mayavi documentation and Google but I can't find any statement about what algorithm the IsoSurface class uses. If it helps, my source data comes from a 3D NumPy array passed to the mayavi.mlab.pipeline.scalar_field function. Here's the code for using the iso_surface function on an image containing a 3D cube:
import numpy as np
from mayavi import mlab
img = np.pad(np.ones((5,5,5)), 1, mode='constant')
src = mlab.pipeline.scalar_field(img, figure=False)
iso = mlab.pipeline.iso_surface(src, contours=0.5)
The iso_surface function generates an instance of IsoSurface. The code in mayavi\modules\iso_surface.py shows that mayavi.components.contour is used. The comments in mayavi\components\contour.py state that it wraps tvtk.ContourFilter. From the code found at tvtk\tvtk_classes.zip\tvtk_classes\contour_filter.py in my local installation, I found this in the __init__ method for the ContourFilter class:
tvtk_base.TVTKBase.__init__(self, vtk.vtkContourFilter, obj, update, **traits)
Looking at the source code for vtkContourFilter and associated documentation on www.vtk.org I don't see a reference to a publication or the name of the algorithm implemented there.
As you've already discovered, Mayavi's iso_surface module uses (eventually) VTK's vtkContourFilter. There are a couple of sentences in the book "Visualization Toolkit: An Object-Oriented Approach to 3D Graphics, 4th Edition" (Schroeder, Martin and Lorensen) that say something about the algorithms used by vtkContourFilter. This is from p.198 of that book:
Contouring in VTK is implemented using variations of the marching cubes algorithm presented earlier. [...] For example, the tetrahedron cell type implements "marching tetrahedron" and creates triangle primitives, while the triangle cell type implements "marching triangles" and generates line segments.
There's also a vtkMarchingCubes filter that's specific to the case of image data (regularly spaced data on a 1d, 2d or 3d grid); the book goes on to compare execution times between vtkMarchingCubes and vtkContourFilter for a 3d volume.