If I open a notebook in Mathematica, and type "nb=NotebookCreate[]", it creates a new notebook window, as expected.
However, if I try the same command from within C#, it fails, and doesn't open a new notebook window:
IKernelLink ml = MathLinkFactory.CreateKernelLink();
ml.WaitAndDiscardAnswer();
string cmd = @"nb = NotebookCreate[]";
string result = ml.EvaluateToOutputForm(cmd, 0);
// result is now "$Failed".
Note that if I pass in "2+2", it returns "4", which proves the link to the Mathematica kernel is working perfectly well.
Unfortunately, it appears as if this is impossible for Mathematica v8 (and I don't think much has changed in v9).
From what I understand, when you access the Mathematica kernel from C#, this does not allow access to the Notebook classes, which are part of the GUI.