When I select an NSTreeController in Interface Builder (in Xcode 4.6.3) and look at the attributes inspector, I see two sections named Tree Controller and Object Controller. The Tree Controller part makes some sense, but I'm having trouble finding an explanation of the Object Controller section. The first item is "Mode", with options "Class" and "Entity Name", and I think that you use the latter when using Core Data and the former when not. I'm not using Core Data. So, what's the meaning of the next item, Class Name? It defaults to NSMutableDictionary, but I thought the standard tree node type was NSTreeNode. Specifically, if I want my tree node to be an NSTreeNode whose represented object is an NSMutableDictionary, how would I fill in the Object Controller section?
Post-answer afterthought: Although I now see that one can use nodes of type NSMutableDictionary to build the content tree for an outline view, there are advantages to making a subclass of NSTreeNode. First, NSTreeNode automatically keeps track of parent links, which can be useful in some cases. Second, I can define my data members as properties, and then access them using dot notation rather than objectForKey: messages.
The Object Controller panel is simply Interface Builder's way of allowing you to set either the entityName property of NSTreeContoller or the objectClass property. (NSTreeController is a subclass of NSObjectController, which is where these properties are defined.)
NSTreeNode is a class used by Cocoa to wrap your objects (or entities) before they're placed in the tree. You have no say in this process, it happens automatically and is thus nothing to do with the Attributes Inspector. What's more you'll rarely (never?) need to create an NSTreeNode instance yourself - though you will interact with them pretty regularly.
So it looks like you don't actually need to do anything in this panel - the default values are what you're after. Of course, you'll still need to fill in the Key Paths section.
Here's a very simple demo project (created with Xcode 6.3). Hopefully this will help.