Can anyone give advice, or point to any guides, on how to manage a community of open source software developers in writing API documentation?
A typical, unmanaged, starting point for most projects is to have a project wiki where anyone can freely create pages, add content to existing pages, edit existing content etc. The problem is that, despite people's best intentions, the wiki can easily end up being a disorganised, poorly written, incomplete, written in disparate voices etc etc.
So, what to do to improve the quality of the documentation?
I suspect a key ingredient is clear editorial/style guidelines, something similar to http://en.wikipedia.org/wiki/Wikipedia:Encyclopedic_style#Information_style_and_tone. Can anyone point to an example of such a guide tailored specifically to software APIs?
Are there any other practices that people have found useful? E.g. form a core team of editors and accept that most documentation that gets added by the community will most likely need to be 'strongly edited'?
The short answer, that the solution is social/human and not technical. The way to get good documentation for any project is to have someone with time, in charge of doing high level organization for the documentation, and then being involved in the development and user communities to ensure that the documentation remains up to date and continues to address the problems and confusions that users typically have.
Community projects have accepted that you need point people (i.e. "managers," for aspects of the project like "translation," and "release," and for various components. The same thing needs to happen for documentation.
As for tools, Sphinx is really great though it's not "wiki like," exactly you can use whatever version control system your project is comfortable with to store documentation and configure your web server to rebuild the documentation following commits/updates/pushes. Which has always worked just fine for any project I've worked on/with.