I'm not able to directly download some source files from my project using Trac.
I understand that it's possible to use a SCM client, like git or svn to do this, but surely it can be done without leaving Trac!
Do I need some plug-in, or is there a command or configuration that I've missed?
You don't need a plugin for this. When using the Repository Browser, look for a "Download In Other Formats" link at the bottom of pages. Individual files will have an "Original Format" link that lets you download the original file, and folders will have a "Zip Archive" link to download the entire folder.
Update:
Also, make sure that you have Trac configured with the repository paths that you want to make downloadable. In trac.ini, look for a property named downloadable_paths in the [browser] section. The repository paths listed here (and only those listed here) will be made available for download through the web interface. For example, my trac.ini has the following:
[browser]
downloadable_paths = /trunk, /branches/*, /tags/*
For more information about this configuration parameter, see the official Trac documentation for trac.ini.