We're considering a "functional branch" strategy with DEV, QA, and RELEASE branches. I see how to control check-ins from users, but is it possible to configure the release branch to only accept merges from the QA branch?
You cannot fully restrict a branch to allow merges from specific branches only.
However, you could prevent the check-ins through a custom Check-In Policie. You also need to combine it with a custom path policy for configuring it to a speficic folder. Detail ways please refer: How to apply check-in policies to only certain branches or folders. As for how to create a custom check-in policy by API, you could take a look at this tutorial: Creating TFS Custom Check-in Policy
For a more easier way, you might consider limiting the security on those release branches to the person(s) who would be responsible enough to do it right. It's more like an internal process constraint:"David, please avoid merging changes other than QA branches, thanks!"