I'm trying to rebuild some Matlab code in C that uses their fsolve function. From the documentation it's using a "trust region reflective" algorithm (I already built it using a Levenberg-marquardt algorithm and it's converging completely differently). Can anyone recommend a library for doing this type of optimization in C/C++ ?
There are two types of general-purpose algorithms for which there is a global convergence guarantee (under standard assumptions, don't ask :) ). These methods are the line search and the trust region methods. If you wish, you can read more on this topic in the book of Nocedal-Wright: Numerical Optimization.
I haven't tried Knitro recently.
Ipopt is the most robust solver among those I have tried, I highly recommend it. It implements the line search method and is written in C++.