Does anyone know of such a library that performs mathematical optimization (linear programming, convex optimization, or more general types of problems)? I'm looking for something like MATLAB, but with the ability to handle larger problems. Do I have to write my own implementations, or buy one of those commercial products (CPLEX and the like)?
A good answer is dependent on what you mean by "convex" and "more general" If you are trying to solve large or challenging linear or convex-quadratic optimization problems (especially with a discrete component to them), then it's hard to beat the main commercial solvers, gurobi, cplex and Dash unless money is a big issue for you. They all have clean JNI interfaces and are available on most major platforms.
The coin-or project has several optimizers and have a project for JNI interface. It is totally free (EPL license), but will take more work to set-up and probably not give you the same performance.