I have an objective function which calculate the difference between actual values and my prediction of non-linear ODE system according to some Parameters
residuals_vector = actual_values-predictions
[residuals_vector]=objfun(actual_values,time,parameters)
with P parameters N residual,
is there an easy way to calculate the sensitvity matrix P x N of this system in Matlab
There are several approaches you can try here. If your objective function is differentiable, and what you are looking for is local sensitivity analysis, then what you want is to compute the gradient. If your objective function is nice enough and you don't want to compute the gradient yourself, you can use automatic differentiation packages such as https://www.mathworks.com/matlabcentral/fileexchange/61849-autodiff_r2016b. For global sensitivity analysis, you will want to do a monte carlo simulation, which will be more computationally intensive. See :https://www.mathworks.com/help/sldo/sensitivity-analysis.html for tools to set up your sample if this is what you want.