I am building a custom stat space model using statsmodels' MLEModel class, and fitting the unknown parameters with the .fit() method. I thought that it was using Expectation Maximization, but I am not sure of that and I cannot find any hint to this in the documentation. Moreover, the verbose output of the .fit() method shows the steps of a single optimization, and this makes me even more doubtful.
If it is not using EM, what is it doing? maybe I am missing something here, but I thought that for models with hidden variables (like state space models) you cannot directly minimize the likelihood (since you don't know and don't observe the hidden states).
thanks for any suggestion
By default, MLEModel uses a quasi-Newton approach to numerically find parameters that maximize the likelihood function. Ultimately, it relies on the Scipy minimize function. The default algorithm is the BFGS algorithm, but you can select others using the method argument. For example, method='nm' uses the Nelder-Mead algorithm.
The Statsmodels state space framework provides all the tools (e.g. the appropriate smoothers) to use the EM algorithm, but you would have to implement it yourself. For example, the EM algorithm is the default method used to fit the DynamicFactorMQ model, because it has a large number of parameters, and quasi-Newton methods can be slow and unreliable for such problems.