I want to customize the genetic optimization of MT5. For example, I want experiment with different population sizes and the selection methods. Am I able to tune the default genetic optimization or I should rewrite it as a Script or Expert Advisor. Is there an API (or template) for writing custom optimizers and is it possible in MT5?
It is not possible to alter the optimization engine in MT5 as far as I am aware. If you are not totally restricted to MT5, I can recommend joining Asirikuy, who collaboratively develop their own trading software, which would allow you to customize the engine. There may be other groups or open source software projects who do not require fees, but Asirikuy is the most advanced I know of.