How can I create new custom controls in Orbeon Form Builder and add it to the list of controls on the left side? Is it neccessary to download form builder's source code and interfere with it? Or can I just create some xbl/xml files and upload it to form builder?
You don't need to modify the Orbeon Forms source code. Instead you create XBL controls (see the tutorial). The toolbox can then be configured via properties to show the new controls created.