I wrote a Lua script with zerobrane, which can run in the editor, but it has a utf8 encoding error when it is called and executed by other languages, such as python. So, how should I set it?
Yes, the IDE provides full UTF-8 support (in the editor). If your script returns results that are non-UTF-8 encoded, you may specify the codepage you are using to convert (see the codepage setting in the documentation).
I'm not sure what "encoding error" you are referring to, so the a script sample and some description of your setup may help.