I am currently writing a very simple 2.5D game engine in java and game. This is basically just a fun project I am working on so the amount of time it will take to implement features is no problem, as I can always put it aside and work on it whenever. With that said, I was planning on implementing a custom sort of scripting system. Basically the engine would be a shell of a 2.5D game, which can be modified and extended through simple scripts. So far I have the following ideas of how to go about this but I am unsure of what is practical or not.
I really thought about option 1 as being what I wanted to do, however I have never really done such a thing, I was wondering if someone could recommend what I should do here. If I do go with writing a simplistic language, I am just looking for some material or documentation on such topics, would I just look at how people wrote basic compilers/interpreters?
The short answer would be 2, go with the API. It's the easiest way to add or change features to your game engine.
Creating your own scripting language and parser is tempting for every developer. It's a great learning experience, but it's also a downside. When you start to write the scripts and the result isn't what you expected, you don't know if the bug is in your script or in the parser. You will need some debugging info from the parser maybe some stacktrace, even a very simple scripting language may start to take over your project. When you start to write your scripts you will probably feel limited by the features missing, the ones you are used to from writing java code and maybe start to expand the scripting language.
Some of the downsides with creating your own language is also true for allready existing languages. If you really need a very free way to extend your game engine you can choose that road, but I would not recommend it. You would still risk shifting your focus from your game engine to the scripting implementation.
One way to think about it is also from the writer of the mods point of view. Would they want to write in a new script language, in a existing script language or in java? The answer will most likely be java and probably not a new language.