I am currently making a JVM-based programming language. Instead of having operators, I chose to allow symbols as method names and create compiler reference classes for primitive data types. These are annotated with so-called @Intrinsic annotations that take Bytecode instructions as their parameters. Instead of an INVOKE instruction, the compiler uses these to generate the Bytecode at every call.
I am now wondering whether or not this (public) annotation can be used to do anything malicious on the JVM, and if it should be restricted by the compiler, for example by static analysis.
(The language itself also supports Bytecode Expressions)
The JVM will verify the bytecode before use and the bytecode’s history is irrelevant to this process. In the end, if there is a certain invalid, possibly malicious, bytecode pattern that the verifier doesn’t recognize, its danger doesn’t depend on whether this byte code pattern was created using your compiler and annotations or created by hand.
That’s the reason for the verification concept in the first place. The JVM doesn’t assume that compilers are always error-free.
However, it’s still a good idea to let the compiler perform plausibility checks, static analysis or even perform its own verification of the created byte code. It’s not about security, as said, if a JVM’s security would rely on your compiler doing everything right it would be broken anyway. But it’s about usability, most users like to get an immediate response on errors without the need to actually run the code to learn that it’s invalid.