I would like to do this in Scala:
I have a messaging system, where Kafka is the entry point. The data arrives in binary Thrift encoded to the Kafka. From there I collect it with a Spark Job. In there I would like to deserialize the messages into JSON Strings. Now as far as I understood, I could per Thrift generate Scala classes, make Scala objects out of my messages and then convert them to JSON. Since I really don't care about working with the classes, is there a way to directly transform the binary encoded data into JSON by supplying the Thrift files? Or maybe create the classes at runtime? The point is that I would like to keep the system dynamic for new message types and thus would like to avoid relying on forehand manually created Scala classes.
I hope this kind of makes sense, if you have any further questions go ahead :)
Thrift isn't really intended for dynamic usage like this. Consider e.g. Avro instead.
If you have to do this with Thrift, you can "just" run the Thrift Java or Scala compiler as an external process, compile the generated files (and your own generated glue code if any) (see How do I programmatically compile and instantiate a Java class? or Scala - How to compile code from an external file at runtime? respectively, or just Google for more links), load the classes with an URLClassLoader and access them using reflection.