The output of my program will be some 2D drawings, with primitives like drawCircle, fillRect, and so on. Originally, I planned to draw it with Matlab or Java, but then I would have to provide options for zooming, printing, editing and so on by myself.
So I think it would be a good idea to just output the graphics into a standard vector graphics format and then open it with an existing viewer.
Which format could be a good choice? I will need 2D drawings, maybe I'd like to extend to 3D later on.
Two easy options for 2D vector graphics are PostScript and SVG.
PostScript is probably easier to output and you can have an external header that implements your primitives on top of PostScript primitives, which will give you flexibility. But you'll need to use a PostScript viewer or to convert the output to PDF (some PDF viewers can display PostScript).
SVG is a little more complicated to output because of its syntax, but still not hard. The main advantage of SVG is that all major browsers can display SVG. Another advantage of SVG is that you can have transparency, which PostScript does not support. On the other hand, you'll have to map your primitives to SVG primitives; there is no programmability in SVG.
PDF does support transparency but is not easy to generate by hand; you'll have to use a library.