I am using instanced rendering to draw a large amount of cubes. Now, obviously the maximum amount of visible faces in a cube is 3, which means that in terms of shading, I am doing twice as more works as needed.
Because this is instancing I cannot simply pass only the geometry to be drawn to the shaders. But if in the shaders (I'd imagine this would be in either geometry or tesselation) I can use the normal information to discard a primitive, I can reduce the number of unnecessary operations.
For example if the angle in between the looking direction and the normal to the surface is less than 90 degrees I can assume there is a triangle in front of the current one and thus discard this given primitive.
Is there a way in the pipeline where I can determine if a fragment is needed and discard it accordingly?
Before answering the question, let me first solve your actual problem: use back-face culling. Give the triangles of each cube face a consistent ordering, such that when seen from the front, the ordering of the faces in screen space will be one way, but when seen from the back the ordering will be the other. Which you pick is a convention that's up to you, but pick one and be consistent.
Then use glFrontFace to tell OpenGL which convention is "front" for you, and use glEnable(GL_CULL_FACE) and glCullFace(GL_BACK) to tell OpenGL to cull back-facing triangles.
This culling happens as a part of primitive assembly, so it's long past vertex processing and post-processing.
This is all you really need to do. Oh sure, you're still spending up to half of vertex processing on triangles that won't be seen, but if you want to cull them, you'd still have to spend time figuring out if you want to cull them. And remember: GPUs execute shader invocations in lock-step. Culling triangles/faces/whatever due to conditional operations doesn't always make anything faster, since those invocations will still continue to be executed. Their results will just be thrown away.
Face culling is far more efficient than anything you could code.
Now, let's answer the actual question: what are your options for shader-based culling?
You cannot discard a vertex from a vertex shader. Primitives are based on a specific count of vertices (GL_TRIANGLES means that every group of 3 vertices is a triangle), so how that would work if a vertex is killed is non-functional.
You can use user-defined culling output variables to cull triangles from just the vertex shader. This is analogous to user-defined clipping planes, where your vertex shader provides a special numeric output. For clipping, if the interpolated output is positive, then that part of the triangle is visible; when the interpolated value becomes negative, that part of the triangle becomes non-visible through clipping.
With user-defined culling planes, if any of the vertices for a particular primitive has a negative value for culling, then the entire primitive is culled. This allows one shader invocation to choose to cull whatever primitive it is in by giving a negative culling value.
Tessellation Control Shaders have the ability to discard patches. If the TCS for a patch results in a zero value for any of the outer tessellation levels, then the patch they're working on will be culled by the tessellation primitive generator.
Geometry shaders can of course output no primitives at all.