Why does polygon smoothing produce a broken line?

When using polygon smoothing in OpenGL, I get some broken lines inside my polygons as can be seen in the images:

enter image description here

My code looks like this:

glEnable(GL_POLYGON_SMOOTH)
glHint(GL_POLYGON_SMOOTH_HINT, GL_NICEST)

glPolygonMode(GL_FRONT, GL_FILL)
glPolygonMode(GL_BACK, GL_LINE)
glLineWidth(5)
glBegin(GL_QUADS)
glVertex3f(0.1, 0.1, 0)
glVertex3f(0.2, 0.1, 0)
glVertex3f(0.2, 0.2, 0)
glVertex3f(0.1, 0.2, 0)

glEnd()
glDisable(GL_POLYGON_SMOOTH)

Why does this happen and how can I solve this?

Solution

Drawing quads or polygons is usually emulated on modern hardware by drawing several triangles. When smoothing borders of these triangles, the diagonal which splits the quad into the two triangles will get smoothed too, producing this artifacts.

The OpenGL API you are using (fixed function pipeline) is deprecated for a decade now and should not be used anymore. Especially "special features" like polygon smoothing or drawing polygons might behave very differently on different hardware.

Modern OpenGL (3.3+ Core Profile) works completely different and comes with a set of better methods to deal with antialiasing like MSAA or post processing (FXAA, ...)