When configuring OpenGL with glTexParamteri(GL_Texture_2D, GL_TEXTURE_MAG_FILTER, ...) and glTexParamteri(GL_Texture_2D, GL_TEXTURE_MIN_FILTER, ...) how does OpenGL decide which filter to use when accessing a texture in shader with texture(...)?
My only guess it's that it is calculating the pixel footprint but since you could access the texture in either the fragment or vertex shader it can't know on which primitive what texture is projected.
My only guess it's that it is calculating the pixel footprint
Yes, that's what it does. It will approximate the pixel footprint in the texture space by calcualting the derivatives of the texcoords with respect to the window space x and y direction, and it will approximate these derivatives by finite differencing in a 2x2 pixel quad, just like the dFdx and dFdy GLSL functions are working. It will use the longer of the two partial derivative vectors as the size, and calculate the Level-Of-Detail value based on that.
but since you could access the texture in either the fragment or vertex shader it can't know on which primitive what texture is projected.
Correct, that's why the GLSL specification, (Version 4.60) states the following in the beginning of section 8.9 Texture Functions:
Texture lookup functions are available in all shading stages. However, automatic level of detail is computed only for fragment shaders. Other shaders operate as though the base level of detail were computed as zero