What the equivalent of OpenGL glStencilFunc in Metal?

Question

How to translate to Metal the following OpenGL instruction:

glStencilFunc(Func, Ref, Mask);

Answer 1

Let's try to extend Hamid's correct answer with some actual Swift code...

Assuming you already have the basic Metal setup done:

let metalView: MTKView = /* ... */
let commandBuffer: MTLCommandBuffer = /* ... */

let device: MTLDevice = metalView.device!

Let's create a MTLTexture to store the stencil buffer:

let stencilTexture: MTLTexture

// Set up a texture for rendering the *8 bit unsigned* stencil buffer.
let stencilTextureDescriptor = MTLTextureDescriptor()
stencilTextureDescriptor.textureType = .type2D
stencilTextureDescriptor.width = Int(metalView.frame.width)
stencilTextureDescriptor.height = Int(metalView.frame.height)
stencilTextureDescriptor.pixelFormat = .stencil8
stencilTextureDescriptor.storageMode = .private

// Set this option if you use the given texture 
// as a stencil render target in any render pass.
stencilTextureDescriptor.usage = [.renderTarget]

stencilTexture = device.makeTexture(descriptor: stencilTextureDescriptor)

Create the MTLRenderPipelineState as usual but be sure to set the stencil attachment too:

let rendePipelineState: MTLRenderPipelineState

let renderPipelineDescriptor = MTLRenderPipelineDescriptor()
/* ...standard render pipeline configuration (e.g., render states, etc)... */
renderPipelineDescriptor.stencilAttachmentPixelFormat = .stencil8 
rendePipelineState = try! device.makeRenderPipelineState(descriptor: renderPipelineDescriptor)

Next configure the MTLRenderPassDescriptor and create a MTLRenderCommandEncoder:

let renderEncoder: MTLRenderCommandEncoder 

let renderPassDescriptor = MTLRenderPassDescriptor()

renderPassDescriptor.colorAttachments[0].loadAction = .clear
renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1)
renderPassDescriptor.colorAttachments[0].storeAction = .store

// Configure these depending on your rendering pass stencil semantics.
renderPassDescriptor.stencilAttachment.texture = stencilTexture
renderPassDescriptor.stencilAttachment.clearStencil = 0
renderPassDescriptor.stencilAttachment.loadAction = .clear
renderPassDescriptor.stencilAttachment.storeAction = .store

renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
renderEncoder.setRenderPipelineState(rendePipelineState)

Now create a MTLDepthStencilState with the desired stencil buffer configuration:

let depthAndStencilState: MTLDepthStencilState

let stencilDescriptor = MTLStencilDescriptor()
stencilDescriptor.depthStencilPassOperation = .incrementClamp

// ## glStencilFunc(FUNC, _, _) ##
stencilDescriptor.stencilCompareFunction = .equal 

// ## glStencilFunc(_, _, MASK) ##
stencilDescriptor.readMask = 0xFF_FF_FF_FF 

let depthAndStencilDescriptor = MTLDepthStencilDescriptor()
depthAndStencilDescriptor.depthCompareFunction = .always
depthAndStencilDescriptor.isDepthWriteEnabled = false
depthAndStencilDescriptor.frontFaceStencil = stencilDescriptor
depthAndStencilDescriptor.backFaceStencil = stencilDescriptor

depthAndStencilState = device.makeDepthStencilState(descriptor: depthAndStencilDescriptor)!
renderEncoder.setDepthStencilState(depthAndStencilState)

// ## glStencilFunc(_, REF, _) ##
renderEncoder.setStencilReferenceValue(1) 

Finally, do the rendering as usual:

/* ...encode rendering primitives... */
renderEncoder.endEncoding()