Stencil Buffers in a few pointers

This assumes you know the graphics pipeline, how the depth buffer works as the pixel is being considered for drawing.

1. The Stencil Buffer is an integer buffer.
2. If attached to a framebuffer used for rendering and stencil test is enabled, the stencil test will happen before the depth test. It has to pass the stencil test before the pixel goes to the depth test, otherwise it is discarded.
3. You can set the conditions for the stencil test, essentially a mathematical function with the destination stencil value and additional reference values as input.
4. You can set the results for what happens to the current values in the stencil buffer based on the result of the depth or stencil test.

Graphics API context

• glClear & glClearStencil/ClearDepthStencilView allows the depth stencil to reset the buffer to it’s stencil value parameter.
• For (1), you may have seen some depth texture having a descriptive format similar to D24S8 (ie. 24 bits for depth 8 bits for stencil).
• For (3), look at glStencilFunc, or D3D11_DEPTH_STENCILOP_DESC’s StencilFunc.
• For (4), look at glStencilOp, or D3D11_DEPTH_STENCILOP_DESC’s StencilFailOp/DepthFailOp/StencilPassOp.