Picking objects in 3D space is a common problem. If you want your 3D simulation to interact with a mouse, we need to solve this problem. To implement picking, we need to find the pixel that the user has clicked relative to both the near and far planes of the camera. We can construct a ray from the point on the near plane to the point on the far plane. Finally, we can query the world using this ray.
The job of a graphics pipeline is to take a 3D point in world space and project it onto the screen. This transformation from world space to screen space is called Projection. To find the 3D world space position of a point based on the 2D pixel position of that same point we need to do the opposite of what the graphics pipeline does. Putting a pixel through the inverse of the graphics pipeline is called Unprojection.
When we unproject a pixel, it has no Z coordinate. We will provide a Z component that is a linear depth value. That is, a Z value of 0 will result in the pixel on the near plane...
