Tuesday, September 19, 2017

How to Process Mouse Events in a 3D Scene

Generally, to tell what object a user clicked on in a 3D scene, you have to cast a ray from the camera through the mouse cursor, and find the nearest object that it hits.  Incidentally, your GPU already does this each frame as it renders the scene.  This can be leveraged so that you don't need to actually cast a ray in your program, instead using the information that your GPU computes anyway each frame.

As your graphics card writes to the color buffer each frame, it manages a depth buffer tracking the distance of each fragment from the viewer.  Generally, it only updates the color buffer if it also updates the depth buffer.

Your GPU can easily be programmed to write to another buffer at the same time as it paints the scene - another "color buffer" in OpenGL terminology - but one that does not store a color, but rather an integer giving you the ID of the object beneath that pixel.

This way, when the user clicks or moves the mouse, you can find out what object they interacted with just by accessing this buffer.  There is no need to loop through all of the objects in the scene or do any messy ray casting operations at all.

Here are the basic steps:

1. Create a new color buffer with a 32-bit unsigned int as its internal format.  This is a suitable data type to hold the unique IDs of objects in a 3D scene.

2. Add a uniform to the vertex shader representing the object ID, and pass this value unchanged to the fragment shader.  From there, write this value to our new color buffer.

3. After drawing each frame, copy this new color buffer into a local array in your program.  When the user clicks or moves the mouse, you can tell what object was clicked on just by indexing into this array.

Part 1
Part 2
Part 3

No comments:

Post a Comment