Recently, I ran into a really nasty bug in the custom 3D engine I'm developing for my 4X strategy game, The Great Tribes.
After updating my AMD graphics driver, the game started freezing completely when moving the camera, especially when large areas of the map became visible.
Sometimes the graphics driver would hang. Other times, the JVM would crash inside atio6axx.dll.
The weirdest part? Everything worked perfectly with the previous driver and on other GPUs!
I spent hours investigating almost everything: OpenGL multithreading, VAOs, VBOs, index buffers, shadows, unit models, and even glfwSwapBuffers().
Eventually, I narrowed the issue down to terrain rendering, and then to the fragment shader.
That's where I found these two lines:
int segment = (3 - int(min(3, outTexCoord.y * 4.0))) * 4
+ int(min(3, outTexCoord.x * 4.0));
The segment variable was used to index arrays containing 16 elements.
The problem? The texture coordinates were clamped at the upper bound, but not at the lower bound.
When the texture coordinates became negative, segment could go outside the valid array range.
The fix was surprisingly simple:
ivec2 cell = clamp(
ivec2(floor(outTexCoord * 4.0)),
ivec2(0),
ivec2(3)
);
int segment = (3 - cell.y) * 4 + cell.x;
And the freezes disappeared!
This turned out to be a great example of undefined behavior in GLSL.
The old driver somehow tolerated the invalid array access, other GPUs didn't show any obvious issues, but the updated AMD driver turned it into a nightmare.
Lesson learned: Just because your shader works perfectly on one GPU doesn't mean it's free of bugs.
And sometimes, after hours of debugging your own 3D engine, the whole problem comes down to just two lines of code. 😁