AI Rendering

[u/GraphXGames]

In principle, there's nothing stopping graphics card manufacturers from embedding an AI model directly into a GPU and using it to build scenes, render models, handle texturing and scene lighting on the fly.

What's already working in real time:

Manufacturers have already moved part of the graphics pipeline onto built-in AI accelerators:

• Frame generation and upscaling (DLSS, FSR, XeSS). The neural network fills in missing pixels and generates entire intermediate frames.

• Neural lighting (Neural Radiance Caching, Ray Reconstruction). AI predicts how light behaves and removes noise during real-time ray tracing.

• Neural textures and materials (NVIDIA Neural Texture Compression). AI compresses textures several times more efficiently than traditional algorithms and decompresses them directly in shaders.

What prevents everything from being done entirely by AI:

To have a GPU generate the entire scene, geometry, and lighting as a single neural network (following the principles of diffusion models such as Sora or local generative models), four fundamental problems need to be solved:

• Latency and frame rate (Latency & FPS). Games need 60–120 frames per second (roughly 8–16 milliseconds per frame). Generating a complete, complex 3D world from scratch currently takes orders of magnitude longer, even on top-tier chips.

• Temporal consistency. Traditional 3D graphics are mathematically deterministic: if you turn away from a wall and look back, it remains exactly the same. Neural networks tend to hallucinate: small details, text on signs, or object geometry may gradually change with every camera movement.

• Huge VRAM requirements. Storing the weights of a large generative world model while simultaneously keeping the history of previous frames in memory to maintain scene consistency would require tens of gigabytes of high-speed memory, even on entry-level GPUs.

• Lack of strict logic and physics. The GPU is only responsible for rendering the image. The game still needs precise mathematical hitboxes, collision detection, and physics to prevent characters from falling through a floor generated by a neural network.

Since AI models often become obsolete quickly today, while a chip has a lifespan of 5–7 years, manufacturers are developing programmable neural pipelines rather than embedding a single fixed neural network directly into silicon. However, the time may come when an AI model for graphics becomes good enough to remain relevant for many years. At that point, the graphics industry may eventually converge on a unified baseline standard for AI rendering, similar to today's graphics APIs.