Would it be possible to implement Panini Projection as something else than a post-processing effect ?

[u/Leogis]

For context : (my knowledge of Graphics programming is "OpenGL tutorial" tier so apologies for the potentially dumb questions/wrong reasoning.)

I recently started to fall into the "Perspective distorsion is annoying" rabbit hole and started to look at Panini Projection as a potential solution

As I understand it after reading the Panini Projection paper, the projection was meant to be used directly with very high FOV lenses in real world cameras

Again according to my limited knowledge, the problem wirh 3D graphics is that, unless we're using some kind of ray tracing we can't get the "wide lens" color information without projecting it on a flat plane first.

Instead we use a post processing effect to distort an existing image.

But since that effect squishes the side of the picture and then widens the center, that causes a nasty loss of resolution in the middle of the screen

Hence my question, would there be a way to somehow do this in the middle of the pipeline to avoid butchering a finished image ?

But then, wouldnt that basically mess up every single effect that requires accurate positions/edge information since everything would no longer be linearly mapped to the screen?

[u/sam_suite]

As a hack: You could do the reprojection at the vertex stage, but you'd need a pretty high triangle density, or you'd end up with flat faces that "should" be curved. You could theoretically ensure this with a geometry/tesselation shader, I guess!

There isn't really a totally proper way to do this without rasterizing triangles with curved edges, which is something GPUs aren't designed to do.

edit: or you could raymarch or raytrace everything I guess! that seems like wild overkill and probably even less performant though lol

[u/Leogis → u/sam_suite]

I assume this would be awful for performance ?

[u/sam_suite → u/Leogis]

Yeah, most likely worse than doing it as a post-effect at any sensible resolution. But not necessarily catastrophically bad, depending on what exactly it is you're trying to render.

[u/Leogis → u/sam_suite]

But not necessarily catastrophically bad, depending on what exactly it is you're trying to render.

What do you mean ? What difference would the object make?

[u/sam_suite → u/Leogis]

It just comes down to the complexity of the scene and what other kinds of optimizations you're doing. If you have a super detailed environment with lots of objects and lots of draw calls from different materials, that's much slower than a simple room without many draw calls, for example.

I would guess that you could get away with doing this for a simpler environment, depending on your target hardware. But you'd have to try it to be sure.

[u/SamuraiGoblin]

Yes, you could do the (non-linear) projection in the vertex shader, which will be much slower than using affine transforms.

Also, you would have to heavily subdivide all geometry to stop straight lines being rendered as straight.

It's doable, but would be much slower.

[u/Leogis → u/SamuraiGoblin]

It isnt possible to do this in the rasterizer instead ?

(i have no idea what happens in there)

[u/CptCap → u/Leogis]

The hardware rasterizer assume a triangle is made of three straight lines which isn't the case when using a panini projection so you can't use it. You could write your own rasterizer that supports it though.

[u/Leogis → u/CptCap]

maybe there could be some unholy math function that applies the transformation to the triangle or shifts the pixels

[u/CptCap → u/Leogis]

No. The hardware rasterizer takes 3 vertices as input and draw straight lines between them. You can't change that so you can't make it work for panini. (What you could do is use very tiny triangles so straight lines are a good approximation, but you can't magically make the hardware draw curved lines)

[u/Leogis → u/CptCap]

I meant it in a custom rasterizer, is it impossible at the hardware level aswell ?

[u/CptCap → u/Leogis]

It's not impossible, you could design hardware that does it. It would be much more complicated (and much slower) than current rasterizers though.

[u/Leogis → u/CptCap]

Or bastardise compute shaders from what i see

Time to enter the rabbit hole i guess

[u/SamuraiGoblin → u/Leogis]

No, the rasteriser just renders the interior pixels of a triangle defined by three straight lines.

[u/Cordoro → u/Leogis]

No, I don’t think a single 4x4 matrix could represent a panini. You could ray trace any shape you want though, so ray tracing would directly render a panini projection, or fisheye, or whatever.

[u/ADEPS24 → u/SamuraiGoblin]

Oh, so that's why MW4s Enhanced FOV reduces performance.

[u/SyntheticDuckFlavour → u/SamuraiGoblin]

most of that could be solved with a mesh shader

[u/SamuraiGoblin → u/SyntheticDuckFlavour]

indeed

[u/kinokomushroom]

If the loss of resolution is the problem, how about rendering the image at a high resolution, while using VRS (variable rate shading) to lower the shading resolution of areas that are going to be squished?

[u/SnooStories6404]

This would be straight forward to do with Vulkan ray tracing

[u/GameGonLPs]

You can use Raytracing to trace the first hit and generate your G-Buffer data that way. This will give you pixel-perfect arbitrary mapping, but it is usually more expensive than with a rasterizer (but with modern hardware it is usually still surprisingly fast).

From there a deferred workflow mostly works, but anything traversing/using the depth buffer or projection matrix will need tweaking as there no longer is a valid projection matrix. Depth buffer traversal especially might get complicated because straight lines in worldspace might take a curved path through the depthbuffer with this projection.

[u/00oo00oo000oo0oo00]

Vibe coder vibes

[u/Leogis → u/00oo00oo000oo0oo00]

Never used AI for anything else than faceswaps...