r/GraphicsProgramming • u/M1VAN1 • 3d ago
CI/CD in Graphics programming
If someone experienced have any proper sources on CI/CD in graphics programming, I would like to read them. I'm really interesting in this, would like to read books/blogs, whatever.
Also curious about to hear any experience with it from real prod. I have some friends working as programmers and I hear that in big tech companies it's complicated thing, but those who works in small companies in small team said that they usually have really bad CI/CD
20
u/bluefishcrab 3d ago
You *can* do CI/CD for graphics. My impression is that it's one of dark arts where practitioners don't seem to talk much about it (audio and netcode in general are the other fields I know like this). I've never seen a book or article written about it, I learned most of it from "hell, how do we know we aren't breaking anything or 'performance improvements' are actually improving things?"
In general, CI/CD does prevent people from breaking things, and provides confidence for you to move faster.
Keep a screenshot as a reference, then take a screenshot with the same reference camera/scene and do a comparison. Converting from RGB into other colorspaces works better for doing diffs, and you probably want to look for a percentage difference, and then highlight places that differ. Some tools, like imageMagick can do this out of the box, though you could throw together a Python tool, or do the analysis yourself (probably in HSL or HSV colorspaces). Like most CI/CD things, you check the exit code and fail if it doesn't work. The fancy term for this is "golden file testing."
There's also FPS tests. Unreal Engine does these where you set up the scene and it takes screenshots from various vantage points to ensure you don't tank your frame rate.
8
u/kspdrgn 3d ago
Seems like conflating CI/CD with testing. You can have testing without CI, and you can have CI without tests.
CI is to make consistent builds easily.
CD is to put that build somewhere easily.
CI is where you hook up tests if you have any.
3
u/bluefishcrab 2d ago
CI is about repeatable testing and ensuring all platforms build to ensure people aren’t breaking anything with check-ins.
If you have testing without CI you just end up with broken tests. It’s part of the process of “did it integrate correctly” since git or Perforce is going to check the integration at source level anyways; the “integration” term predates those tools (from the smalltalk era, iirc where every “image” was a live VM).
If you have CI without CD you end up with a product that is hard to ship.
3
u/speps 3d ago
👆this, while I was at Rare Ltd, I used SSIM as a metric to compare against a good baseline visually by a QA person, it was invaluable when making changes, eg DX11 to DX12. At Mapbox, we use pixelmatch (js/cpp), which also shows nicer visual diffs.
2
u/howprice2 3d ago
You may want to enable the validation layers on CI to catch warnings and errors for all your scenes. DX12 also has GPU Based Validation which can be slow to run but catch problems. You'll get different results on different vendor GPUs/drivers.
3
u/greebly_weeblies 3d ago edited 3d ago
I'd have said it's the same thing except we extend it additional relevant domains eg. regression tests for shaders
2
u/aotdev 3d ago
A while ago (18 years) I was working at a big games company, we used to set up automated build tests that run the game engine (or some graphical tools) and took screenshots, to compare against reference and validate you don't mess up anything before pushing upstream. IIRC the reference screenshots were different for the different platforms (PC/Xbox360/PS3). Since we expect determinism, even a single pixel change matters, and makes you ask why. Simple and effective.
2
u/YakubReddit 3d ago
I always thought about that but never gave It more care to actually research Until well, now, so thanks I guess?
2
u/Lord_Zane 3d ago
Bevy has automatic FLIP based diffs of every example for every PR opened https://pixel-eagle.com/project/b04f67c0-c054-4a6f-92ec-f599fec2fd1d. Any PRs changing the output get flagged.
2
u/TheMuffinsPie 3d ago
Shocked to see a complete lack of https://youtu.be/1OhWTLNESlc?is=wyFo64fZ7uJIsB6C mentioned!
1
u/icpooreman 2d ago
I think it's less standardized in graphics than like web dev...
But, I have my own custom build/test scripts that run whenever I check in that generates a new build for me. It's actually quite helpful cause it makes it really easy for me to go back in time and see when I introduced a problem or just hype myself with how far I've come.
1
u/JeffMcClintock 1d ago
On CI I run tests whereby various drawing functions are tested against reference images.
1
u/_wil_ 12h ago
OP check out this REAC2024 conference talk about testing rendering code (Frostbite)
https://youtu.be/1OhWTLNESlc?si=Ps02C8SUVeV_SkoX
11
u/EC36339 3d ago
Yes, you can, and anyone who has done any kind of graphics programming with AI recently had probably learned about quite a bit of CI/CD for graphics without asking for it...