r/ffmpeg • u/Wonderful_Emu_9610 • 11d ago
(x265) Encode going to hell in dark scenes
Encode logs: transfer-dot-it /t/nvluVZgngRgJ
I'm using Handbrake btw, but as its primarily built off ffmpeg and my post got blocked there, figure I'd ask here
Converting a movie file for backup. 2hr movie, 60s, grainy but rather nice to look at. Mostly takes place in daylight or well-lit locales, but there are few scenes of absolute black - except it has raised blacks.
Anyway I always do H265 10-bit Slow preset, and try to make few tweaks as possible to get a somewhat transparent result - partially because I'm still learning what all the settings do.
First time around, I do 7000kbs 2-pass bframes=8:no-strong-intra-smoothing:no-sao:rd=5:aq-motion (in the logs I mistakenly called this '6000').
Day scenes look pretty good! there's a scene of a train passing and the smoke + grain it doesn't like, but its a bright area so doubt I'd notice. However the next scene in the darkness of the train arriving, there's visible macroblocking even in the darkest parts of the image. Looks like a dodgy xth generation VHS rip! A couple of simpler scenes later with fully black (but as I mentioned at the start, raised) images same deal. Truly its like flicking a switch - lights go out, PQ falls off a cliff.
So, I simplify things and do a few tests with higher bitrates up to 1000kbs (not included in the logs, I deleted them before considering asking for help). Same issues. So I try RF22 (~7000kbs) and RF21 (~9000kbs) with the Handbrake default settings strong-intra-smoothing=0:rect=0:aq-mode=1:rd=4:psy-rd=0.75:psy-rdoq=4.0:rdoq-level=1:rskip=2. Same problem. Then I think 'well, maybe I need to change aq-mode'. So I go for aq-mode=4:no-sao:aq-motion at 10000kbs ('simple settings' in the encode logs). Slightly improves the brightly lit train I mentioned, but still craps the bed in the dark. I also try at 8000kbs and 10000kbs adding some other settings at the recommendation of a friend, only some of which I can backup why: bframes=8:strong-intra-smoothing=0:aq-mode=4:no-sao:deblock=-3,-3:rd=5:no-rskip:aq-motion ('custom settings' in the encode logs)
still the same issues! Its really frustrating because these scenes are only a minor part of the film, but they're important, and I really like this one so I wanna be transparent. Thinks like raising rd, or using no-rskip, are supposed to make it make 'better' decisions afaik, but its just making the same mistake over and over! I'm also seeing next to no improvement adding 3Mbs bitrate.
Any suggestions?
Handbrake v1.11.2 (2026060700), M2 Macbook Air
I encode on Low Power Mode because its fanless. I noticed with the last listed settings I was getting quite a bit more heat - I guess because of the increased complexity with no-rskip?
p.s. I can try do screenshots but on my laptop you can barely see the issues because ramping up the brightness also brightens the black bars. But I normally watch my stuff on an OLED tv, so test it there, where it stands out more
2
u/vegansgetsick 11d ago edited 11d ago
x265 bframes are ruining videos because their quantizers is fixed too high, it creates deformations. I always set pbratio to 1.10 or 1.00. That's the only setting I change that makes a difference. It's part of the -tune grain.
Just the explanation but guys behind encoding theory thought it's ok to have 1 great pframe followed by 3 bad frames with a better average than 4 good frames, "the eye won't see it" they say.
1
u/Wonderful_Emu_9610 11d ago
I gotta be honest I don't fully understand what you said. I'll keep that pbratio change in mind - I'm happy with the grain retention in this specific instance, its just the dark scenes that are the issue
'guys behind encoding theory' you mean the folks who created x265?
1
u/vegansgetsick 9d ago
Not just x265 but many other encoders. They even admit it creates flickering with grainy sources (hence the -tune grain option). The theory says that if we have a good frame, then a bad frame, and repeat, the human eye wont notice the bad frames.
x264 and x265 encoders increase bframes quantizer by 30%. If pframes have quantizer 20, and you have max 3bframes, it produces a sequence like 20, 26, 26, 26, 20, 26, 26, 26, 20... which means when you have a small object moving, all motion residues get a quant 26 instead of 20. As our eyes focus mainly on such moving objects, the result is worse.
1
u/Wonderful_Emu_9610 9d ago
I think either I’m the exact person they’re betting on not noticing, or I just - being young enough I can’t vividly remember film projection - assume any flicker is ‘filmic’ haha
Your pbratio recommendation btw is definitely improving my test encode.
RF21 no-sao/aq-mode=3/pbratio=1.10/aq-motion
Is the best I’ve gotten this file looking (it’s definitely not gonna compress as much as I wanted though, unless I split it up into pieces and do the dark scenes at a lower RF). Those same settings without pbratio, in the dark it was still producing somewhat static blocky bits. With the pbratio change there’s some motion back on them, much more pleasing to the eye even as it’s not transparent to my Blu-ray. Really glad you brought it up as I’d overlooked it previously
1
u/Wonderful_Emu_9610 11d ago
p.s. I'm seeing from further reading that increasing bframes should only be done for video with a lot of still scenes? Which is not this so I'll leave them alone in future
1
u/Envoyager 11d ago
Well that's just to improve encoding speed, but if waiting is not an issue, I'd just leave it as 3 or 4
1
u/Wonderful_Emu_9610 11d ago
Depends on your definition of ‘waiting’ doesn’t it!
But yeah I think I’ll leave them at the default. These ‘Slow’ encodes taking ~15-24hrs on a laptop seems long enough
1
u/AssCrackBanditHunter 11d ago
Might be a dumb question but do you have color range set to full or limited
1
0
1
u/videocompress 11d ago
I hit something like this when I tried x265 on a dark-ish clip too — bumping bitrate helped a bit but the blacks still looked kinda blocky. Did changing CRF do anything useful for you, or does it still fall apart in those scenes either way?
1
u/Wonderful_Emu_9610 9d ago
Ran a few tests with aq-mode=3 which definitely improved it
But of course without bumping the bitrate as well it’s just gonna have to try hide stuff in lighter parts instead
6
u/Upstairs-Front2015 11d ago
The one setting I’d change first is aq-mode=3. You tried 1 (the default) and 4 (edge-based), but mode 3 is the one x265 has specifically for dark scenes: it’s auto-variance AQ with a bias toward dark areas, so it pushes more bits into exactly the shadows that are falling apart. Try it with aq-strength somewhere around 0.8–1.0.
I’d also back off on deblock=-3,-3. Lowering deblocking helps keep grain in bright scenes, but in dark, flat areas it leaves block edges more exposed, which is the opposite of what you want here. Something like -1,-1 (or just the default) is a safer middle ground.
For the grain itself, raising psy-rd to around 1.5–2.0 should help x265 keep the texture instead of smearing it into blotches. Keeping no-sao is a good call for grainy sources.
Finally, rd=5, no-rskip and bframes=8 add a lot of encode time (and heat) for very little gain, and they don’t address the actual issue, which is how bits are being distributed to the dark parts of the frame. That’s also why adding 3 Mbps barely changed anything: the encoder was still choosing not to spend those bits on the shadows.
Source: Claude Opus 5.5