Encoding with CQ besides CRF

[Pete4K]

Is your feature request related to a problem? Please describe.
The encoding is only possible by using CRF.

Describe the solution you'd like
When you could do an encoding with CQ you could maybe encode it to H265 with 10Bit in lossless quality and smaller filesize immediately without the step to use Handbrake. Anime looks much better in the quark with this encoding.

Describe alternatives you've considered
When this don't work, then maybe it is possible to do something against the color stripes in the dark or you could use some code from Handbrake. I think that is Open-Source, too.

Additional context
You could see the results by using Handbrake with a 4K file from Video2X. You can choose Video Encoder to "H.265 10-bit (NVEnc)" use the CQ "30" and then setup to slowest and start.

[k4yt3x]

I'm not that familiar with Handbrake since I've mostly been using FFmpeg directly from command line, but if I'm not mistaken when it comes to H.264 and H.265, CQ in Handbrake seems to just be CRF?

[k4yt3x]

It does say RF (rate factor) here:

[Pete4K]

You have to choose an Option with (Nvenc) after H264 or H265. I think 12-Bit doesn't have (Nvenc).

[Pete4K]

I think Handbrake is based on ffmpeg

[Pete4K]

Maybe it is possible to use this as ffmpeg command?

[k4yt3x]

I don't have nvenc since I'm using it in a Windows VM. My workstation is Arch. Could you please do me a favor and screenshot the exact setting that you're referring to so I don't have to guess?

[Pete4K]

Ok, I'll send a Screenshot later

[Pete4K]

That is my setting to encode.

[Pete4K]

Maybe you could built-in this in ffmpeg for 10-Bit and 12-Bit:
https://askubuntu.com/questions/922563/set-bit-depth-in-ffmpeg-encoding-for-hevc

[k4yt3x]

Thanks for providing the screenshot. I think you could set the custom options to something like this to achieve that:

I need to validate it though.

[Pete4K]

Ok, thank you very much. I will try this. But I don't know if this works with 10-Bit then.

[Pete4K]

Or could I set the value for pix_fmt = yuv420p10le for 10 Bit and then the other values? Maybe that works. I will try this, too.

[k4yt3x]

I think for hardware H.265 the pixel format might be p010le? Give that a try

[Pete4K]

Ok, thanks. I will try this.

[Pete4K]

p010le results in a failed process immediately. yuv420p10le works.

[k4yt3x]

Yeah you're right. I also tested it and got this:

I was reading this post but it was too long so I didn't go through all of it. I think the pix fmt it's referring to might be for the hardware, not for the final video output.

[Pete4K]

Ok, seems to work at the moment. I try and test the difference between a H264 Video with 14 GB to the small output I get now. I see that CRF seems to work better, so I test crf = 20 and preset medium with yuv420p10le as pix fmt. When I get the same quality like with Handbrake I can save one step and still have all subtitles and audios, so that I don't have to separate the subtitles from the file.

[k4yt3x]

Great. With that said, I don't think I need to implement any new features in v2x? What you want to achieve can all be achieved through settings it seems.

[Pete4K]

Yes, I see now if that gives the same result. I say later if that works. I have to look especially at the difference in the dark, if there are some spots. With 10-Bit the spots should be very small, almost unrecognizable.

[k4yt3x]

Gotcha. Please let me know your test results. If it proves to work as expected, you can also close the issue.

[Pete4K]

Ok, I tested it with the parameter 23 for crf, yuv420p10le as pix fmt and preset set as medium, I get around 420 MB for a 4K file without those spots, so that seems that 10-Bit really works. With making the 4K file with libx264 and crf 10, preset medium I get a 13,5 GB file. When I convert this again with Handbrake, I am at 408 MB for this file. So the difference is only 12 MB, what means that the one-in-all-conversion with Video2X is almost the same size and the look is almost without losing quality. You can only recognize some spikes in the Netflix Logo for example. But that is the same with Handbrake. The only thing where Handbrake is better is with BluRay-Content, because there is a difference of 500-600 MB, but I think the post can be closed and I close it.

[Pete4K]

Oh and CQ is not necessary. That was the main idea of this issue.

[k4yt3x]

FYI if you process the file with Real-ESRGAN/Real-CUGAN or RIFE, they get converted into RGB24 because that's what the models accept and spits out. For those cases, I'm not sure if there's still a point storing the output with 10-bit.