-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Libass Integration #448
base: master
Are you sure you want to change the base?
Libass Integration #448
Conversation
I just tested my C# sample from #439 (comment) , and instead of using a timer to update the UI, I switched to mediaPlayer.PlaybackSession.PositionChanged for updates. However, I noticed that PositionChanged is significantly slower compared to using a timer to repeatedly query mediaPlayer.PlaybackSession.Position. |
Thanks for sharing this. |
This is almost complete. The problem you are having, which is the same problem I was having is that ass_render_frame does not work. It returns NULL, and therefore nothing is being blended for all my test files. This happens despite the ass_read_memory and ass_process_chunk being called and the track correctly having events inside. My approach was slightly different, using an ass_track for each cue, which is safer in seek and flush situations, but this can be refactored later. |
when you enable subtitle streams, throws exceptions, I have no idea what is this: FlushCodecsAndBuffers libass: Event at 24354, +96: 499,0,Default - Copier,,0,0,0,fx,{\an5\pos(649,63)\bord2\shad0\be0\}To libass: Event at 24354, +28: 514,2,Default - Copier,,0,0,0,fx,{\galovejiro\an5\blur0\bord4.0909\pos(755,63)\fad(0,200)\t(0,100,\blur8\3c&H0000FF&\fscx125\fscy125)\t(100,180,\fscx100\fscy100\bord0\blur0)}the libass: Event at 24354, +28: 515,2,Default - Copier,,0,0,0,fx,{\galovejiro\an5\blur0\bord4.0909\pos(755,63)\fad(0,200)\t(0,100,\blur8\3c&H0000FF&\fscx100\fscy100)\t(100,180,\fscx100\fscy100\bord0\blur0)}the Exception thrown at 0x00007FFCECCEFB4C (KernelBase.dll) in MediaPlayerCPP.exe: WinRT originate error - 0xC00D36B2 : 'The request is invalid in the current state.'. Seek SeekFast - ### Backward seeking FlushCodecsAndBuffers Exception thrown at 0x00007FFCECCEFB4C in MediaPlayerCPP.exe: Microsoft C++ exception: winrt::hresult_error at memory location 0x000000EACCBFEBB8.
Happy New Year. This issue occurs when you don't call Regarding your point, I'm not entirely sure you're correct. Calling I made some adjustments, and while the changes work to some extent, the Here's what I tested: private async void OnTimedTrackCueEntered(TimedMetadataTrack sender, MediaCueEventArgs args)
{
if (args.Cue is ImageCue cue)
{
await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
{
var sub = cue.SoftwareBitmap;
var bitmapSource = new SoftwareBitmapSource();
await bitmapSource.SetBitmapAsync(sub);
image.Source = bitmapSource;
Debug.WriteLine($"{cue.StartTime} | {cue.Duration} | {sub.PixelWidth}x{sub.PixelHeight}");
});
}
} <Image x:Name="image" Grid.Row="1" Width="400" Height="300" />
|
Using ImageCues for ASS rendering is not suitable. It just doesn't go together. Even though the definition of ASS events involves a start time and a duration, an ASS event doesn't necessarily stand for a bitmap which remains static (unchanged) over the duration of an event - but that's what ImageCue bitmaps are designed for. Also, there's another mismatch: ASS event can overlap in time, so there can multiple be active at the same time. You are creating an ImageCue for each ASS event - which would still be fine in case of static bitmaps and without libass. But libass doesn't render any output that is related to a specific ASS event, so in turn it also can't render anything that is related to a specific image cue. Even further, an ImageCue is supposed to have a fixed position and size, but libass doesn't give you anything like that. Both can change from frame to frame. The overall conclusion is simply that You need to call ass_render_frame() once for each video frame being shown (or for every second one, etc..) and when detect_change is 1, you need to bring that output on screen in some way. |
In case when you would want to render ass subs statically without animation: That would of course be possible with ImageCues - but the problem here is that the FFmpegInteropX SubtitleProvider base implemention is not suitable for this, since it assumes that each AVPacket would create one MediaCue and that doesn't work out in this case. It would work like this:
Finally, there's one problem to solve: You don't want to create all the images on playback start, so you need to synchronize in some way with the playback position and make sure you only create image cues for e.g. the next 30s. This will give you static non-animated rendering of ASS subs - and that can be done using ImageCue. You also don't need to care about the rendering but let Windows Media to it (so no listening to cue entered events). PS: Happy New Year as well! |
Happy new year everyone! |
I’ve added a new function to the Despite this, subtitles still don’t display unless they’re manually added in C#. Additionally, I implemented a new function in https://1drv.ms/v/c/6ad13c09a43a4b36/Ef1Xvke1IG9MutjWh7NkQUkBP_BewvVVJwhKaahjI9nmNg?e=fIAgZ0 However, there’s an issue with blending colors—the color calculation is incorrect. None of the displayed colors match the actual intended colors. For reference, the correct colors should look like this: |
Hmm I was under the impression that the blending algorithm came from the JavaScript implementation? Haven't looked much at it, although iirc something jumped my eye at some point that seemed incorrect. In any case, colours aren't so important. We need to think through the animation side. Assuming the animation fps is the same as the video fps (the libass API seems to point in that direction), we can use the sample request events to drain the subtitle provider of animation frames. It would work similarly to avcodec or the filter graph. So some more ample refactoring might be necessary here. I see no reason ImageCue cannot handle animations, assuming 1 cue = 1 animation frame. Other than maybe potential performance problems in MPE. |
I do. One full-size PNG image for each video frame? Seriously? |
I found out why the cue doesn't appear in the UI: the (x, y) cuePosition was set to 100. I changed it to 0, and the subtitle displayed correctly. The ConvertASSImageToSoftwareBitmap function was created with the help of ChatGPT, so I’m not sure where it came from. However, I referenced multiple sources from different projects to ChatGPT, but none of them seem to work correctly. I also tried animations again, but there are many dropped cues, and most of them don't show up. However, as you can see, it works fine when you render it yourself with a timer—it's fast and works (except for the color part, of course). As @softworkz mentioned, I think the ImageCue is not meant to be used for animation effects. A side thought: Is it possible that our data (styles and colors) are incorrect when appending it to libass? |
You can easily find out by not doing it. For actual ASS subtitles, this shouldn't be done anyway. |
For the canvas control, only the .Dispose() needs to be done on the UI thread: https://microsoft.github.io/Win2D/WinUI2/html/M_Microsoft_Graphics_Canvas_UI_Xaml_CanvasVirtualControl_CreateDrawingSession.htm I'm still not sure whether the canvas control is the right thing for the job. It's definitely easy, though. |
CanvasAnimatedControl is not available in winui3. We should stick to swap chains. |
Yes of course. I thought this would generally apply to a CanvasDrawingSession. |
Wrong, please look closely.
await is not blocking. The UI thread is blocked less than 1ms per frame. During the 30ms of render, UI thread is free to do whatever it likes. Only after render is done, the rest of the loop is scheduled on UI thread again. If I call
This is what RenderSubtitleAsync does
It does not matter if you run the loop on dispatcher and run RenderSubtitle on a thead, or if you do the inverse (run loop on background and invoke present on dispatcher thread). In the end, the same parts are done on the dispatcher. The only thing that matters is how long and how often do you block and invoke the UI thread. And that's exactly the same for both approaches. |
I was refering to this loop, which has a single await only:
It does matter (at least it can matter). What I meant by "blocking" is that you don't give other things a chance to execute which are dispatched as well. The red lines are all places where the UI thread might be needed to execute something else, which you do not allow because you are running on that thread. That's what I meant by "blocking". |
No - not in the same order.
And at which moments in time.
No. When you'd do like I'm suggesting (doing only what's absolutely needed on the UI thread) there would be multiple dispatch invokes and thus more chances for the UI thread to execute other things. With the way I'm suggesting, no Task.Delay will be needed. EDIT: |
That single await gives the UI thread 30ms of time to do anything else. The rest of the loop (in this case) takes about 2ms, that's the only time where the UI thread is blocked. So more than 90% of the time, the UI thread is free. It might be possible to do even more stuff on the background thread in case of the swap chain. The canvas image (I originally tought you were talking about that one) does not allow background thread access. But then, on the canvas image loop, the UI thead is blocked about 0.1 ms per 30ms loop. So we sure do not have any issue there.
Yes, blocking 2ms of 32ms per loop. I guess it can be further reduced, since the swap chain allows more stuff to be called from background. But the app is super responsive anyways. I am not saying that this is finished. Just saying that we do not have any major issue with UI thead being blocked. If we'd block the UI thead for the 30ms of render (like in the very first version), then we'd have a major issue. But that's not the case anymore. |
What you are saying is all fine, but you miss my point: It is not about the amount of time for which the UI thread is running. It is about in which moments and states of other objects it can execute something. For example, it is not possible that a UI thread execution is run at a moment in time when there's no active drawing session: I mean - what are we talkiing about? You are wondering why you need an additional |
The longest period where the UI thread is blocked is for about 2ms. Yes this can be further improved, but it is nowhere critical. And it does not matter if a drawing session is open or not.
The additional Delay() is not needed in the swap chain loop. It is only needed in the canvas image loop, and it has to do with the fact that the device seems to be used. As I said, the UI thread is only blocked for ~0.2ms in a 30ms loop, in that scenario. It is not about UI thread being blocked. |
Oh, and I am not holding a drawing session or anything else in the canvas image loop! Still, there I need an additiona await inbetween draw and starting next render. |
Ah, I thought you meant it's needed there. But the same applies to the other case. You need to understand that other components can be using threads as well and at times they need to dispatch code to run on the UI thread - but the point in time matters at which this code is executed - for example before you continue your loop and do other things. Then it can be already too late for the other component's dispatch code to be run - and that's why you need to force the yielding via Task.Delay(). It doesn't matter whether it's 20ms or 0.2ms. If there's some dispatched code which needs to be executed in a certain state and you don't let it, things can go wrong.
Because it hadn't had the chance to dispatch some code on the UI thread |
In the swap chain loop, aside from the trivial size checks and the ass_image rendering, nothing is going on the CPU. And the ass_image is on a background thread. So that loop is actually very light cause almost everything is done either on background thread or GPU. So all the UI thread does are some trivial checks and calls to keep everything in order. Nothing to be worried about. |
It was @lukasf being worried about the need for the |
Sorry, got it. |
There are probably private calls happening in the Image control that's cause the app to freeze and thus lose the directx device. But that's no matter, since the swap chain is significantly faster and can achieve the theoretical maximum fps, we don't really need the image implementation other than maybe fall backs for regression checks. We do need to handle the device lost scenario on the swap chain as well. Reallocate everything and so on. Technically our inhouse cpp swap chain should work but it does not. I'll see what's up with that. |
What's actually the advantage of using Why not use |
That would be what the cpp sample front is trying to do. Except without win2d. |
@softworkz Nothing time critical can ever be done on the dispatcher thread, this is sure not the reason I had trouble. The dispatcher thread can easily and repeatedly be bogged for extensive periods of time. E.g. navigate to a new page with lots of list items with complex templates. The dispatcher thread might spend half a second or more creating and layouting hundreds of controls, and no one will get their continuations run on the dispatcher during that extensive period of time. If anyone uses it, expecting tight timing behavior and instant response, is totally going to fail. That's absolutely not what it is made for, and it never has and never will provide that. Using it for few ms is nothing, though it's best to use it as little as possible. We will be working on that, but it is trial and error to find out which APIs you can call from background and which you can't.
That would of course be the best way. You just can't easily do that from C# code. When going native code, we can as well directly go with the DXGI interfaces, allowing better optimizations. |
Actually the truth is somewhere in between. You may get device lost errors in directx if the dispatcher thread gets bogged down. However, the timeouts are somewhat on a different scale (miliseconds vs seconds).
So I was able to create the swap chain, attach it to the panel, and rendering + presenting seems to go without errors. |
As expected, the canvas swap chain is much more thread friendly than the canvas image. Basically the whole loop can run on a background thread (just done that). I'd guess even the size change could be done on bg, we'd just need to pass in the new size and dpi. I wonder what's the best approach to handle size changes with swap chain, to allow smooth resize without artifacts. Resize buffers will cause the image to disappear until a new one was rendered. Maybe it would be better to create a new swap chain, render to it, and then exchange the old for the new swap chain? Not that it's important now. Just noticing that the sub disappears and re-appears during resize. |
In my frame server mode implementation, I simply redraw the swap chains after a resize, somewhat outside the main callbacks. This is only done when playback is Paused, because when it is playing, the loop will simply pick up the change before the user can see anything. I think in the end we could go down the frame server mode way and render video + subs on the same swap chain. This should be theoretically the most efficient way. We can even use the Media Foundation subtitle rendering for non-ass subtitles. |
Create a static copy of the current image and display it in an image control with auto-resizing. The swapchain remains hidden (or has clear content). On each resize message, restart a timer (like 500ms). When it fires, resize the swapchain, hide the static image and continue swapping. |
Or - for not stopping animations:
In both cases this allows to do just few resizings for the swapchain - opposed to the many size changes. |
You are still totally misunderstanding what I'm trying to say. It's not about the period of time that it is unavailable (said it 4 times).
You can :-) |
This also gives you better control about the scheduling of things that need to run on the UI thread, as you can set a prioriy when invoking.
I would have started by putting every call inside a Dispatcher.RunAsync() lambda, and then try each one after another to exec directly on the bg thread. |
var height = mediaPlayerElement.ActualHeight; | ||
swapChain?.ResizeBuffers((float)width, (float)height, displayInfo.LogicalDpi); | ||
swapChainSizeChanged = false; | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should use .ConfigureAwait(false)
now on all await
calls. Currently, it synchronizes back to the same threadpool thread like before the await and when that thread isn't available, it will block until it is.
This sounds a bit suspicious - probably there's some magic in the canvas swap chain to make it convenient? In that case, the question would be what's the cost of it.. |
This is actually what I expected and how it is documented (dxgi swap chain docs). Only I was not sure if the win2d abstractions add some dependency on the dispatcher. It's good that that's not the case. The canvas swap chain could be quite usable, if only it would expose the buffers directly. |
Hi,
I've made some changes to
FFmpegInteropX
to supportlibass
as a subtitle renderer. The implementation is largely inspired by thelibass
integration for JavaScript, which you can find here:https://github.com/libass/JavascriptSubtitlesOctopus
By default,
libass
operates as follows:ass_library_init
.ass_renderer_init
.ass_read_memory
(other methods exist, but we're constrained by UWP).ass_process_codec_private
.FFmpeg
usingass_process_chunk
.IMediaCue
.Libass
usesass_render_frame
to generate anASS_Image
, which works well for rendering. However, since this process must happen in real-time, I’m unsure if it's feasible to createIMediaCue
instances based on current implementation. Is it possible to display subtitles accurately using media duration?P.S. I’ve noticed a recent issue compiling
FFmpegInteropX
with the target platform set to10.0.22000.0
. To resolve it, I switched to10.0.26100.0
.Thanks.
#439