Realtime Rendering to OBS (NDI)

[havi05]

I want to render animations (and other things) in realtime from Godot to OBS for streaming (some kind of custom alerts). I think the NDI protocol could be used for this and implemented as an addon (maybe official?) because not many people need this for Games.

There is an Q&A from 2020 about this: https://godotengine.org/qa/62202/godot-engine-for-virtual-production

What do you think about this?

[infohub-gather]

Having more support for video input and output and in general various protocols would be very beneficial. It would open up Godot for more and wider professional use cases in broadcast, streaming, presentations, visualisations and other applications (other than gaming)

Since we love open source there is a new protocol called teleport that is being used in OBS : https://github.com/fzwoch/obs-teleport

Especially Godot 4 new impressive rendering capabilities will be great for these new use cases.

[XeroWolf]

I've actually attempted this a couple years ago and got it working in a very limited form. So I'm here to say it's definitely possible. However with the many updates to Godot and the move to GD Extension I have not been able to get it compiled since then. Take note that my C++ knowledge is very limited as I'm a video editor not a programmer lol
I'm willing to take another stab at it in the next couple weeks or so. Even if is to just get a proof of concept working and then have a more experienced coder take it from there.

@infohub-gather Teleport looks promising. I'm definitely going to take a look at it as well. Maybe I might have better luck with it that NDI.

[unvermuthet]

I reckon output should be done as a Node that looks for the closest viewport up the tree the same way a camera node does. I’ll have a look at that example, and how to obtain a viewport texture. But as of now I’m still concerned with some bugs in the receiving part. Also I might turn the NDIFind into a Node instead of a RefCounted. That way it would be possible to send signals whenever new sources are discovered.

Last post here. For any follow ups I’ll make an issue over at the project repo.

[XeroWolf]

Would you be interested in sharing your work? Took a look at your git but no repos...

@GoranKovac
I pretty much scrapped that project. It was rubbish to be honest lol but I upload a new one you guys can check out.
https://github.com/XeroWolf/Godot-NDI-CSharp

I spent the better part for two days giving this another go and I've had some promising results. I've opted to use C# this round instead of C++ with GD extensions. Currently I'm able to send frames of the Viewport in near real-time over NDI.
Basically it's grabbing the data from the viewport and performing and copying it to the memory location allocated for NDI to use for sending frames. It's not very performant but it's way better that what I was able to do before. Hopefully someone more experienced can chime in on how to improve this. Here's a sample for the part that does the heavy lifting.
If any of the Godot Devs know of a faster way to do this it would be much appreciated. 👍

   //This method is called by _Process(double delta)
    public void ProcessFrames()
    {
        //Grab the viewport data as bytes. Godot Docs recommends not doing this every frame
        imageData = subViewport.GetTexture().GetImage().GetData();
    
        //copy the viewport data to the memory allocated for use by NDI
        Marshal.Copy(imageData, 0, bufferPtr, imageData.Length);
    }

In my example I'm using two Sub Viewports. One main one and the other to hold the 3D scene. There's an option to hide the background.

I exported a couple options in the editor like frame rate and send rate. The send rate is a way to adjust the performance for now. It's a percentage of the frame rate so 0.1 is 10% and 1.0 is 100%. Again 100% can be slow if the frame rate and or resolution is high.

You need to set the viewports background to transparent for this to work properly. The image get messed up otherwise.

[GoranKovac]

Nice work! I've already figured out whats needed to send the data (pretty straight forward). Unfortunately cannot try it since I'm on linux. Anyway regarding your last question it seems godot 4.4 has implemented some improvements to getting data async. You can try it out:
godotengine/godot#100110 (comment)

[XeroWolf]

Cool thanks! This is just what I was looking for. Initial tests are already showing improvement in the frame rate.

[unvermuthet]

My extension now supports transmitting the Viewport as a NDI source, as well as receiving and playing back. Minimum Godot version is 4.4-dev7. I rely on the new RD::texture_get_data_async.

https://github.com/unvermuthet/godot-ndi/

Could this be listed with the Godot Community SDK Integrations? @akien-mga

[lostminds]

Another option that's already supported could be just using the transparent window mode, and just have your godot app render in a borderless window with transparent background you put on top of whatever you're streaming on screen, and include it that way. Like this example seems to do https://www.reddit.com/r/godot/comments/1aq5xb3/thought_about_streaming_tekken_8_but/ for an arcade controller overlay.

[Calinou]

Note that borderless windows that are always on top can't be displayed if the other app is in exclusive fullscreen mode.

For OBS streaming, this isn't an issue as you can compose both windows together, but this is something to keep in mind if you're trying to record the entire screen at once (e.g. using a capture card).

[XeroWolf]

I made a recording of it in action.

Godot.NDI.mp4

[GoranKovac]

I started working on a tool that would utilize godot in broadcast graphics. Plan is to make it simple as possible:
Create scenes with animations, drag it into broadcast panel, assign which animation should be played (animation player is mandatory) and play it in out etc.
Here is the very early concept and ofc it uses excellent NDI work by @unvermuthet which without this would not be possible :

Broadcast.Graphics.mp4