Stream lags with any sort of motion

[BringZeWubs]

When any sort of motion is on screen, whether a video or a game, the frames drop to unplayable levels and spam the log with "Waiting for IDR frame," occasionally crashing minutes later.

Steps to reproduce the behavior:

1. Connect to PC and open any sort of moving picture (YouTube, any game, Wallpaper Engine, etc.)
2. Watch frames drop. Moving the mouse around shows the delay and dropped frames.

Expected behavior
A smooth stream with minimal lag and lack of dropped frames.

** Logs **
The log is filled with things like:
IDR frame request sent
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Network dropped 30 frames (Frame 8299 to 8329)
Unrecoverable frame 9033: 13+7=20 received < 30 needed
and so on.

Guest Information:

• Xbox Model: Xbox Series X
• TV Resolution: 1080p

Host Information

• Operating System: Windows
• Host Software: Sunshine v0.23.1
• Monitor Resolution: 1080p

Additional context
I tried streaming my PC with the regular Moonlight Android app and it worked fine with the usual hiccups from Wi-Fi usages, so I'm wondering if it's an Xbox version only issue