Camera jitters when you look and move at same time

[BlackGoku36]

To see this jitter, you should circle around an object, while looking at it.

Untitled.mp4

Maybe it is not much visible in the above video. But it will be clearly visible if you, try it on a single triangle.

Platform: x86_64 macOS

[sotrh]

I'm not able to replicate it on my machine. Are you on M1?

[BlackGoku36]

🤔
Nope

[dartvader316]

Same problem. I am able to replicate it on Linux + Wayland and even with Windows build under Wine.
It happens on any kind of diagonal mouse move. Does not replicate with straight vertical or horizontal mouse move.
I used showcase/mouse-picking for bulding executables.

[sotrh]

Hmm, I wonder if it's an integrated graphics thing. I'm running on a discrete Nvidia GPU.

[dartvader316]

I tried my discrete Nvidia GPU and still able to replicate this issue.
However i found that disabling Vsync with present_mode: wgpu::PresentMode::AutoNoVsync in wgpu::SurfaceConfiguration fixes this issue. Also i added a simple frame rate limiter and limited my frame rate to 60 and 30. And then i was able to replicate this issue again.
So basically from my observations the less frame rate you have -> the more camera jitters on diagonal move. With limited 15 FPS you cant even move camera straight diagonal at all.

[CatCode79]

I have not been able to reproduce the problem but I wonder if the problem is actually on the input side.

In the update_camera method there are many lines of code that depend on dt, a value that changes proportionally to the time of the last redraw (i.e. it is frame dependent).
But it's counterintuitive because decreasing the number of frames should increase the value of dt and therefore it doesn't degenerate towards zero (thus blocking the movement of the camera), however my best guess is to have a look at the values of that function.

Does the jitter happen even just moving diagonally (using only WASD keys) or does it exist on some sort of rotation as well (as in the OP's test)?

Edit: I don't think it's a useful question to solve the problem anymore, I found material on the internet about it and I'll write it in another message

[CatCode79]

I found this problem described in some forum about gamedev unity, example:
https://forum.unity.com/threads/fixed-update-jitter.893686/
usually the solution they give is to move the input update after the frame draw.

I don't know if this would fix it in our case, what I expect is that unity has a pipelined renderer across multiple threads (so game and input logic is parallel to that of draw, or at least any serious engine has a similar system).

There is also this page which is interesting:
https://gamedev.stackexchange.com/questions/97972/render-draw-or-input-first
and in particular it says:
If you're using vertical synchronization (or it's forced), it will typically happen at the end of draw(), when the current scene/back buffer is presented. This means that if you're drawing first, your input will always lag behind by one (display) frame. Depending on your frame rate this might be very noticeable to the user.
From what I understand it says that in this case it is better to update the input (state.update() for us) before the draw, which however is already done in the tutorial and therefore I am not sure what happens to change the order...

[sotrh]

I agree with @CatCode79, it would make sense that this is some weirdness with the input. @dartvader316, try putting the update method after the render match statement and turn v-sync back on. See if that fixes it. The RedrawRequested code should look something like this:

Event::RedrawRequested(window_id) if window_id == state.window().id() => {
    let now = instant::Instant::now();
    let dt = now - last_render_time;
    last_render_time = now;
    match state.render() {
        Ok(_) => {}
        // Reconfigure the surface if it's lost or outdated
        Err(wgpu::SurfaceError::Lost | wgpu::SurfaceError::Outdated) => state.resize(state.size),
        // The system is out of memory, we should probably quit
        Err(wgpu::SurfaceError::OutOfMemory) => *control_flow = ControlFlow::Exit,
        // We're ignoring timeouts
        Err(wgpu::SurfaceError::Timeout) => log::warn!("Surface timeout"),
    }
    state.update(dt); // Updated!
}

[jaydevelopsstuff]

I'm able to reproduce this issue on Windows, even using what @sotrh suggested, the jittering persists.

[sotrh]

@BlackGoku36 and @jaydevelopsstuff , do either of you get the same jittering when just moving via the keyboard?

[jaydevelopsstuff]

@BlackGoku36 and @jaydevelopsstuff , do either of you get the same jittering when just moving via the keyboard?

No, the jittering only occurred for me when moving the mouse around.

[CruizK]

Same here on Linux, X11, disabling vsync fixed it, but not ideal :/

[CruizK]

Ok, I think I found out why & a couple solutions.

The problem
The actual problem appears to be that vsync blocks the main thread which causes winit to build up events, usually this would be fine but since we only take the latest MouseMotion delta, it means we lose any motion that would happen before that. I'm still not 100% sure why this causes this "jitter" effect my inital thought is that "It shouldn't really matter", maybe someone else could answer this.

The solution(s)
The first solution I thought of is obviously to just "run rendering on another thread & do updates/input on the main thread or vice versa", but I'm a noob when it comes to threading, especially in rust, and it seemed like a job for future me, so I tried another solution.

Basically we average the MouseMotion, simply, instead of assigning, we add all the motion deltas together, also keeping track of how many we added, then at some point (I do it before the mouse_delta calcualtions) you calculate the actual average mouse motion of that "frame" and then reset it all back to zero & repeat.

My code looks something like

// Wherever you track your mouse_pos I use an InputState struct that gets passed to my update method
self.mouse_delta += Vector2(..);
self.mouse_samples += 1;
/*
...
...
*/
// In our update method
let mut real_delta = Vector2::zeroes(); // I'm using Nalgebra idk if its the same in cgmath
if input.mouse_samples != 0 { // Have to make sure they actually moved the mouse or we get / by 0
    real_delta = input.mouse_delta / (input.mouse_samples as f32);
}
// do calculations with real_delta now
input.mouse_delta = Vector2::zeroes();
input.mouse_samples = 0 

Below is a video of a before and after

Before.mp4

After.mp4

I don't know how well that comes across on video but on my 144hz monitor its a world of difference. Anyway hopes this helps. Still not sure why only some people seemed to experience this though.

[sotrh]

Glad you found a work around. Currently threading is not supported on web directly, so unless I want to mess around with web workers and offscreen canvas, the motion averaging/disabling vsync might be the best idea.

[Namaarie]

Instead of averaging multiple mouse_deltas, you can also refactor the update_camera() function inside camera.rs to separate position updates and rotation updates. Then, you can change your State struct to update the camera rotation on each MouseMotion event, and only update camera position inside update(). You also don't need to multiply the rotations by delta_time since it is updated every event regardless of fps. A more idiomatic solution would be to append all events to a queue, and then read each event from the queue during updates. This way, no events are missed due to Vsync (this seems to be how Bevy handles this).

For example:

// inside State struct
fn update_camera_rotation(&mut self) {
    self.camera_controller.update_camera_rotation(&mut self.camera);
}

// inside State.update()
fn update(&mut self) {
    self.camera_controller.update_camera_position(&mut self.camera, self.delta_time_f32);
    // everything else
    // ...
}

// mouse motion event
Event::DeviceEvent {
    event: DeviceEvent::MouseMotion { delta },
    ..
} => {
    state.camera_controller.process_mouse(delta.0, delta.1);
    state.update_camera_rotation();
}

// camera.rs
pub fn update_camera_rotation(&mut self, camera: &mut Camera) {
    camera.yaw += Rad(self.rotate_horizontal) * self.sensitivity;
    camera.pitch += Rad(-self.rotate_vertical) * self.sensitivity;
    self.rotate_horizontal = 0.0;
    self.rotate_vertical = 0.0;
    if camera.pitch < -Rad(SAFE_FRAC_PI_2) {
        camera.pitch = -Rad(SAFE_FRAC_PI_2);
    } else if camera.pitch > Rad(SAFE_FRAC_PI_2) {
        camera.pitch = Rad(SAFE_FRAC_PI_2);
    }
}

pub fn update_camera_position(&mut self, camera: &mut Camera, dt: f32) {
    // everything else left over from update_camera()
    // ...
}

learn-wgpu-mouse-fix.mp4