🐢 Create inter-thread executor API

[joshua-holmes]

This is a sub-issue of #17667. See that issue for more high-level details.

Borrows ideas from maniwani's PR to remove !Send resources

What problem does this solve or what need does it fill?

In order to finish removing !Send resources (started here), we need a way of forcing certain tasks to run on main thread, without depending on NonSendRes system params.

What solution would you like?

Overall plan

We will spawn a new thread which holds an event loop proxy and holds a sender and receiver for winit. World will hold another send/recv for the proxy. A system param will be created for use in systems and will come with a method that takes a callback function and executes the function on the main thread. It does so by communicating with the proxy, which wakes the event loop and sends the callback to the event loop, which will then block until execution is finished.

Details

@maniwani provided the following process:

• Leave an executor in the main thread with winit.
• Store a handle/channel—that you can use to send tasks/callbacks to the executor—as a resource in the world. Note that you also have to use winit's EventLoopProxy to actually wake up the event loop.
• Create a system param that borrows the handle and the proxy and exposes a method that sends a task/callback, wakes the loop, and blocks on the task's completion.
• Have winit run all submitted tasks/callbacks in its own ApplicationHandler::proxy_wake_up callback (or ApplicationHandler::user_event, idk what version of winit we're on).

and provided this rough example:

pub struct BevyWinitAppHandler {
    /* ... */
    send: Sender<Foo>,
    recv: Receiver<Bar>,
}

impl BevyWinitAppHandler {
    pub fn new(send: Sender<Foo>, recv: Receiver<Bar>) {
        /* ... */
    }
}

impl ApplicationHandler for BevyWinitAppHandler {
    fn proxy_wake_up(&mut self, event_loop: &dyn ActiveEventLoop) {
        /* ... */
        while let Ok(callback) = self.recv.recv() {
            // execute task we've received from some system
        }
    }
    /* ... */
}


fn example_runner_function() {
    let mut event_loop_builder = EventLoop::builder();

    /* ... */

    let event_loop = event_loop_builder
        .build()
        .expect("`winit` event loop can only be created once");

    let waker = event_loop.create_proxy();

    // winit can send directly to world
    let (winit_send, world_recv) = std::sync::mpsc::channel::<Foo>();

    // world must send through proxy (to wake up the loop)
    let (world_send, proxy_recv) = std::sync::mpsc::channel::<Bar>();
    let (proxy_send, winit_recv) = std::sync::mpsc::channel::<Bar>();

    // spawn a thread to manage the `EventLoopProxy`
    // (this way `bevy_ecs` can avoid depending on `winit` or having to define a trait)
    std::thread::Builder::new()
        .name("main-event-loop-proxy".to_string())
        .spawn(move || {
            while let Ok(callback) = proxy_recv.recv() {
                waker.wake_up();
                proxy_send.send(callback).unwrap();
            }
        })
        .unwrap();

    // spawn a thread to manage the app
    std::thread::Builder::new()
        .name("app".to_string())
        .spawn(move || {
            let result = catch_unwind(AssertUnwindSafe(|| {
                /* move the main sub-app/world in here */

                // you'll actually have to newtype these channel halves,
                // but hopefully you see the point...
                world.insert_resource(world_recv);
                world.insert_resource(world_send);

                /* ... */
            }));

            if let Some(_) = result.err() {
                // TODO: log panic
            }
        })
        .unwrap();

    // start the event loop
    event_loop.run_app(BevyWinitAppHandler::new(winit_send, winit_recv));
}

NOTE: the example includes separating the world from the winit thread, but that is out of the scope of this issue and will be completed as work for a future issue. The scope of this issue is only to create this functionality, along with a few tests, but not to apply it anywhere (other than the tests).

This would allow code that needs to be run on main thread to stay on main thread without depending on NonSendRes.

What alternative(s) have you considered?

We have talked about sending entire Systems to be executed on different threads, but this would prevent