Get packet buffer size from operating system

[XAMPPRocky]

Currently Quilkin always allocates a u16::MAX length buffer for every message, this is incorrect for a number of reasons. Firstly, the max length is slightly less than that as it's really 65535 - (IP + UDP headers), further is far too large compared to realistic workloads which are typically in the 200–500 bytes range (to prevent IPv4 fragmentation), and is much bigger than what most OS' set as the limit for UDP payloads. For example; macOS by default has a limit of 9216.

Just testing #321 with different workloads and buffer sizes, having a lower buffer has a significant impact on the performance. For example having a buffer of 1500 bytes had a 15% increase in throughput for smaller workloads. So instead of always using the maximum, we should try to be a bit smarter about how much we allocate. A good default would be to query the OS for the maximum UDP buffer, I also think it would be worth adding a server level option for setting a buffer limit manually allowing you to lower it for when you're using it with a protocol with a smaller MTU.

quilkin/src/proxy/server.rs

Lines 240 to 242 in 35005f0

	// Initialize a buffer for the UDP packet. We use the maximum size of a UDP
	// packet, which is the maximum value of 16 a bit integer.
	let mut buf = [0; 1 << 16];

[iffyio]

The u16::MAX buffer isn't allocated per packet, notice its outside of the loop - we reserve that space inline once on startup (the ip+udp header is so small it shouldnt be significant here). for each packet we allocate its exact size

quilkin/src/proxy/server.rs

Line 250 in 35005f0

.send((recv_addr, (&buf[..size]).to_vec()))

It's interesting that there's a significant difference with different buffer sizes then, can't really see why that would be the case but I can imagine the future will be pretty big due to the 64K buffer so that moveing it at any point will be a more expensive operation - I thought this would only be done once on startup too but maybe (pure speculation :) tokio internally moves futures e.g while it schedules tasks? Otherwise I suspect the benchmark is noisy such that startup changes are significantly affecting its result.

I'd suggest to instead move the buffer to the heap (i.e use a Vec) in any case to avoid the big future issue and see how far that goes, I suspect that would resolve this issue. Adding some cross platform code to detect OS buffer size complicates things and this doesnt feel worthy of a config option so would be nice to avoid those if we can.

[XAMPPRocky]

Well you can try the benchmark yourself, it records the previous and tells you the diff , it will tell you if it's within the noise threshold. I included an example output (though not from the same run) Some of it is probably noise, but yeah it is also not great that we're allocating 65kb on the stack or in general, because if we want to run a lot of instances of Quilkin it will add up pretty quickly. We'd be able to get a more fine grained view of the performance in tokio with #317.

Adding some cross platform code to detect OS buffer size complicates things and this doesnt feel worthy of a config option so would be nice to avoid those if we can.

Yeah I wouldn't want to add the OS code to Quilkin itself, not the least because it would be generally more useful to other projects, but if someone in the community makes or has already made a cross platform lib that handles all the complexity and we just get the size (similar to num_cpus), I wouldn't see an issue with adding it.