Spatial Audio & Enhanced Audio Processing

[BenCheung0422]

Introduction

The previous issue, #624, was quite ambiguous and undetailed about the features, this one is made, with more complementary details and information about this extended topic. Now, it not only involves spatial audio but more areas in the audio system. And, some concerns may also be addressed.

Details

The implementation would be based on OpenAL (OpenAL Soft, supported by LWJGL) with the spatial audio support. Firstly, the current audio system would be replaced by the addition of OpenAL with more functionalities and expansions. OpenAL is basically a 3D audio library, so spatial audio would be mainly handled by it.

The major feature is spatial audio, which has been mentioned before and is common in 3D game, but applicable to 2D game with proper implementation. Since the player character's direction might change in a flash, which is quite different from 3D that the player character changes the direction progressively, the implementation in 2D should not rely on player character's direction, but the relative coordinates or screen view. The disadvantage for the former one, as well briefly mentioned by Makkkkus¹. This could cause an abrupt audio outcome if the player constantly changes direction, which is not ideal. That is why I suggested implementing it with respective to the screen/top view, while technical playing may benefit from it by the reflection of directions of sources. Hence, this could improve gameplay experience, and I add that, in-depth spatial audio speakers like 5.1/7.1 may be experimentally supported.

Another key feature is distance attenuation, which has already been supported by OpenAL. This could greatly enhance player gameplay experience by the gradation of sound levels. In the current implementation, in-game sources are played at either 0% or 100% depending on whether they are in enough distance from the player, but the change may be sudden. Also, it may sometimes cause choppy transition if the sound source is continuous. Speaking of technical playing, this could reflect the distance from the sound sources. Therefore, distance attenuation could help solving this problem whilst make sure that the resultant volumes may not be too high in most scenarios.

The important part is the nature of audio data. For the sound effects which would be played in world, they should always be mono, as to be compatible with the spatial audio system. Currently, in the vanilla game assets, only explode.wav is stereo while others are mono, which would be something to solve before implementing this system, but should have been replaced by new assets.

On the other hand, audio mixing and compression, which I have made a pull request before to solve problems with simultaneous multi-playing of the same clips, as #544. Actually, OpenAL may require the similar techniques for the clip playing², but it is not today's topic. The major reason I used an algorithm for signal compression is that, no audio source is attenuated in the current implementation, so when multiple sources are played, the maximum thresholds of integer data can be easily reached, causing integer value overflow, requiring necessary clipping, as loudness war. Obviously, this is absolutely unideal and can be frustrating if not handled correctly. Instead, when there are audio attenuations applied while there are only single audio input for sounds like BGM, the necessity of compression after mixing is reduced, while the corelation of signals may be low.³ Thus, only little compression may be applied, or none at all. In-game audio volume controls for different types of sound sources should be made for such situation, like the one in Minecraft.⁴ In fact, optional but customizable (and may be advanced) equalizer or compressor could still be made into settings for users, though may be quite complicated to be in game.

Last but not least, the key points here still mainly focus on user experience and gameplay smoothness. Still, ensuring unhindered experience is the essential. If there are still doubts regarding the above implementations, you are welcome, but this is still not exhaustive enough to the final actual implementation.

Footnotes

1. Commented by Makkkkus in Enhanced spatial audio support #624:

   [...] Also the player changes direction instantly, which would also be uncomfortable for the listener. [...]

   ↩

2. "OpenAL: how to play multiple sounds at the same time and mix them?" answered by Damon ↩

3. "Algorithm(s) to mix audio signals without clipping" answered by endolith ↩

4. "Algorithm(s) to mix audio signals without clipping" commented by pichenettes ↩

[zandgall]

This should be rather smooth and nice to implement. As I've done this already, happy to work on this. In case someone else also wants to look over it, one of my previous implementations can be seen here. This branch is quite far behind this repo, and the actual implementation isn't commented and not what the accepted implementation should look like, but it's there for reference on getting working audio. (Does not include spatial audio, but would be trivial to implement. Accepting in-world coordinates for sound sources, and changing the listener position to the in-world player position.)

[BenCheung0422]

Thanks for providing a reference to sample implementation, I guess it would be useful for later development.

[Makkkkus]

I'll copy some statements from the last issue so it sinks in.

I advice against; speaking from experience, positional sounds in a 2D game sound very uneasy. Also the player changes direction instantly, which would also be uncomfortable for the listener. So I advice as much as I can against using positional audio.

Originally posted by @Makkkkus in #624 (comment)

Okay, honestly please stop trying to fix things that aren't broken. We will be adding background music and ogg support eventually.

Originally posted by @Litorom in #624 (comment)