Working with Audio Observations on ViZDoom

[romulofff]

Hello Alex and SF community.
I have come across the paper where you guys included audio/sound observations in ViZDoom (Farama-Foundation/ViZDoom#486) and became fascinated by it. I'm doing research on this subject and I want to expand the ideas presented in your previous work, possibly tackling some of the future works points mentioned.

However, I'm struggling to get this to work with Sample Factory. I understand that the code from the paper is from an old version of Sample Factory and that the library has changed so much since the paper publication that it is not compatible anymore. So does Sample Factory supports audio observations natively or should I make changes to the gym wrapper in order to support it? If you have any tips or guidance on how I could get this running it would be amazing.

Congratulations on this amazing work with both the ViZDoom improvement and Sample Factory!

[alex-petrenko]

Hi @romulofff ! I'm glad you found this research interesting!

I'm sure you've seen the original repo associated with the paper?
There is code there for models that we used which you can rather easily use with the newest version, although I imagine it will require some adaptation

https://github.com/hegde95/Agents_that_Listen/blob/main/envs/doom/doom_model.py

Documentation is a useful resource for implementing custom models etc. https://www.samplefactory.dev/03-customization/custom-models/

Please let me know if you have specific questions! I'm glad to answer here or in Discord

[romulofff]

Hi @alex-petrenko, thanks for the reply!

Yes, I'm familiar with that repo, however I was wondering if there was already support for audio observations in the current version of Sample-Factory ViZDoom wrapper. From your answer (and from the hours reading the docs/code) I'm understanding that there is not, correct?

Well, guess I'll lift the sleeves and start working on adapting the paper code to the newest version. Perhaps that could lead to a PR in the Sample-Factory project, would that be desirable? Anyway, thank you for the suggestion, I'll start working on it immediately and if something more specific comes up I'll reach out again :D

[alex-petrenko]

From your answer (and from the hours reading the docs/code) I'm understanding that there is not, correct?

Yes, the audio support was done in a separate repo :) The motivation was that it is not really the part of the library, nor it is some kind of obvious example. So it was wiser to put it in a separate repo.
But we were able to merge the sound support into the official VizDoom repo!

My advice would be to make a separate repo for it (or fork the repo mentioned above) - should be the most convenient option for you. The library was designed to be easily extensible in this way.

However if you're willing to write a PR and some documentation for it, I'll gladly review it too!