Building the Large model in WSL requires 32G of RAM and Home Assistant questions

[mikey60]

I thought I would relate my experience in case others have issues. I have an old Windows PC with a 4th generation I7 processor and a Nvidia GTX 1060 with 6G of RAM that I just purchased used on ebay and installed in the PC. I was not able to build the large model in WSL with the 16G of RAM installed in the PC. I tried multiple times. Once I doubled the PC RAM to 32G the large model built without issues. The additional 16 G of DDR3 RAM was less than $20 on Amazon. The Willow Inference Server is running without issue in WSL under Windows 10. I am not using the graphic card in Windows. I am using the motherboard (Intel) GPU.

From reading the Willow documentation I understand that the TTS response from HA is still a work in progress, but I do not see the text responses on the ESP32 Box display for the status questions like "Is the garage door open." I am running version 2023.4.x of Home Assistant. Did I miss a configuration setting or is the text response of the ESP32 BOX also a work in progress.

[nikito]

I'm using WIS local and I get both spoken responses as well as text on screen. Did you set Willow to use TTS instead of beeps when running through the config?

[kristiankielhofner]

More RAM never really hurts but for WIS 1.0 (coming early July) we will download the correct models directly and skip the conversion step to save on RAM and disk space.

In terms of TTS output, as @nikito said you need to configure TTS response mode prior to build. Do you see output/status response text on the display? That is the text that will be sent to WIS for TTS generation. Also, make sure you have your volume configured to at least 60.

[mikey60]

@nikito Thanks for the quick response. I do have TTS enabled in the configuration and the volume set at 100. I do hear one syllable responses that are very low volume. I believe that 'error' is the response I hear when the HA intent is not found. I also see the the "What I heard" response and the "Command Status" response on the display. I do not the see HA intent response on the display or hear the HA TTS response on the speaker. As an example I have an HA intent for area temperatures. The intent responds with "The area i.e basement temperature is xx degrees F in the assist screen of Home Assistant. The ESP32 Box display only shows the correct speech heard and Command Status Success as it does for all correctly heard intents. I am using Home Assistant version 2023.4.x which might be part of the problem.
@kristiankielhofner I am looking forward to version 1.0 of WIS

[nikito]

So just to be clear, you have these?


Note in the second image that is my local server, if you are not running local you would need to use the tovera hosted server, otherwise replace with your own address.
I think the Command Status: Success shows when it is not in TTS Mode? On mine I do not have a Command Status, I just have the "I Heard:" with what was heard, and "Response:" with the TTS response text.

Also, if the intent is unknown to HA, it will reply back with "Sorry, I didn't understand that." So you would still get something.

[mikey60]

My settings are as follows:
Willow Speech Recognition Mode (Willow Inference Server) --->
Willow audio response type (Audio response using WIS TTS - English only for now) --->
(https://192.168.1.51:19000/api/willow) Willow Inference Server URL
(https://192.168.1.51:19000/api/tts) Willow Inference Server TTS URL
Willow Wake Word (Hi ESP) --->
Command Endpoint (Home Assistant) --->
(192.168.1.13) Home Assistant host
(8123) Home Assistant port
[*] Home Assistant TLS/SSL

My WIS startup message is as follows:
[2023-06-20 02:58:03 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-06-20 02:58:03 +0000] [1] [INFO] Listening at: https://0.0.0.0:19000 (1)
[2023-06-20 02:58:03 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2023-06-20 02:58:03 +0000] [80] [INFO] Booting worker with pid: 80
[2023-06-20 02:58:04 +0000] [80] [INFO] Willow Inference Server is starting... Please wait.
[2023-06-20 02:58:08 +0000] [80] [INFO] CUDA: Detected 1 device(s)
[2023-06-20 02:58:08 +0000] [80] [INFO] CUDA: Device 0 name: NVIDIA GeForce GTX 1060 6GB
[2023-06-20 02:58:08 +0000] [80] [INFO] CUDA: Device 0 capability: 61
[2023-06-20 02:58:09 +0000] [80] [INFO] CUDA: Device 0 total memory: 6442188800 bytes
[2023-06-20 02:58:09 +0000] [80] [INFO] CUDA: Device 0 free memory: 5466226688 bytes
[2023-06-20 02:58:09 +0000] [80] [WARNING] CUDA: Device 0 has low memory, disabling chunking support
[2023-06-20 02:58:09 +0000] [80] [WARNING] CUDA: Device 0 is pre-Turing, forcing int8
[2023-06-20 02:58:09 +0000] [80] [INFO] Started server process [80]
[2023-06-20 02:58:09 +0000] [80] [INFO] Waiting for application startup.
[2023-06-20 02:58:09 +0000] [80] [INFO] CTRANSLATE: Supported compute types for device cuda are {'float32', 'int8'} - using configured int8
[2023-06-20 02:58:09 +0000] [80] [INFO] Loading Whisper models...
[2023-06-20 02:58:24 +0000] [80] [INFO] Loading TTS models...
[2023-06-20 02:58:30 +0000] [80] [INFO] Warming models...
[2023-06-20 02:58:47 +0000] [80] [INFO] Willow Inference Server is ready for requests!
[2023-06-20 02:58:47 +0000] [80] [INFO] Application startup complete.

[mikey60]

Yes, I did build and re-flash. Just to be sure I just did a git pull on the repository, re-build, erase flash, and flash again. It still acts the same. The might be an I2S issue with the audio. I have captured the last lines of the ESP32 Box re-boot and the "basement temperature" command interaction. There is one warning and error I received during the initialization.

I (799) spiram: SPI SRAM memory test OK
W (00:00:04.610) PERIPH_WIFI: WiFi Event cb, Unhandle event_base:WIFI_EVENT, eve                                          nt_id:4
I (00:00:06.111) WILLOW: initializing SNTP client
I (19:00:06.112) WILLOW: Using configured SNTP server 'pool.ntp.org'
I (19:00:06.170) WILLOW: audio_hal_ctrl_codec: ESP_OK
I (19:00:06.172) WILLOW: HASS URL: https://192.168.1.13:8123/api/components
I (19:00:07.074) WILLOW: HTTP status='200' content_length='3513'
I (19:00:07.084) WILLOW: audio_element_getinfo(hdl_ae_hs): sample_rate='44100' c                                          hannels='2' bits='16' bps = '0'
----------------------------- ESP Audio Platform -----------------------------
|                                                                            |
|                       ESP_AUDIO-v1.7.2-20e6bd0-b92a149                     |
|                     Compile date: Nov 30 2022-07:50:12                     |
------------------------------------------------------------------------------
I (19:00:07.126) WILLOW: audio player initialized
E (19:00:07.127) I2S: register I2S object to platform failed
I (19:00:07.133) WILLOW: Using record buffer '12'
MC Quantized wakenet9: wakeNet9_v1h24_hiesp_3_0.63_0.635, tigger:v3, mode:2, p:0                                          , (Jun 14 2023 11:15:21)
I (19:00:07.314) WILLOW: app_main() - start_rec() finished
I (19:00:07.317) WILLOW: MAC address: f4:12:fa:fd:0d:40
I (19:00:07.317) WILLOW: Startup complete! Waiting for wake word.
I (15:06:40.574) WILLOW: SNTP client synchronized time to 1687378000
I (15:06:49.774) WILLOW: Wake LCD timeout, turning off LCD
I (15:08:20.194) WILLOW: AUDIO_REC_WAKEUP_START
I (15:08:20.363) WILLOW: AUDIO_REC_VAD_START
I (15:08:20.366) WILLOW: Using WIS URL 'https://192.168.1.51:19000/api/willow'
I (15:08:20.368) WILLOW: WIS HTTP client starting stream, waiting for end of speech
I (15:08:25.196) WILLOW: session timer expired - forcing end stream
I (15:08:25.259) WILLOW: WIS HTTP client HTTP_STREAM_POST_REQUEST, write end chunked marker
I (15:08:26.122) WILLOW: WIS HTTP client HTTP_STREAM_FINISH_REQUEST
I (15:08:26.127) WILLOW: WIS HTTP Response = {"text":"basement temperature","language":"en"}
I (15:08:26.129) WILLOW: HASS URL: https://192.168.1.13:8123/api/conversation/process
I (15:08:26.137) WILLOW: sending '{"text":"basement temperature","language":"en"}' to Home Assistant API on 'https://192.168.1.13:8123/api/conversation/process'
I (15:08:27.102) WILLOW: HTTP status='200' content_length='225'
I (15:08:27.108) WILLOW: home assistant response_type: action_done
I (15:08:27.110) WILLOW: Using WIS TTS URL 'https://192.168.1.51:19000/api/tts?speaker=CLB&text=ok'
I (15:08:28.457) WILLOW: WIS TTS playback finished
I (15:08:28.458) WILLOW: HTTP POST response body:
{
        "response":     {
                "speech":       {
                        "plain":        {
                                "speech":       "The basement temperature is 69 °F",
                                "extra_data":   null
                        }
                },
                "card": {
                },
                "language":     "en",
                "response_type":        "action_done",
                "data": {
                        "targets":      [],
                        "success":      [],
                        "failed":       []
                }
        },
        "conversation_id":      null
}
I (15:08:38.486) WILLOW: Wake LCD timeout, turning off LCD
I (15:08:41.555) WILLOW: AUDIO_REC_VAD_END
I (15:08:41.556) WILLOW: AUDIO_REC_WAKEUP_END
I (15:09:11.223) WILLOW: Wake LCD timeout, turning off LCD

[nikito]

The I2S error is actually normal. This part is the odd thing:
I (15:08:27.110) WILLOW: Using WIS TTS URL 'https://192.168.1.51:19000/api/tts?speaker=CLB&text=ok'

In the tts url the text it is sending to process is "ok", not the speech reply text from HA. Maybe this is caused by the HA version?

[mikey60]

That could be. Version 2023.4.x does not have the Wyoming integration. What version of HA are you using? I was waiting until HA version 2023.5 or 2023.6 supports manually installed Google Assistant before I upgrade. See issue (home-assistant/core#92445)

[nikito]

Using the very latest one. 🙂

[mikey60]

Thanks for all your help!
UPDATE: I updated to Home Assistant 2023.7.2 and I was still not displaying or hearing the response from Home Assistant. So I did a git pull and re-build and re-flash of Willow on the ESP32 Box and now everything is working. Thanks again!

[tombloemker]

I sure would love a quick overview of the process exactly what must be done to Home Assistant.
I connected to my ESP32-Box with termite: WILLOW: home assistant response_type: error[1B][0m
Which I would expect since I've done nothing to home assistant, has someone made a detailed instruction to configure for Willow? If so, I can't find it. :(

From read.me "configure your server" I assume this is accomplished in the willow config to point to my HA device?
We recommend testing both your Home Assistant server address and token before flashing. How do I do this?
Where do I check HA for TSL?

[nikito]

There's instructions about it in the readme here: https://github.com/toverainc/willow#home-assistant 🙂 you just need to point willow to your home assistant server, and put the access token in the config, then it will work with HA. When WAS is released with the 1.0 launch it will make this process even easier! 😀