Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add force activate button to wyoming satellite #114291

Closed

Conversation

chatziko
Copy link
Contributor

@chatziko chatziko commented Mar 27, 2024

Proposed change

This PR adds a server-side "force activate" button to wyoming satellite that causes the satellite to go directly to ASR (a sort of "push to talk" button). This allows to activate the satellite via any automation.

Example use-case: when my TV is on wake-word detection is very unreliable. At the same time I have a remote control in my hand. So I setup an automation that mutes the satellite when the TV is on to avoid false detections, and I configure a button of the TV remote to run another automation that lowers the TV volume, activates the satellite, and raises the volume afterwards. So I can just press the button and give whatever command I want while watching TV.

I considered various ways to implement this, here are some design considerations:

  • It should work when the satellite is either muted (i.e. push-to-talk only) or unmuted (normal satellite with ability to force activate).
  • The code should handle the various statellite states as uniformly as possible, without creating a dozen different event flows and with minimal changes to the satellite.
  • If vad/wake is active on the satellite we might not have an active pipeline even if the satellite is unmuted.
  • run-pipeline is normally initiated by the satellite, not the server

So I ended up with the following design (open to discussion, of course):

  • The user presses the "force activate" button
  • If the server has a currently running pipeline, it closes it.
  • Then run-satellite is sent to the satellite, with start_stage == asr. This event was already sent to unpause the satellite, only start_stage is added.
  • When the satellite receives run-satellite with start_stage == asr it unpauses as usual, but goes directly to streaming.
  • run-pipeline is sent from the satellite as usual, but with start_stage == asr.
  • After the force activated pipeline is over:
    • the satellite will act as usual (going to vad / wake / ...), no change is needed.
    • if is_muted == True, the server will send pause-satellite to re-pause the satellite.

Overall this design covers the requirements with relatively small number of changes in both the server and client.

Some more notes:

  • I used a button which is a bit more user-friendly, but clearly we can use a service instead.
  • Concerning terminology, I used "force activate" in the code, since "active" was already used to indicate that the satellite is past the wake stage. For the user the button is simply labelled "Activate". Let me know if there are better terms to use.
  • I didn't add any tests yet, I wanted to confirm the design first.
  • Same for documentation, I can add it once we decide to move forward.
  • The PR depends on this wyoming PR and this wyoming-satellite PR .

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@home-assistant
Copy link

Hey there @balloob, @synesthesiam, mind taking a look at this pull request as it has been labeled with an integration (wyoming) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of wyoming can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign wyoming Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

Copy link

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
If you are the author of this PR, please leave a comment if you want to keep it open. Also, please rebase your PR onto the latest dev branch to ensure that it's up to date with the latest changes.
Thank you for your contribution!

@github-actions github-actions bot added the stale label May 28, 2024
@chatziko
Copy link
Contributor Author

Ping to keep open

@github-actions github-actions bot removed the stale label May 29, 2024
@synesthesiam synesthesiam marked this pull request as draft June 7, 2024 17:37
@synesthesiam
Copy link
Contributor

Let's keep this open for now. I am converting to a draft to indicate we're still discussing how we want to implement this feature. It may be a button, or likely a service call on assist_pipeline.

@chatziko chatziko force-pushed the wyoming-satellite-force-activate branch from ac36b79 to 59f2b30 Compare June 10, 2024 11:13
@chatziko chatziko force-pushed the wyoming-satellite-force-activate branch from 59f2b30 to 045d68d Compare June 10, 2024 11:20
@chatziko
Copy link
Contributor Author

Glad to see that the feature is "in the pipeline" 😄

In the meanwhile I rebased on dev to keep it fresh. Existing tests still pass (once the latest wyoming from git is installed).

@Jyumpp
Copy link

Jyumpp commented Jun 12, 2024

I was looking into this today for a project with our lab (UGA Innovation Factory) for a project using a converted wall telephone and was very pleased to see that work for this is already so far progressed. I agree fully with @synesthesiam that it should likely be implemented as a service call rather than a button. @chatziko seemed to have pretty good implementation for the devices.py, and satellite.py seems to operate mostly fine as is, but I'm dubious as to whether bypassing mute is the right option. I can see a scenario where it would be better to ignore a force-pipeline-start request when the device is muted, and should likely throw a service call error. I'd be happy to implement that as well if that route seems more secure, but could be a non-issue since the Wyoming Protocol is unencrypted as is.

I cherry-picked @chatziko 's commit over the latest dev branch and modified what would be required to remove the button but reimplement as a service call through assist_pipeline. I used a new event listener (modified from the button's call implementation) and modified assist_pipeline to accept a new service call and write an event to the bus for Wyoming Satellite to listen for and react accordingly. The service call takes the following format:

service: assist_pipeline.force_activate
target: 
  device_id: [device id of the Wyoming satellite]

I've attached a diff below for the changes I made versus the commit active for this draft.
TLDR:

  • Wyoming Integration:
    • Remove button.py
    • Adapt force_active_listener to function as an Event callable rather than a button callback
  • Assist Pipeline Integration:
    • Add services.yaml and icons.json for the service
    • Adapt const.py to include identifiers for the event and service
    • Add English translation to strings.json
    • Add force_activate as a service call
    • Add force_start_pipeline as event to trigger force_active_listener in the Wyoming integration

index f481411e55..2a3f8496dd 100644
--- a/homeassistant/components/assist_pipeline/__init__.py
+++ b/homeassistant/components/assist_pipeline/__init__.py
@@ -7,7 +7,8 @@ from collections.abc import AsyncIterable
 import voluptuous as vol
 
 from homeassistant.components import stt
-from homeassistant.core import Context, HomeAssistant
+from homeassistant.const import CONF_DEVICE_ID
+from homeassistant.core import Context, HomeAssistant, ServiceCall
 from homeassistant.helpers.typing import ConfigType
 
 from .const import (
@@ -16,6 +17,8 @@ from .const import (
     DATA_LAST_WAKE_UP,
     DOMAIN,
     EVENT_RECORDING,
+    OVERRIDE_START_EVENT_TYPE,
+    SERVICE_FORCE_ACTIVATE,
 )
 from .error import PipelineNotFound
 from .pipeline import (
@@ -53,6 +56,7 @@ __all__ = (
     "PipelineNotFound",
     "WakeWordSettings",
     "EVENT_RECORDING",
+    "OVERRIDE_START_EVENT_TYPE",
 )
 
 CONFIG_SCHEMA = vol.Schema(
@@ -74,6 +78,13 @@ async def async_setup(hass: HomeAssistant, config: ConfigType) -> bool:
     # wake_word_id -> timestamp of last detection (monotonic_ns)
     hass.data[DATA_LAST_WAKE_UP] = {}
 
+    def force_activate(call: ServiceCall) -> None:
+        """Handle force_activate service call."""
+        device_id = call.data.get(CONF_DEVICE_ID)
+        hass.bus.async_fire(OVERRIDE_START_EVENT_TYPE, {CONF_DEVICE_ID: device_id})
+
+    hass.services.async_register(DOMAIN, SERVICE_FORCE_ACTIVATE, force_activate)
+
     await async_setup_pipeline_store(hass)
     await async_run_migrations(hass)
     async_register_websocket_api(hass)
diff --git a/homeassistant/components/assist_pipeline/const.py b/homeassistant/components/assist_pipeline/const.py
index 36b72dad69..74dd4e9881 100644
--- a/homeassistant/components/assist_pipeline/const.py
+++ b/homeassistant/components/assist_pipeline/const.py
@@ -9,6 +9,9 @@ DEFAULT_PIPELINE_TIMEOUT = 60 * 5  # seconds
 
 DEFAULT_WAKE_WORD_TIMEOUT = 3  # seconds
 
+OVERRIDE_START_EVENT_TYPE = "force_start_pipeline"
+SERVICE_FORCE_ACTIVATE = "force_activate"
+
 CONF_DEBUG_RECORDING_DIR = "debug_recording_dir"
 
 DATA_LAST_WAKE_UP = f"{DOMAIN}.last_wake_up"
diff --git a/homeassistant/components/assist_pipeline/icons.json b/homeassistant/components/assist_pipeline/icons.json
new file mode 100644
index 0000000000..4b433d30d8
--- /dev/null
+++ b/homeassistant/components/assist_pipeline/icons.json
@@ -0,0 +1,5 @@
+{
+  "services": {
+    "force_activate": "mdi:phone"
+  }
+}
diff --git a/homeassistant/components/assist_pipeline/services.yaml b/homeassistant/components/assist_pipeline/services.yaml
new file mode 100644
index 0000000000..97e0519826
--- /dev/null
+++ b/homeassistant/components/assist_pipeline/services.yaml
@@ -0,0 +1,5 @@
+# Services relating to the assist pipeline
+
+force_activate:
+  target:
+    device:
diff --git a/homeassistant/components/assist_pipeline/strings.json b/homeassistant/components/assist_pipeline/strings.json
index 8fa67879fc..6cb5cbf2b4 100644
--- a/homeassistant/components/assist_pipeline/strings.json
+++ b/homeassistant/components/assist_pipeline/strings.json
@@ -21,5 +21,11 @@
         }
       }
     }
+  },
+  "services": {
+    "force_activate": {
+      "name": "Force Activate",
+      "description": "Force the device to activate the assistant pipeline"
+    }
   }
 }
diff --git a/homeassistant/components/wyoming/__init__.py b/homeassistant/components/wyoming/__init__.py
index 0d72c20233..9d62bdd9ed 100644
--- a/homeassistant/components/wyoming/__init__.py
+++ b/homeassistant/components/wyoming/__init__.py
@@ -20,7 +20,6 @@ _LOGGER = logging.getLogger(__name__)
 
 SATELLITE_PLATFORMS = [
     Platform.BINARY_SENSOR,
-    Platform.BUTTON,
     Platform.SELECT,
     Platform.SWITCH,
     Platform.NUMBER,
@@ -50,6 +49,10 @@ async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool:
     if (satellite_info := service.info.satellite) is not None:
         # Create satellite device, etc.
         item.satellite = _make_satellite(hass, entry, service)
+        hass.bus.async_listen(
+            hass.components.assist_pipeline.OVERRIDE_START_EVENT_TYPE,
+            item.satellite.device.force_activate,
+        )
 
         # Set up satellite sensors, switches, etc.
         await hass.config_entries.async_forward_entry_setups(entry, SATELLITE_PLATFORMS)
diff --git a/homeassistant/components/wyoming/button.py b/homeassistant/components/wyoming/button.py
deleted file mode 100644
index dd125097f1..0000000000
--- a/homeassistant/components/wyoming/button.py
+++ /dev/null
@@ -1,43 +0,0 @@
-"""Wyoming button entities."""
-
-from __future__ import annotations
-
-from typing import TYPE_CHECKING, Any
-
-from homeassistant.components.button import ButtonEntity, ButtonEntityDescription
-from homeassistant.config_entries import ConfigEntry
-from homeassistant.core import HomeAssistant
-from homeassistant.helpers.entity_platform import AddEntitiesCallback
-
-from .const import DOMAIN
-from .entity import WyomingSatelliteEntity
-
-if TYPE_CHECKING:
-    from .models import DomainDataItem
-
-
-async def async_setup_entry(
-    hass: HomeAssistant,
-    config_entry: ConfigEntry,
-    async_add_entities: AddEntitiesCallback,
-) -> None:
-    """Set up button entities."""
-    item: DomainDataItem = hass.data[DOMAIN][config_entry.entry_id]
-
-    # Setup is only forwarded for satellites
-    assert item.satellite is not None
-
-    async_add_entities([WyomingSatelliteForceActivate(item.satellite.device)])
-
-
-class WyomingSatelliteForceActivate(WyomingSatelliteEntity, ButtonEntity):
-    """Manually activate the satellite instead of using a wake word."""
-
-    entity_description = ButtonEntityDescription(
-        key="activate",
-        translation_key="activat",
-    )
-
-    async def async_press(self, **kwargs: Any) -> None:
-        """Turn on."""
-        self._device.force_activate()
diff --git a/homeassistant/components/wyoming/devices.py b/homeassistant/components/wyoming/devices.py
index 44f9c77fa8..395bcb98f8 100644
--- a/homeassistant/components/wyoming/devices.py
+++ b/homeassistant/components/wyoming/devices.py
@@ -2,10 +2,10 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable
+from collections.abc import Callable, Coroutine
 from dataclasses import dataclass
 
-from homeassistant.core import HomeAssistant, callback
+from homeassistant.core import Event, HomeAssistant, callback
 from homeassistant.helpers import entity_registry as er
 
 from .const import DOMAIN
@@ -26,7 +26,7 @@ class SatelliteDevice:
 
     _is_active_listener: Callable[[], None] | None = None
     _is_muted_listener: Callable[[], None] | None = None
-    _force_activate_listener: Callable[[], None] | None = None
+    _force_activate_listener: Callable[[Event], Coroutine] | None = None
     _pipeline_listener: Callable[[], None] | None = None
     _audio_settings_listener: Callable[[], None] | None = None
 
@@ -47,10 +47,11 @@ class SatelliteDevice:
                 self._is_muted_listener()
 
     @callback
-    def force_activate(self) -> None:
+    def force_activate(self, event: Event) -> Coroutine | None:
         """Request to activate without a wake word."""
         if self._force_activate_listener is not None:
-            self._force_activate_listener()
+            return self._force_activate_listener(event)
+        return None
 
     @callback
     def set_pipeline_name(self, pipeline_name: str) -> None:
@@ -96,7 +97,8 @@ class SatelliteDevice:
 
     @callback
     def set_force_activate_listener(
-        self, force_activate_listener: Callable[[], None]
+        self,
+        force_activate_listener: Callable,
     ) -> None:
         """Listen for force activate requests."""
         self._force_activate_listener = force_activate_listener
diff --git a/homeassistant/components/wyoming/strings.json b/homeassistant/components/wyoming/strings.json
index 464a863d7b..f2768e45eb 100644
--- a/homeassistant/components/wyoming/strings.json
+++ b/homeassistant/components/wyoming/strings.json
@@ -53,11 +53,6 @@
         "name": "Mute"
       }
     },
-    "button": {
-      "activat": {
-        "name": "Activate"
-      }
-    },
     "number": {
       "auto_gain": {
         "name": "Auto gain"

@balloob
Copy link
Member

balloob commented Jun 13, 2024

@Jyumpp can you open a PR and add a link in the description to this PR? That way, we can discuss your specific changes there. Thanks!

Copy link

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
If you are the author of this PR, please leave a comment if you want to keep it open. Also, please rebase your PR onto the latest dev branch to ensure that it's up to date with the latest changes.
Thank you for your contribution!

@github-actions github-actions bot added the stale label Aug 12, 2024
@github-actions github-actions bot closed this Aug 19, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants