Assist Speaker ID

Recognize household members by voice for Assist in Home Assistant. The assistant knows who is asking and answers that person. Everything runs locally.

Polski · English · GitHub

Language: speaker recognition itself works for any language. The spoken prompts of guided enrollment come in Polish and English (following the HA language); the "Who is speaking?" and announcement keywords live in Assist Router, which currently understands Polish.

1. What it is and why

Assist turns speech into text, but it does not know who is speaking. Voice satellites (Home Assistant Voice PE, ESPHome) send the command without a user, so to Home Assistant every sentence is anonymous. This becomes a problem with first-person questions:

• "What classes do I have today?" – whose timetable should it read?
• "Where is my phone?" – which phone should it ring?
• "What's on my plate today?" – whose calendar should it read?
• "Good night" – said differently to a child than to an adult.

Whisper and other speech-to-text models do not recognize the speaker. Diarization (e.g. WhisperX) only answers "who spoke when" with anonymous labels, not "which household member is this". Assist Speaker ID adds the missing piece: it turns a voice into a "fingerprint" (a vector of numbers), compares it with the stored voiceprints of household members and passes the result on – to the conversation agent, to an LLM and to automations.

The project has two parts in one repository:

It gives the most when paired with Assist Router: it uses the speaker for first-person questions, the speaker's room, family terms ("my brother"), answers to „Kto to mówi?” (“Who is speaking?”) and announcements. Assist Speaker ID works with any agent, though.

2. How it works

1. The satellite detects the wake word and sends the recorded command to the Assist pipeline.
2. The pipeline uses the "Speech to text with speaker" entity instead of plain Whisper. The entity talks to the service over the Wyoming protocol, just like the built-in Wyoming integration.
3. The wyoming-speaker-id service forwards the audio to your speech-to-text server unchanged and keeps a copy for itself. When the recording ends, it computes the voice fingerprint while Whisper is still transcribing anyway.
4. The service returns the same text as Whisper, with extra fields: speaker (person id or unknown), speaker_name, speaker_score, speaker_scores (scores of all people), speech_seconds.
5. The integration remembers the speaker for the current pipeline run. HA calls speech-to-text and the agent in the same task, so the agent gets the speaker of exactly this sentence, not of a sentence spoken in another room at the same moment.
6. The speaker can then be used by the wrapper agent, by Assist Router or another agent via get_speaker(), and by automations via the sensor and the event.

Why a proxy and not a separate model alongside? The Wyoming client in HA reads only the text from the result, and the speech-to-text entity does not know which satellite the recording came from. So the service adds the speaker to the transcription result, and the integration has its own client that reads these fields. A regular Wyoming client ignores the extra fields, so the service can safely replace Whisper: in the worst case it behaves like plain Whisper.

3. Recognition under the hood

• Speech detection: a simple energy detector cuts out silence. Only actual speech counts, at most the first 8 seconds.
• Voice fingerprint: the 3D-Speaker CAM++ model (zh_en "advanced" version, Apache-2.0 license) turns speech into a vector of 192 numbers. It describes the timbre of the voice: pitch, resonance, articulation. It does not depend on what is said, so recognition works for any command.
• A person's voiceprint is the average of the vectors of their samples. More samples from different situations give a more stable voiceprint.
• Comparison uses cosine similarity (the angle between vectors), usually from 0 to 1. The decision is described in Scores, thresholds and calibration.
• No added latency: the fingerprint is computed in parallel with transcription. Measured on a Jetson AGX Orin (CPU, 2 threads): median of the whole command unchanged (1329 → 1330 ms), 90th percentile +95 ms.

Choosing the model

Before choosing, we compared several models available in sherpa-onnx. Test sets: 6 Polish Piper voices × 12 sentences, and recordings of 3 people from the sherpa-onnx examples. Accuracy was measured with leave-one-out. EER (equal error rate) is the point where false accepts equal false rejects – lower is better.

Through the whole chain (Piper → wyoming-speaker-id → Whisper), every utterance with at least 1 s of speech was assigned to the right person, with scores of 0.76–0.92. A different model can be set with the model option (a sherpa-onnx model name or a path to an .onnx file); after changing the model, voiceprints must be recorded again.

4. Requirements

• Home Assistant with Assist and at least one voice satellite.
• A speech-to-text server speaking the Wyoming protocol: the Whisper add-on or your own faster-whisper (e.g. on a computer with a GPU). Assist Speaker ID does not replace it, it sits in front of it.
• For the service: an amd64 or aarch64 CPU (e.g. Raspberry Pi 4/5, NUC, Jetson). The model takes about 30 MB and runs on the CPU; no GPU needed.
• HACS to install the integration.

Proven in practice: the author runs the service daily on an NVIDIA Jetson AGX Orin 64 GB, as a container next to faster-whisper (Whisper on the GPU, voice model on the CPU), for the whole family and 7 satellites. It works very well: recognition takes about 90 ms and runs while transcription is in progress, so commands are no slower than with Whisper alone.

5. Installation

Step 1a: the service as an HA add-on

The simplest route when Home Assistant runs as HA OS or Supervised.

1. Settings → Add-ons → Add-on store → ⋮ → Repositories, and add https://github.com/witold-gren/home-assistant-assist-speaker-id.
2. Install Assist Speaker ID.
3. On the Configuration tab set upstream (the speech-to-text server): tcp://core-whisper:10300 for the Whisper add-on, or the address of your own server, e.g. tcp://192.168.1.10:10300.
4. Start the add-on. On first start it downloads the model (about 30 MB). It then announces itself to Home Assistant with a random API token, and HA offers the integration (step 2).

Step 1b: the service as a container (e.g. next to Whisper on another computer)

When Whisper runs on a separate machine, it is best to run the service next to it. The audio then does not cross the network twice.

git clone https://github.com/witold-gren/home-assistant-assist-speaker-id
docker build -t wyoming-speaker-id home-assistant-assist-speaker-id/speaker_id/
docker run -d --name speaker-id --restart unless-stopped \
  -p 10350:10350 -p 10351:10351 -v speaker-id-data:/data \
  -e SPEAKER_ID_UPSTREAM=tcp://192.168.1.10:10300 \
  -e SPEAKER_ID_API_TOKEN=dlugi-losowy-token \
  wyoming-speaker-id

The same container in docker-compose.yaml, next to Whisper:

services:
  whisper:
    image: rhasspy/wyoming-whisper
    # … your existing configuration
  speaker-id:
    build: ./home-assistant-assist-speaker-id/speaker_id
    restart: unless-stopped
    ports: ["10350:10350", "10351:10351"]
    volumes: ["./speaker-id/data:/data"]
    environment:
      SPEAKER_ID_UPSTREAM: tcp://whisper:10300
      SPEAKER_ID_API_TOKEN: dlugi-losowy-token

Port 10350 is Wyoming (speech-to-text with speaker), port 10351 is the HTTP API (voiceprints, calibration, last recording).

Step 2: the integration

1. HACS → ⋮ → Custom repositories → add the repository URL, type Integration. Install Assist Speaker ID and restart HA.
2. With the add-on: HA shows the discovered integration in Settings → Devices & services. Click Configure.
3. With a container: Settings → Devices & services → Add integration → Assist Speaker ID and enter the host, Wyoming port (10350), API port (10351) and token.

The integration creates an "Assist Speaker ID" device with the speech-to-text entity "Speech to text with speaker", the agent "Agent with speaker" and the sensor "Last speaker".

The integration options (Configure) are a menu with two entries:

• Agent and people: mapping each voice to an HA person (person.*), the target agent for "Agent with speaker", the prompt line (default The person speaking is {name}.) and whether to run commands as the speaker's HA user (context.user_id).
• Recognition thresholds: shared threshold, margin, minimum speech and each person's threshold. Saved values go to the service and apply from the next sentence, without a restart.

6. Pipeline setup

Settings → Voice assistants → pick the pipeline (assistant) of each satellite:

1. Speech-to-text: "Speech to text with speaker". Set the language to what you used in Whisper so far.
2. Conversation agent:
   • Assist Router or another agent that reads the speaker itself, or
   • "Agent with speaker", which adds the speaker for any agent (chosen in the integration options).
3. Turn off "Prefer handling commands locally".

Why turn off local handling? When it is on, HA tries its own intents before asking the agent. The built-in broadcast intent would then catch „Ogłoszenie, …” (“Announcement, …”) and read the text with speech synthesis instead of playing your recording, and during enrollment the agent would not get to reply with a short "Recorded 3 of 10". Assist Router and "Agent with speaker" pass everything HA understands back to HA anyway, so regular commands keep working.

Switch satellites one at a time. If something does not work, just switch the pipeline back to plain Whisper.

Check: Settings → Voice assistants → pipeline → ⋮ → Debug. After a command from a satellite, the "Last speaker" sensor shows unknown (no voiceprints yet), and the speech_seconds attribute shows the length of speech. That means the audio is passing through the service.

7. Recording voiceprints

Each household member records 10–15 ordinary sentences. The easiest way is guided enrollment: the satellite asks for each next sentence itself, no wake word needed.

1. Developer tools → Actions → speaker_id.start_enrollment:

   action: speaker_id.start_enrollment
   data:
     speaker: anna            # person id: lowercase, no spaces
     name: Anna               # name that is spoken and displayed
     count: 10
     satellite: assist_satellite.salon   # the satellite Anna is standing next to

2. The satellite says: "Recording a voiceprint for Anna. Say a normal command, for example: what is the weather tomorrow." and listens.
3. Anna says any sentence, e.g. "What will the weather be like in Kraków tomorrow?". The command is not executed; the satellite immediately replies "Recorded 1 of 10. Next sentence." and keeps listening.
4. A sentence that is too short (under 1.5 s of speech) does not count – the satellite says "Too short. Please say a longer sentence.". After silence or a cut-off conversation it asks again ("I did not hear anything. Please say a sentence."), and after 3 attempts without a new sample it stops.
5. After the last sample: "Done, recorded 10 voice samples for Anna.". The voiceprint works right away.

Enrollment also works without the satellite field, but then each sentence starts with the wake word. Progress is shown in the enrollment attribute of the "Last speaker" sensor. To cancel: speaker_id.stop_enrollment.

Enrollment went wrong? Add more samples with the same action (new ones are added to the old ones) or start over: speaker_id.delete_speaker and enroll again. You can add a single sentence afterwards with speaker_id.add_last_sample: the last utterance (at most 10 min old) becomes a sample of the given person.

Good samples: record through the satellite and from the distance the person usually speaks from – the microphone and the room change the voice more than the words do. Use ordinary commands of a few words, in a normal tone (shouting, whispering and laughing lower the scores). Enroll everyone in the household before you calibrate. Children's voices change as they grow, so add new samples for them every few months.

8. Scores, thresholds and calibration

Similarity

Each utterance is compared with each person's voiceprint. The score is the similarity, usually from 0 to 1 (given as a percentage in the answer to „Kto to mówi?”):

This is not a measure of how "unique" a voice is. A high score only means that this utterance is very similar to this voiceprint.

Threshold and margin

The best-matching person is recognized only if both conditions hold:

1. similarity ≥ that person's threshold (their own, or the shared one if they have none);
2. similarity − runner-up's score ≥ margin. With two nearly equal scores it is better to say "I don't know" than to guess.

Otherwise the speaker is "unknown" (unknown), and Assist behaves as if there were no recognition. A wrong guess is worse than no recognition, so the system is cautious by default.

• A higher threshold is stricter: fewer mistakes, but "unknown" more often.
• A lower threshold is more lenient: "unknown" less often, but mistakes are more likely.
• Similar voices (siblings) need a higher threshold or a larger margin. Not because their voiceprint is weaker, but because other voices come close to it – the bar has to be higher so a sibling does not clear it.

Values from the integration options are stored in the service (speakers.json) and take precedence over add-on options and environment variables, which are only initial values.

Calibration

The speaker_id.calibrate action compares each person's samples with their voiceprint and with the other people's samples, then suggests a threshold. With apply: true it saves the per-person thresholds right away.

Example (Tomek, adult):

samples: 10
own_min: 0.709      # worst own sample: 71% similar to the voiceprint
own_p10: 0.778      # 9 out of 10 own samples have at least 78%
own_mean: 0.803     # 80% on average
others_max: 0.305   # most similar sample of someone else: only 31%
separable: true     # 0.709 > 0.305 - voices clearly separated
suggested_threshold: 0.541   # (0.778 + 0.305) / 2

There is a 40-point gap between the worst own sample (71%) and the closest one from someone else (31%). Any threshold between roughly 0.35 and 0.70 will work, and the suggested 0.54 sits in the middle with headroom on both sides. For a child you might get e.g. own_p10: 0.74 and others_max: 0.68 (a sibling): a threshold of 0.71 and only a 6-point gap. Then it is worth raising the margin as well.

Similar voices (e.g. siblings)

1. Record more samples: 15–20 per person, from the usual satellite and distance.
2. Calibrate only after everyone is enrolled: others_max is computed only from people who have voiceprints.
3. With separable: false the culprit is often a single bad sample (someone talked over it, noise). Add samples or re-enroll that person.
4. Set the threshold between others_max and own_p10: closer to own_p10 means fewer mistakes, closer to others_max – fewer "unknowns".
5. Margin 0.08–0.15: the assistant will say it is not sure instead of guessing.
6. Test „Kto to mówi?” on a few longer sentences from each person and look at the runner-up's score – the gap between them is your real headroom.

Calibration with apply: true overwrites per-person thresholds. Make manual tweaks after it.

9. Everyday use

„Kto to mówi?” (“Who is speaking?”)

With Assist Router, ask „Kto to mówi?”, „Kim jestem?” (“Who am I?”), „Czy mnie rozpoznajesz?” (“Do you recognize me?”) or „Czyj to głos?” (“Whose voice is this?”). The answer gives the person and the similarity, e.g. „Mówi Anna, podobieństwo 82%. Następna osoba: Kasia, 41%.” (“Anna is speaking, similarity 82%. Next person: Kasia, 41%.”). It is the quickest test of thresholds and voiceprint quality.

"Agent with speaker": the speaker for any agent

Set it in the pipeline and choose the target agent in the integration options (e.g. Ollama, OpenAI, HA's built-in agent). Before passing the command on, "Agent with speaker":

• adds a line with the speaker to the system prompt (default "The person speaking is Anna."), so an LLM agent can answer "Anna, today you have…";
• optionally sets context.user_id to the speaker's HA user, so the history shows who switched the light;
• handles enrollment: while recording it does not execute commands and replies immediately.

Your own agent or integration: get_speaker()

try:
    from custom_components.speaker_id.api import get_speaker
except ImportError:
    get_speaker = None

speaker = get_speaker(hass) if get_speaker else None
if speaker and speaker.speaker:
    ...  # speaker.speaker (id), .name, .person, .user_id, .score

An agent that handles enrollment itself calls enrollment_reply(hass) first: it returns a short reply ("Recorded 3 of 10. Next sentence.") and whether to keep listening, or None for a regular utterance. Assist Router does this: a first-person question without a name ("what classes do I have") refers to the speaker's data, "here" is the speaker's room, and "my brother" is a specific family member.

Automations

The "Last speaker" sensor (state: name or unknown; attributes: speaker, name, person, score, scores, speech_seconds, enrollment) and the speaker_id_identified event. The utterance text is deliberately kept out of the history.

HA runs automations with a sentence trigger before the agent, so they do not get the speaker directly. The sensor is updated in the same command, just before them, though – a result from a few seconds ago is the speaker of this sentence. A handy macro in /config/custom_templates/mowca.jinja (mowca is Polish for "speaker"; you can rename it):

{% macro mowca(max_s=20) -%}
  {%- set s = states.sensor.assist_speaker_id_last_speaker -%}
  {%- if s and (now() - s.last_updated).total_seconds() <= max_s -%}
    {{- s.attributes.speaker or '' -}}
  {%- endif -%}
{%- endmacro %}

After creating the file, run the homeassistant.reload_custom_templates action. Example: "Where is my phone?" rings the phone of the person asking:

triggers:
  - trigger: conversation
    command: "where is my phone"
actions:
  - variables:
      who: "{% from 'mowca.jinja' import mowca %}{{ mowca() }}"
  - if: "{{ who in ['anna', 'tomek'] }}"
    then:
      - action: script.ring_phone     # your script, e.g. ringing via the HA app
        data: { person: "{{ who }}" }
      - set_conversation_response: "Ringing your phone."
    else:
      - set_conversation_response: "I don't know who is asking. Whose phone should I find?"

The sensor id depends on the HA language; in Polish it is sensor.assist_speaker_id_ostatni_mowca.

10. Announcements in your own voice

„OK Nabu, ogłoszenie, dzieci, proszę przyjść na obiad.” (“OK Nabu, announcement, kids, please come to dinner.”) The other speakers play your recording – exactly what you said after the wake word – instead of reading the text with speech synthesis. You can hear who is calling and in what tone.

• The recording is the last utterance from the service, kept only in RAM.
• HA serves it to the speakers at a one-time signed URL /api/speaker_id/announcement/<token>, valid for 10 minutes. Nothing is written to disk.
• Assist Router recognizes „ogłoszenie” (“announcement”), „komunikat” (“message”), „do wszystkich” (“to everyone”) and „intercom” at the start of a sentence. The speaker that recorded it only replies „Ogłoszenie wysłane.” (“Announcement sent.”).
• Without Assist Router, use the speaker_id.announce_last action in an automation:

A conversation agent can do the same with async_announce_last(hass, exclude=…, media_players=…) from custom_components.speaker_id.api. Remember to turn off local handling in the pipeline (section 6).

11. Actions and API

The service also has an HTTP API (port 10351, header Authorization: Bearer <token>), useful for scripts and tests:

12. Limitations

• Short commands ("Yes", "Stop", under 1 s of speech) give "unknown".
• The wake word is not used: the satellite detects it itself and sends only the command. Recognition does not depend on the words, but it needs at least a second of speech.
• Typed text in the app has no speaker.
• Several people at once: when two people speak at the same time, the result refers to the dominant voice or comes out "unknown".
• Spoken prompts (e.g. during enrollment) are in Polish and English, and Assist Router keywords are in Polish; the recognition method itself works for any language.

This is a convenience, not security. A voice can be recorded and replayed, and siblings can sound very much alike. Do not use the speaker to unlock doors, disarm an alarm or anything else that requires authentication.

13. Privacy

A voiceprint is biometric data (GDPR, Article 9). Therefore:

• everything runs locally, with no cloud;
• the service stores only vectors (/data/speakers.json), no recordings (a vector describes the timbre of a voice, but is not a recording);
• the last utterance is kept only in memory, for at most 10 minutes (for add_last_sample and announcements);
• the utterance text does not go into the sensor history;
• delete_speaker removes a person's voiceprint completely;
• household members – and parents, for children – should know about the recognition and agree to it.