fix(asr): count VAD false alarms outside the reference extent by Kayvan-Zahiri · Pull Request #16380 · NVIDIA-NeMo/Speech · GitHub
Skip to content

fix(asr): count VAD false alarms outside the reference extent - #16380

Open
Kayvan-Zahiri wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Kayvan-Zahiri:fix/vad-deter-uem-span
Open

Kayvan-Zahiri wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Kayvan-Zahiri:fix/vad-deter-uem-span

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown
Contributor

What does this PR do ?

_DetectionErrorRateAccumulator (used by vad_tune_threshold_on_dev and frame_vad_eval_detection_error) replaced pyannote's DetectionErrorRate in #15632, but it undercounts false alarms. With no UEM, md-eval scores only the reference's extent, so speech predicted before the first or after the last reference segment is never counted. pyannote scored the union of the reference and hypothesis extents.

Example: reference [2, 4], hypothesis [0, 1] + [2, 4] + [5, 6]. pyannote gives FA = 2.0 s (DetER 100%). main gives FA = 0 (DetER 0%).

I compared both on 1,600 random reference/hypothesis pairs, with and without overlapping segments. main disagreed with pyannote on false alarm in 1,066 of them. With this change, every pair whose reference has speech matches.

Collection: ASR

Changelog

  • When no UEM is given, score from the earliest start to the latest end across reference and hypothesis.
  • Add test_detection_error_rate_counts_false_alarm_outside_reference. It fails on main and passes here.

One difference remains, and it's out of scope here: a file with no reference speech is skipped by evaluate(), so its false alarms still don't count.

Checks run

  • pytest --cpu -m "not pleasefixme" tests/collections/asr/utils/test_vad_utils_asr.py tests/collections/speaker_tasks/utils/: 700 passed, 37 skipped
  • pre-commit run on the changed files

No documentation change needed.

PR Type:

  • Bugfix

Signed-off-by: Kayvan Zahiri <kayvanandre@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

@github-actions github-actions Bot added the ASR label Oct 14, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ASR community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants