Support for realtime audio input

Noting that the processing time is considerably shorter than the length of speech, is it possible to feed the models real time microphone output? Or does the inference run on the complete audio stream, instead of sample by sample?

This would greatly reduce the latency for voice assistants and the like, that the audio does not need to be fully captured and only after that fed to the models. Basically the same as I did here with SODA: https://github.com/biemster/gasr, but then with an open source and multilang model.

The Whisper model processes the audio in chunks of 30 seconds - this is a hard constraint of the architecture.

However, what seems to work is you can take for example 5 seconds of audio and pad it with 25 seconds of silence. This way you can process shorter chunks.

Given that, an obvious strategy for realtime audio transcription is the following:

T  - [data]
-------------------
1  - [1 audio, 29 silence pad] -> transcribe -> "He"
2  - [2 audio, 28 silence pad] -> transcribe -> "Hello"
3  - [3 audio, 27 silence pad] -> transcribe -> "Hello, my"
...
29 - [29 audio, 1 silence pad] -> transcribe -> "Hello, my name is John ..."

The problem with that is you need to do the same amount of computation for 1 second audio as you would do for 2, 3, ... , 30 seconds of audio. So if your audio input step is 1 second (as shown in the example above), you will effectively do 30 times the computation that you would normally do to process the full 30 seconds.

I plan to add a basic example of real-time audio transcription using the above strategy.

I was a bit afraid that that would be the answer, but I'll definitely will check out that basic example when it's ready!

ref #10 : quick-and-dirty attempt for real-time audio transciption

Just added a very naive implementation of the idea above. To run it, simply do:

# install sdl2 (Ubuntu)
$ sudo apt-get install libsdl2-dev

# install sdl2 (Mac OS)
$ brew install sdl2

# download a model if you don't have one
$ ./download-ggml-model.sh base.en

# run the real-time audio transcription
$ make stream
$ ./stream -m models/ggml-base.en.bin

This example continuously captures audio from the mic and runs whisper on the captured audio.
The time step is currently hardcoded at 3 seconds.

The results are not great because the current implementation can chop the audio in the middle of words.
Also, the text context is reset for every new iteration.

However, all these things can be significantly improved.
Probably we need to add some sort of simple VAD as a preprocessing step.

Here is a short video demonstration of near real-time transcription from the microphone:

rt_esl_csgo_1.mp4

Nice, but somehow can't kill it (stream) I had to do a killall -9 stream. On a AMD 2 thread 3ghz processor with 16GB RAM, there is significant delay. However I found that I get 2x realtime with the usual transcription on audio file. Great work. I love this.

ref #10 : handle Ctrl+C in "stream" app

Thanks for the feedback.
I just pushed a fix that should handle Ctrl+C correctly (it can take a few seconds to respond though).

Regarding the performance - I hope it can be improved with a better strategy to decide when to perform the inference. Currently, it is done every X seconds, regardless of the data. If we add voice detection, we should be able to run it less often. But overall, it seems that real-time transcription will always be slower compared to the original 30-seconds chunk transcription.

Thanks for the quick fix. I have some suggestions/ideas, for faster voice transcription. Give me half an hour to one hour, I'll update here with new content.

Edit / Updated:

Here are some ideas to speed up offline non real time transcription:

Removing silence helps a lot in reducing total time of audio (not yet tried but obvious):

http://andrewslotnick.com/posts/speeding-up-a-speech.html#Remove-Silence

Things that I tried with good results:

First I ran an half an hour audio file through https://github.com/xiph/rnnoise code. Then I increased the tempo to 1.5 with sox (tempo preserves pitch). After that I got good results with tiny.en but base.en seemed to be less accurate. Overall process is much faster - real fast transcription except for the initial delay.

cd /tmp
./rnnoise_demo elon16.wav elon16.raw
sox -c 1 -r 16000 -b 16 --encoding signed-integer elon16.raw elon16_denoised.wav
sox elon16_denoised.wav elonT3.wav tempo 1.5
./main -m models/ggml-tiny.en.bin -f /tmp/elonT3.wav

Here are some ideas for faster real time transcription:

I noticed that when I ran this on a 5 sec clip, I got this result:

./main -m models/ggml-tiny.en.bin -f /tmp/rec.wav 
log_mel_spectrogram: recording length: 5.015500 s
...
main: processing 80248 samples (5.0 sec), 2 threads, lang = english, task = transcribe, timestamps = 1 ...

[00:00.000 --> 00:05.000]   Okay, this is a test. I think this will work out nicely.
[00:05.000 --> 00:10.000]   [no audio]
...
main:    total time = 18525.62 ms

Now if we could apply this:

1 VAD / Silence detection(like you mentioned) split into chunks. The result is variable length audio chunks in memory or temp files
2 Remove noise with rrnoise on chunks
3 Speed up chunck by 1.5x preserving pitch (the speed up should just be an option. I learned that anything above 1.5x results are bad except if voice is loud clear and slow to start with, 1.5x is safe. Ideal is 1.1-1.5x max 2x)
4 Since we know exactly how long the sped up chunk is, we won't need to wait for transcription to finish...

Example:
[00:00.000 --> 00:05.000] Okay, this is a test. I think this will work out nicely. <--- We could kill it right here (cos this is the total length of that file / chunck I had as example)
[00:05.000 --> 00:10.000] [no audio] <-- This is processing on empty buffer, when killed would not waste processing

VAD: https://github.com/cirosilvano/easyvad or maybe use webrtc vad?

I guess experimentation is needed to figure out the best strategy / approach to real time considering the 30 sec at once issue.

ref #10 : add "step" argument for "stream" example

ref #10 : option to keep context in "stream" example

Some improvement on the real-time transcription:

rt_esl_csgo_2.mp4

I'll check this out and give you feedback here tomorrow. Awesome work! Brilliant.

@ggerganov

hello @ggerganov , thanks for sharing!
Offline main mode tested here on Windows worked fine.

~~Any small tip to include the SDL to make work the real time app?~~

I am a Whisper noob and furthermore not quite proficient at C++ but any pointers (pun intented) would be greatly appreciated

@pachacamac

@pachacamac wrote (a while ago)

Any way to register a callback or call a script once user speech is completed and silence/non-speak is detected? Been trying to hack on the CPP code but my CPP skills are rusty

and AFAICT this isn't the case yet so my basic advice is ./stream -f liveoutput getting the result in a text file then watch cat liveoutput to periodically (here every 2s) show the result.

A cleaner way to do this could be to rely on inotify to check if the file has been modified then act on that. Overall a bit of logic has to be added on top, e.g is the sentence detected new or filtering out things like [typing sounds] or (keyboard clicking), but I imagine it's enough to start without having to touch any C++ code.

To get the very last line that might match content cat liveoutput | grep -v '(' | grep -v ']' | tail -1

Is the -vth parameter working as intended? Just compiling the example and testing with large-v3 model (GPU, step set to 700 works fine) I get "hallucinations" when not talking. With English language there's a constant "thank you" output and with Swedish I get absolutely hilarious and rather long sentences from the training data.

Tried vth from 0.1 to 48000 without seeing any difference.

When I run the command "./stream -m ./models/ggml-large-v3.bin -t 8 --step 500 --length 5000 -l zh", the parameter "-l zh" does not seem to function properly. If I speak English, it transcribes correctly, but when I speak Chinese, there is no response. Additionally, there are some unknown Chinese characters displayed.

When I run the command "./stream -m ./models/ggml-large-v3.bin -t 8 --step 500 --length 5000 -l zh", the parameter "-l zh" does not seem to function properly. If I speak English, it transcribes correctly, but when I speak Chinese, there is no response. Additionally, there are some unknown Chinese characters displayed.

I also encountered this problem

Is the -vth parameter working as intended? Just compiling the example and testing with large-v3 model (GPU, step set to 700 works fine) I get "hallucinations" when not talking. With English language there's a constant "thank you" output and with Swedish I get absolutely hilarious and rather long sentences from the training data.

Tried vth from 0.1 to 48000 without seeing any difference.

You need to set --step 0. See: https://github.com/ggerganov/whisper.cpp/tree/master/examples/stream#sliding-window-mode-with-vad

I recently came upon this project from watching this video. It looks very good to me, perhaps the best Whisper "streaming" example I've seen.

It seems to be using whisper_streaming under the hood. I wonder how hard would it be to implement their algorithm on whisper.cpp?

Dear all, I try to using my Raspberry Pi 5 to display real time speech to text in my RP terminal, and below is my step:

sudo apt install libsdl2-dev
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make -j stream
./models/download-ggml-model.sh tiny

So far it can display, however, why it is always show up additional line paragraph "whisper_mel_init: n_len = 3370, n_len_org = 369, n_mel = 80" on top of my speech to text output?

Anyone know how to remove this annoying line?

Many thanks.

Placing 2>/dev/null at the end of your line will hide the messages and only show the result of the transcription.

I'm trying to hack some stuff together, and while doing that, I read this whole thread, it turns out to be very informative. And I want to thank ggerganov and everyone for contribution here. Thanks all

Is there any intention to get this to working with output devices rather than just input devices?

I have been playing around with this locally to try and force SDL2 to register a Windows output device. While I'm able to force this, Whisper doesn't seem to be able to recognise any audio playing through an output device.

I'm not sure if this is an SDL2 Limitation, or if it has to do with the AudioSpec configuration. I have configured the AudioSpec to be in Stereo (2 channels) and matched the hz of my audio device (48000hz) in common-sdl.cpp to no avail.

Has anyone gotten this usecase working or is it out of scope for whisper-stream?

SDL2 Version devel-2.30.11
Whisper v1.6.0

Is there any intention to get this to working with output devices rather than just input devices?

AudioRelay has a plugin (I don't remember the name, you can just install this one) where you can Simulate an Output as Input.
Although I think it doesn't recognize the text where there is music

@weskerty

@weskerty Thanks for the heads up! However, I'm trying to do this without using virtual audio devices or external dependencies. I've been able to get this working with VB-Audio virtual audio devices, but it's not the most elegant solution.

I'm going to continue digging into this when I have time, but any suggestions are welcome!

Merge pull request ggml-org#10 from tetherto/feat/bci-patches-v184

parakeet-cpp: address PR ggml-org#22 AOSC v2.1 review comments

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Support for realtime audio input #10

58 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

Support for realtime audio input #10

Description

Activity

ggerganov commented on Oct 1, 2022

biemster commented on Oct 1, 2022

ggerganov commented on Oct 2, 2022

ggerganov commented on Oct 2, 2022

trholding commented on Oct 2, 2022

ggerganov commented on Oct 2, 2022

trholding commented on Oct 2, 2022

ggerganov commented on Oct 10, 2022

trholding commented on Oct 10, 2022

moebiussurfing commented on Oct 11, 2022

58 remaining items

helins commented on Feb 17, 2024

Utopiah commented on Mar 10, 2024

troed commented on Mar 13, 2024

Toddinuk commented on Mar 14, 2024

zaccheus commented on Mar 22, 2024

pprobst commented on Apr 8, 2024

pprobst commented on Apr 9, 2024

olayno commented on Jun 22, 2024

weskerty commented on Jul 23, 2024

Kishlay-notabot commented on Aug 15, 2024

Soapwood commented on Jan 4, 2025

weskerty commented on Jan 5, 2025

Soapwood commented on Jan 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions