Timestamps always maximum length when using Silero VAD

[polytr0pe]

Transcription appears to be accurate, however the ending timestamps for each line are always set at the beginning timestamp of the next line, resulting in subtitles constantly displayed long after speech ends, e.g.:

37
00:09:44,419 --> 00:09:56,950
I can't solve the problem at this rate.

38
00:09:56,950 --> 00:10:07,269
What should I do?

39
00:10:07,269 --> 00:10:11,269
I'll take my time and look at it.
``

[jhj0517]

Hi, This seems to be default behavior of the whisper, not just VAD.

Regardless of whether each segment tries to start from the end of the previous segment (if possible), it should still captrue the correct timestamp.

If the longer segment is the problem, try setting min_speech_duration_ms and min_silence_duration_ms to 250ms and turn on the BGM separation filter.

[jhj0517]

Just reproduced hallucinations on a sample.

This happens under all the large-v2 models. With large-v3 models I got better clipped timestamps.

+) And I observed the same hallucinations with faster-whisper as well, and it causes the same in the WebUI, because I followed the same way with faster-whisper. A different implementation of the vad is needed.

[montvid]

pyannote have their own VAD as I understand - maybe one could use it?
From https://github.com/m-bain/whisperX

Valuable VAD & Diarization Models from [pyannote audio][https://github.com/pyannote/pyannote-audio]

From https://github.com/shashikg/WhisperS2T

NVIDIA NeMo Team: Thanks to the NVIDIA NeMo Team for their contribution of the open-source VAD model used in this pipeline.

[montvid]

a fix was merged for faster whisper - maybe that solves the problem? SYSTRAN/faster-whisper#921

[jhj0517]

SYSTRAN/faster-whisper#921

Thanks, but it might not be about improving this, it's probably about fixing the batch transcription bug.

I'm considering implementing whisperX's and my own implementation for the VAD and using the better one after comparing them.

I don't have time for this right now, hope I can do it as soon as possible. Or any PR for this would be welcome.

[montvid]

Faster-whisper programmers say it is not a VAD issue maybe here:
[ After using VAD, the start and end times of the recognized segments are incorrect #1119 ]SYSTRAN/faster-whisper#1119