Try example with file stream

[surasakBoonkla]

Hello,

I tried the example but change from mic to file stream. There was no error but nothing printed out. Please help me.

import rx.operators as ops
import diart.operators as dops
#from diart.sources import MicrophoneAudioSource
from src.diart.sources import FileAudioSource
from diart.blocks import SpeakerSegmentation, OverlapAwareSpeakerEmbedding

segmentation = SpeakerSegmentation.from_pyannote("pyannote/segmentation")
embedding = OverlapAwareSpeakerEmbedding.from_pyannote("pyannote/embedding")
sample_rate = segmentation.model.get_sample_rate()
#mic = MicrophoneAudioSource(sample_rate)
file = FileAudioSource('Toefl-16k-08.wav', sample_rate)

stream = file.stream.pipe(
# Reformat stream to 5s duration and 500ms shift
dops.regularize_audio_stream(sample_rate),
ops.map(lambda wav: (wav, segmentation(wav))),
ops.starmap(embedding)
).subscribe(on_next=lambda emb: print(emb.shape))

#mic.read()
file.read()

[juanmc2005]

Hi @surasakBoonkla,

The problem is that FileAudioSource is already formatted to a specific chunk duration and step, by default these are 5s and 500ms, so my guess is that dops.regularize_audio_stream(sample_rate) is causing problems, could you try again without this line?

This particular aspect of the audio sources is currently not very well designed. As you may have noticed, it's not very intuitive that files are already formatted but not the microphone. This is going to change starting from next release (v0.6), with which your example should work correctly. In fact, it is already implemented in the develop branch if you want to try it out.

Part of the road to v1.0 is to identify issues like this to design a robust and consistent API, so thank you for bringing this up!

[surasakBoonkla]

Dear Juan,

Thanks, it works now after I removed that line.
Your work is interesting. I am trying to understand its workflow.

thanks again

[surasakBoonkla]

Dear sir,

Please teach me more a little bit. I am not good much in Python.

I would like to use both wav (or segmentation(wav)) and embedding to be processed. My idea is to track speakers by using wav (or segmentation(wav)) and embedding in a function.

Can I modified your example like this? I added a variable wav before emb but it did not work.

import rx.operators as ops
import diart.operators as dops
#from diart.sources import MicrophoneAudioSource
from src.diart.sources import FileAudioSource
from diart.blocks import SpeakerSegmentation, OverlapAwareSpeakerEmbedding

segmentation = SpeakerSegmentation.from_pyannote("pyannote/segmentation")
embedding = OverlapAwareSpeakerEmbedding.from_pyannote("pyannote/embedding")
sample_rate = segmentation.model.get_sample_rate()
#mic = MicrophoneAudioSource(sample_rate)
file = FileAudioSource('friday16k.wav', sample_rate)

import torch

def process(wav, emb):
#np_emb = emb.numpy()
embeddings = emb.detach().cpu().numpy()

print(embeddings.shape)
print(wav)

stream = file.stream.pipe(
# Reformat stream to 5s duration and 500ms shift
#dops.regularize_audio_stream(sample_rate),
ops.map(lambda wav: (wav, segmentation(wav))),
ops.starmap(embedding)
).subscribe(on_next= lambda wav, emb: process(wav, emb))

#mic.read()
file.read()

[juanmc2005]

Keep in mind that ops.starmap(embedding) will not output the audio chunk, only the embedding tensor.
You can take a look at how it's implemented in OnlineSpeakerDiarization (this line).

[surasakBoonkla]

OK. I see.

Thanks a lot.