High latency (up to half minute) when doing fft on live audio from mixer

[hamoid]

I'm trying to do FFT on live audio from the mixer, so I can feed data into sketches based on whatever music I happen to be playing. The issue I'm having is latency: between 2 and 20 seconds. And it seems to drift, so I believe it increases while the program runs.

Could it be something related to mismatching bitrates or buffer sizes?

import ddf.minim.*;
import ddf.minim.analysis.*;
import javax.sound.sampled.*;

Minim minim;
AudioInput audio;
FFT fft;

// This affects latency, and in theory lower = less latency
int bufferSize = 512; // try 128 256 512 1024

void setup() {
  size(800, 100, P2D);
  // here you configure your audio device. Run it once and look in the console
  // to see available names. In my system one of the lines looks like
  // [2] default [default], version 4.18.0-12-generic
  // so I grab the part between [num] and the comma.
  startAudioInput("default [default]");
  noStroke();
}

void draw() {
  background(0);
  if (fft != null) {
    fft.forward(audio.mix);
    int bands = fft.avgSize();
    for (int i = 0; i < bands; i++) {
      float val = fft.getAvg(i);
      float x = map(i, 0, bands, 0, width);
      rect( x, height, 2, -val);
    }
  }
}

private void startAudioInput(String deviceName) {
  minim = new Minim(this);

  Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();

  for (Mixer.Info info : mixerInfo) {
    if (info.getName().equals(deviceName)) {
      minim.setInputMixer(AudioSystem.getMixer(info));
      audio = minim.getLineIn(Minim.STEREO, bufferSize);

      fft = new FFT(audio.bufferSize(), audio.sampleRate());
      fft.logAverages(22, 3);
      return;
    }
  }
  println(deviceName, "not found! Use one of these:");
  printArray(mixerInfo);
}

I also asked about it on the forum.

[ddf]

I haven't run the program yet, but my guess would be that the audio from the mixer is somehow not being read fast enough so it backs up over time.

It's also possible this is just a problem on Linux in particular. I just dug around in the code again to remind myself how the AudioInput works:

• it gets a TargetDataLine with a buffer size of 4x of what you request (to prevent buffer underrun). this is a JavaSound object that allows for reading audio from the input
• it gets a SourceDataLine with a buffer size of 4x what you request. this is a JavaSound output, which is what makes the monitoring feature of AudioInput possible.
• the Minim AudioInput class glues these two together by telling the output object to continuously read from the input object and that mix buffer is updated every time a new buffer is read

This should work OK, but I came across a comment I left for myself in the code that gets the SourceDataLine about latency just being bad in Linux:

Minim/src/main/java/ddf/minim/javasound/JSMinim.java

Line 818 in d549722

// remember that time you spent, like, an entire afternoon fussing

All that said, I worked on a project over the summer that also incorporated monitoring the current system audio in order to create reactive visuals. We were working on Windows, essentially through FMOD, and also ran into latency issues there. It wasn't as long as 20 seconds, but it was long enough to look incorrect. Basically I think the only way around this is to somehow mute the audio that is going directly from the browser or wherever and enable monitoring on your AudioInput so that you hear exactly what your program is reading.

You might also try to use a different Linux audio system? I'm not up on what's currently shipping with Linux, but I think I've read in the past that Pulse Audio generally has better latency than ALSA?

Another thing you could try, is to use getInputStream instead of getLineIn. This will give you an AudioStream (https://github.com/ddf/Minim/blob/master/src/main/java/ddf/minim/spi/AudioStream.java) object that you must call read on to get data out of it. It won't automatically pull. So in draw you'd read from the AudioStream until it read fewer samples than your bufferSize. Best would be to create MultiChannelBuffer (https://github.com/ddf/Minim/blob/master/src/main/java/ddf/minim/MultiChannelBuffer.java) with the same bufferSize that you use for the AudioStream and use the read method that takes a MultiChannelBuffer and returns number of samples read. You'd have to decide whether to run the FFT on every buffer or accumulate it somehow or whatever. Definitely it's more complex code and is why AudioInput exists in the first place.

This might help you determine whether the latency is coming from the read loop of AudioInput or if it is latency in putting audio data into the Java's TargetDataLine on the Linux side.

[neilcsmith-net]

Came through from the Processing forum post on this. In my experience Linux has by far the best latency performance of all the OS JavaSound implementations. I'm intrigued why we get such different results. Will try and take a look sometime.

[hamoid]

Thank you for the detailed answer!!! I'll do tests and report back :)

[ddf]

@hamoid Curious if you have anything to report about all this.

[rvorias]

Hey guys, I had the same problem on Elementary Juno (Ubuntu 18.04 LTS), Processing 3.5.3.
I implemented @ddf's recommendation and reading from an inputStream seems to do the trick for me. This is my QaD-code:

import ddf.minim.*;
import ddf.minim.spi.*;

Minim minim;
AudioStream in;

MultiChannelBuffer sampleBuffer;

void setup()
{
  size(512, 200, P3D);

  minim = new Minim(this);
  
  in = minim.getInputStream(2,1024,44100.0,16);
  in.open();
  sampleBuffer = new MultiChannelBuffer(1024, 2);
}

void draw()
{
  background(0);
  stroke(255);
  
  // draw the waveforms so we can see what we are monitoring

  in.read(sampleBuffer);
  for(int i = 0; i < 1023; i++)
  {
    line( i, 50 + sampleBuffer.getSample(0,i)*50, i+1, 50 + sampleBuffer.getSample(0,i+1)*50 );
    line( i, 150 + sampleBuffer.getSample(1,i)*50, i+1, 150 + sampleBuffer.getSample(1,i+1)*50 );
  }
}

void stop() {
  in.close();
}

Could someone verify this on their system?
update: it didn't go as well as I expected. Findings:

• Doing no FFT and audioStream @ 48000Hz, bufferSize 1024, almost no lag. Doing FFT afterwards causes more lag.
• Toyed with ALSA and pulseAudio software, added myself to audio usergroup for high priority.
• You can do pacmd list-sources to check some latency metrics, my latency metrics were all over the place.
• No luck using qjackctl (weird errors)
• No luck using oracle JRE (new to linux)
• It should be possible according to these guys link

Right now I reverted to using an audiosplitter (hardware) and rewire one output to the mic input and use that for audio analysis. It's noisy on the low-end spectrum but at least it's fast.

[hamoid]

I just added a Minim example to OPENRNDR and remembered this issue.

I tried my code above and also the one from @rvorias but I couldn't start diagnosing things because I'm getting the microphone input instead of my system audio, which was my original plan. I tried the different devices returned by AudioSystem.getMixerInfo(); (on Ubuntu).

At least the microphone input seems to have low latency :)