Parallel forced alignment

[m-toman]

This PR solves "ERROR [+7031] during forced_alignment.py" as dicussed in #293.

List of changes:

• Alignment in parallel on multiple chunks of data (checks the "DNN_NUM_PARALLEL" environment variable and uses 8 by default). Speeds up alignment on multi-core machines but also helps with numerical issues on larger datasets (seems to occur already with e.g. 4k sentences)
• Per-speaker normalisation fix (creates new "normaliser" per speaker instead of once)
• Outputs "logprob.txt" containing sentence likelihoods
• "work_dir" and "HTKDIR" can also be passed as commandline arguments instead of sed replacement, compatibility to sed replacement was retained though
• "-a" command line parameters enables multi-speaker normalization

I've tested the latest script in my environment with a 4000 sentence corpus, please report if this breaks any example scripts, which could happen for some sed reasons (pun intended).

[martedva]

Hello @m-toman,
I'm getting the follow error, when trying to run script ./02_prepare_labels, with your forced_alignment.py file.

forced-alignment using HTK tools...
Traceback (most recent call last):
  File "/home/tts/merlin/misc/scripts/alignment/state_align/forced_alignment.py", line 414, in <module>
    work_real_dir = sys.argv[1]
IndexError: list index out of range
ls: cannot access 'database/labels/label_state_align': No such file or directory
Force-alignment unsucessful!! Please check /home/tts/merlin/misc/scripts/alignment/state_align/forced_alignment.py
Copying labels to duration and acoustic data directories...
cp: cannot stat 'database/labels/label_state_align': No such file or directory
cp: cannot stat 'database/labels/label_state_align': No such file or directory
ls: cannot access 'database/labels/label_state_align': No such file or directory
ls: cannot access 'database/labels/label_state_align': No such file or directory
done...!

Any thoughts?
Thanks in advance,

[gillesdegottex]

@m-toman must have done it so well that he also replaced the ugly sed-based arguments replacement by proper command line arguments :)
@martedva: if you can't wait for his update, just have a look at the lines right below "if name == 'main':" in forced_alignment.py and pass the directories as argument in scripts/run_state_aligner.sh

[martedva]

UPDATE: So changing the below two lines of code seems to fix it in all other egs than 'speaker-adaptation'. This one still throws the above error message, unless you run each script individually in the merlin/misc/scripts/alignment/state-align folder.

Hello @gillesdegottex and @m-toman!

It seems that the bug for me was in line 414:
if (len(sys.argv)) >= 1:

This line deemed it impossible to run the script without giving it a command-line argument. I changed both line 414 and 419 to:
414 if len(sys.argv) > 1:
418 if len(sys.argv) > 2:

[m-toman]

You are right, I forgot that python holds the script name in argv[0] when I added then len() ifs.
Would be nice to replace it with getopts (but you know, interspeech deadline and all that ;)).
Thanks for the fix.

I tried to name the hacky temporary variables below that so that they are not hit by the sed replace but the script still works without parameters.

Regarding the speaker adaptation script, I think this should work with the new version:
https://github.com/CSTR-Edinburgh/merlin/blob/master/egs/speaker_adaptation/s1/scripts/run_state_aligner.sh#L47-L51

[martedva]

Hey @m-toman!

Thanks, I will try the run_state_align fix. However, I run into the following error message, when I try to run the run_aligner script in the merlin/misc/scripts/alignment/state_align folder. It looks as if it is right after the feature normalisation, when I reach the "--making proto" step.

---feature_normalisation
Fri Mar 23 19:37:05 2018
---making proto
  ERROR [+2019]  HCompV: Training data file name expected
 FATAL ERROR - Terminating program /home/tts/merlin/tools/bin/htk/HCompV
Traceback (most recent call last):
  File "forced_alignment.py", line 430, in <module>
    aligner.prepare_training(file_id_list_name, wav_dir, lab_dir, work_real_dir, multiple_speaker)
  File "forced_alignment.py", line 262, in prepare_training
    self._make_proto()
  File "forced_alignment.py", line 61, in _make_proto
    '-M', self.cur_dir, self.proto])
  File "/home/tts/anaconda2/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/tts/merlin/tools/bin/htk/HCompV', '-f', '0.01', '-C', '/home/tts/merlin/misc/scripts/alignment/state_align/config/cfg', '-S', '/home/tts/merlin/misc/scripts/alignment/state_align/config/train.scp', '-M', '/home/tts/merlin/misc/scripts/alignment/state_align/model/hmm0', '/home/tts/merlin/misc/scripts/alignment/state_align/config/proto']' returned non-zero exit status 227
tts@tts:~/merlin/misc/scripts/alignment/state_align$ ./setup.sh 
Path to Merlin and other tools configured in config.cfg
setup done...!

[m-toman]

Could you output the parameters for HCompV before the call here https://github.com/m-toman/merlin/blob/master/misc/scripts/alignment/state_align/forced_alignment.py
and see if the paths look correct and actually point to stuff?
Thanks, I'll also re-run it to see if I can reproduce it.

[martedva]

Hello @m-toman!

How do i find the parameters for HCompV?

I have run the prepare_labels_from_text.sh script, and the next script is now forced_alignment.

Best Regards,

[m-toman]

@martedva I just meant something simple as:
## make vFloors print ("HCompV: " + str(self.cfg) + " " + str(self.train_scp) + " " + str(self.cur_dir)) check_call([self.HCompV, '-f', F, '-C', self.cfg, '-S', self.train_scp, '-M', self.cur_dir, self.proto])

the error above sounds to me as if the file self.train_scp points to is not a list of files as expected.
Then either the path constructed in forced_alignment is wrong because I the sed-replacing in the outer scripts does not cooperate with my version, or there's some other issue.

[martedva]

Hello @m-toman!

Sorry for the late response. I have been trying to fix the issues. It seems that it is indeed the sed replacements, that was culprit. I used a fixed path instead and fixed it.

Thanks,

[m-toman]

@martedva great, so is there something I can do in this PR or was that a fix in the outer scripts?

[martedva]

Hey @m-toman,

It seemed, that the sed-replacements resulted in setting both of these paths to being empty. I therefore ran the scripts individually and reset the paths in forced_alignment.py just before running it as shown below.

        work_dir = "/home/tts/merlin/misc/scripts/alignment/state_align"
        work_real_dir = work_dir
    if len(sys.argv) > 2:
        htk_dir = sys.argv[2]
    else:
        #this HTK path
        HTKDIR = "/home/tts/merlin/tools/bin/htk"
        htk_dir = HTKDIR
    #this wav path
    wav_dir = "/home/tts/merlin/egs/speaker_adaptation/s1/database/wav"

Does that make sense?

[ndwuhuangwei]

sorry to bother in 2019, I just come across the same issue with HcomPv and I can't understand the comments. what's sed replacements mean? which fixed path did you use to fix the issue and how to modify this path?
I'm just a undergraduate and new to merlin，I'm sorry if these questions are obviously stupid

[ndwuhuangwei]

I can't find this part of code in forced_alignment.py , should I just add it ?(add to where?)

    work_dir = "/home/tts/merlin/misc/scripts/alignment/state_align"
    work_real_dir = work_dir
if len(sys.argv) > 2:
    htk_dir = sys.argv[2]
else:
    #this HTK path
    HTKDIR = "/home/tts/merlin/tools/bin/htk"
    htk_dir = HTKDIR
#this wav path
wav_dir = "/home/tts/merlin/egs/speaker_adaptation/s1/database/wav"